Cluster Node Design: CPU, Memory, GPUs, NUMA and PCIe

By Chander Kant Updated July 31, 2026

Chapter 1 introduced the cluster node as the basic unit from which a Linux cluster is assembled. We now look inside that unit. A process may execute on a CPU core, obtain data from a cache or memory controller, transfer work to an accelerator, read from local storage, and send a message through a network adapter. Every one of those steps crosses a boundary with finite bandwidth, latency, and capacity.

Node design is therefore a problem of arranging data paths. A processor with a high peak arithmetic rate may spend much of its time waiting for memory. A GPU may sit idle while data crosses a constrained PCI Express path. A fast network adapter may be attached to the wrong NUMA domain, forcing traffic through an additional socket-to-socket link before it reaches the application. These are not unusual corner cases. They are common consequences of selecting components separately instead of evaluating the node as a system.

We will develop that system view by following the work through processors, memory, accelerators, PCIe, local storage, physical construction, and management. The final sections show how Linux exposes the resulting topology and how to test a node before accepting a cluster built from hundreds or thousands of copies of it. Chapter 3 continues from the network adapter into the inter-node fabric.

The node as a balanced data path

The term compute node can describe very different machines. One cluster may use compact single-socket CPU servers. Another may use large dual-socket nodes with several terabytes of memory. An accelerated node may contain four or eight GPUs, high-bandwidth device memory, several network interfaces, PCIe switches, and local NVMe drives. We compare these nodes by the application work they complete within their power, cooling, network, and budget limits.

For this reason, we begin with a description of the application's behavior rather than a preferred processor model. We record the working-set size, memory access pattern, arithmetic precision, vectorization, thread scaling, accelerator support, local I/O, and communication pattern. We also record how these properties change with problem size. A small test that fits in cache can favor a different machine from the production case that streams through hundreds of gigabytes.

Figure 2.1: The principal paths inside a compute node. Each boundary can impose a capacity, bandwidth, latency, or locality limit.

Workload behavior	Likely node pressure point	Evidence to collect
Many independent jobs	Cost, core throughput, memory per job, provisioning rate, and energy per completed job.	Jobs per node, throughput per watt, memory high-water mark, and scheduler occupancy.
Large shared-memory computation	NUMA locality, memory capacity, sustained bandwidth, cache behavior, and thread scaling.	Scaling by core and socket, STREAM bandwidth, NUMA misses, and application memory traffic.
Vector or matrix kernels	Instruction support, compiler quality, data layout, cache reuse, and memory bandwidth.	Compiler reports, hardware counters, effective bandwidth, and time in the production kernels.
GPU or accelerator computation	Device-memory capacity and bandwidth, host-to-device movement, peer paths, and software support.	Kernel profile, device occupancy, transfer volume, peer bandwidth, and time spent staging data.
Communication-intensive MPI	NIC locality, PCIe path, host overhead, memory registration, and inter-node latency.	Rank placement, adapter NUMA node, PCIe topology, pairwise tests, and application communication time.
Data preparation or checkpointing	Local and shared storage paths, CPU overhead, metadata activity, and contention with computation.	Read/write sizes, queue depth, sustained application I/O, and overlap with network traffic.

Balance means that each constrained subsystem meets the application's demand with an acceptable margin. The nominal bandwidths need not be equal. Vendor peak values usually describe one component under favorable conditions, while the application experiences the complete path, including protocol overhead, contention, non-local access, software, and thermal limits.

Processors and software compatibility

Most general-purpose Linux compute nodes use either the x86-64 or AArch64 instruction set. Other architectures remain important in particular systems, and accelerators add their own programming and execution models. Performance follows from the complete implementation. Two processors that use the same instruction set may differ substantially in core design, vector facilities, cache hierarchy, memory channels, I/O lanes, power behavior, and the maturity of the surrounding software.

Software compatibility should be tested before comparing benchmark numbers. Confirm that the required Linux distribution, compiler, MPI implementation, math libraries, containers, drivers, monitoring agents, and application dependencies support the architecture. Source availability does not guarantee a clean port: build systems may contain architecture assumptions, and optimized libraries may not offer equal coverage on every platform. Conversely, an application with portable source and well-supported libraries may move between architectures with little difficulty.

Cores, threads, and frequency

A physical core contains execution resources that retire instructions. Simultaneous multithreading, where available, exposes more than one logical CPU on a core so that one thread may use resources while another is stalled. Those logical CPUs share the core's arithmetic units, cache, and access to memory. Some workloads gain useful throughput from multithreading; others lose time to that competition. We measure both configurations.

Core count and clock frequency describe different parts of the processor. A wide out-of-order core may perform more work per cycle than a simpler core, while a processor with more cores may operate each core at a lower sustained frequency under full load. Turbo frequencies are useful for lightly threaded work but should not be used as the expected all-core rate. The cluster's power and cooling policy can further change sustained frequency, so performance testing should record clocks, package power, and temperature over the complete run.

Vector and matrix execution

Scientific and AI applications often perform the same operation over many data elements. CPUs provide vector instructions for this work, while some processors also provide matrix-oriented units. GPUs execute large groups of threads and include specialized matrix hardware of their own. Compilers and libraries must emit the appropriate instructions, and the data layout must supply those units efficiently.

Precision is part of the design. A weather model that requires double precision should not be sized from a low-precision AI rate. An inference workload may use several integer or floating-point formats in different stages. Peak operation counts should therefore be labeled by precision, sparsity assumptions, and operating mode. Application results remain the final comparison because instruction mix, memory traffic, branching, and synchronization determine how much of a peak rate can be used.

Caches and shared resources

Modern processors place several cache levels between the cores and main memory. Small private caches are close to each core; larger caches may be shared by a group of cores, a chiplet, or an entire package. Cache can turn repeated access to a compact data set into a fast local operation. Once the working set exceeds the relevant cache, the memory subsystem becomes visible.

Shared caches introduce their own interactions. Threads can evict one another's data, and independent variables that occupy the same cache line can cause coherence traffic when different cores update them. This false sharing is a software layout problem that can make a capable node appear to scale poorly. Hardware performance counters and controlled thread-placement tests help distinguish cache and coherence limits from a shortage of arithmetic units.

Memory capacity, bandwidth, and NUMA

Memory must first be large enough for the work assigned to the node. Include application data, temporary arrays, communication buffers, file-system cache where relevant, accelerator staging, and the operating system. Leave headroom for variation rather than sizing to one observed minimum. Capacity per core is a useful initial ratio, but it can conceal the real requirement when only a subset of ranks holds large data structures.

After capacity comes bandwidth. Each processor package has a finite number of memory channels. Adding cores increases potential demand without automatically adding channels, so bandwidth available per core can fall as core counts rise. Memory modules must also be populated according to the platform's channel and speed rules. Empty channels reduce available bandwidth, while placing more DIMMs on each channel can lower the supported memory data rate. The exact result depends on the processor, board, DIMM type, rank, and firmware table, so the approved population must be checked against the server manual and then measured.

STREAM measures sustainable bandwidth for simple operations on arrays larger than cache. It is a useful characterization tool, but it is not a substitute for the application. STREAM uses regular access patterns; graph analytics, sparse solvers, databases, and language-model serving can generate different locality, concurrency, and read/write behavior. Use it to establish whether the memory system is assembled and configured correctly, then compare the application against that baseline.

Local and remote memory

Multi-socket and some chiplet-based systems use non-uniform memory access, or NUMA. A CPU has lower-cost access to memory attached to its local NUMA node and reaches other memory through an internal interconnect. Linux normally attempts local allocation, but placement depends on where a thread runs when it first touches a page, the process memory policy, cpusets, and memory pressure. A process can therefore have enough total memory while still suffering from avoidable remote traffic.

CPU affinity and memory policy must be considered together. Pinning threads without controlling page placement may lock computation beside remote memory. Interleaving memory can increase aggregate bandwidth for some streaming workloads, yet it can hurt an application whose threads and data already have clear locality. There is no universal NUMA policy. The appropriate policy follows from the application's decomposition and must be measured at full thread count.

Linux exposes NUMA nodes, their distances, and allocation statistics. The numactl tools can bind CPUs and memory for an experiment, while /proc/<pid>/numa_maps and numastat help show where pages were placed. The scheduler can make useful locality decisions, but it cannot infer every application's ownership of data. MPI rank placement and threaded affinity remain part of the run configuration.

Reliability and memory expansion

Cluster nodes should use platform-supported error-correcting memory and expose corrected and uncorrected error reporting to the operating environment. Corrected errors preserve the current data, while a rising count can identify a degrading component before it produces an uncorrectable failure. Linux EDAC and platform RAS facilities can report these events; the monitoring system must collect them and associate a logical error with the physical module that should be replaced.

Compute Express Link, or CXL, extends PCIe-based connectivity with protocols for devices and memory. It can provide memory expansion and tiering options on supported platforms. We treat that capacity as a distinct memory domain because latency, bandwidth, topology, operating-system support, firmware, and application placement determine whether it helps a workload. As with NUMA, the system should expose the distance and performance of each domain and the acceptance test should exercise the intended policy.

Accelerators and intra-node topology

Each GPU or other accelerator adds compute units, local memory, links to the host, and sometimes direct links to peer devices. Their arrangement turns the accelerator node into a topology of its own. That topology determines which transfers are local, which cross a PCIe switch or CPU root complex, and which can use a dedicated scale-up fabric.

High-bandwidth memory attached to an accelerator can deliver far more bandwidth to that device than ordinary host memory, but its capacity is finite. A model or simulation that does not fit may require partitioning, recomputation, compression, or movement across a slower path. Unified virtual addressing can simplify programming while the physical memory remains distributed. Performance still favors keeping data near the processor that uses it and minimizing unnecessary migration.

Multi-accelerator nodes add peer communication. Some systems connect devices through PCIe; others provide dedicated links and switches for higher-bandwidth scale-up communication. Communication libraries use the discovered topology to choose rings, trees, or direct peer paths. Verify the resulting map rather than assuming all pairs have equal connectivity.

Figure 2.2: A simplified dual-socket accelerator topology. GPU peer links can provide a fast scale-up path, while CPU, memory, PCIe, and NIC locality still determine how data enters and leaves each device group.

The network adapter is part of this topology. GPU-direct RDMA can allow a compatible adapter to exchange data with device memory without staging every transfer through a host buffer. The path still depends on PCIe placement, peer-access support, drivers, IOMMU and access-control configuration, and the communication library. A NIC placed near one group of GPUs can be non-local to another. This is the first direct connection to the host-overhead and locality questions developed in Chapter 3.

PCIe and the node I/O tree

PCI Express replaced the shared parallel PCI and PCI-X buses with point-to-point serial links arranged as a hierarchy. A root complex connects the processor and memory system to endpoints such as network adapters, GPUs, and NVMe drives. Switches can create more downstream ports, but their uplinks remain finite. A node can therefore contain several devices whose individual link rates add up to more than the path they share.

Generation and lane width provide the theoretical starting point for a PCIe link. The negotiated generation and width provide the actual link configuration. A mechanically x16 slot may be wired for fewer electrical lanes, and some risers divide one upstream link among several endpoints through PCIe bifurcation. Signal quality, platform firmware, slot wiring, risers, retimers, and device support can also cause a link to train below its intended rate. Linux reports both the capability and current status through PCI configuration space, making this a straightforward acceptance check.

Topology matters as much as rate. A transfer between a NIC and memory attached to the same CPU can follow a shorter path than a transfer that crosses the socket interconnect. A GPU and NIC beneath the same PCIe switch may support an efficient peer path, while another pair may be routed through a root complex or disallowed by platform access controls. The server block diagram should be examined before purchase and verified with Linux after delivery.

Component	Topology questions	Failure to avoid
Network adapter	Which NUMA node and root complex are local? Does the intended slot provide the required width? Is a second rail truly independent?	Every message crossing a socket link or two nominal rails sharing one constrained uplink.
GPU or accelerator	Which peers have direct access? Which links use PCIe and which use a dedicated scale-up fabric? Is peer traffic routed as expected?	Assuming equal connectivity among devices that occupy different branches of the topology.
NVMe storage	Are drives direct-attached or behind a switch? What else shares the uplink? Which CPU services interrupts and I/O queues?	Local checkpoint traffic contending with the NIC or accelerators on the same constrained branch.
PCIe switch or retimer	What is the upstream width and generation? Is oversubscription acceptable for simultaneous production traffic?	Adding endpoints without adding upstream capacity, then sizing from the sum of endpoint rates.

Local storage

Local storage may hold the operating system, temporary files, container images, a burst buffer, or data reused by jobs on that node. NVMe reduces device latency and supports many queues. The application still crosses a file system, block layer, PCIe path, and memory copies, and its queue depth and transfer size can produce very different results from a maximum-throughput storage test.

Diskless compute nodes remain useful where administrators want one centrally managed software image and little persistent state on the workers. A node may boot through PXE or iPXE and obtain its root image from the network, copy it into RAM, or assemble it from an immutable image plus local temporary storage. This design moves boot and image traffic onto the service network and consumes memory if the root resides in RAM. It also makes the availability and capacity of provisioning services part of node acceptance.

Decide which data is disposable and which must survive a node failure. Node-local scratch can improve performance without becoming a system of record. Checkpoints or unique results normally require movement to shared storage or another protected tier. Chapter 4 develops this distinction and follows the storage path beyond the node.

Turning workload requirements into a node

Once the subsystem behavior is understood, we can size the node as a set of ratios: memory capacity per job or accelerator, sustainable memory bandwidth per busy core, accelerator memory per model partition, PCIe capacity per active device, network capacity per process or accelerator, and power per completed unit of work. These ratios expose which resource becomes scarcer when another component is added.

Node size also changes the cluster-level design. Larger nodes reduce the number of operating-system images, network ports, and some forms of inter-node communication. They also increase the amount of work lost when one node fails and can create larger NUMA and I/O domains. Smaller nodes may provide finer scheduling granularity and simpler locality, at the cost of more nodes, cables, ports, and management activity.

Cluster category from Chapter 1	Common node tendency	Questions that decide the design
Throughput cluster	Cost-efficient nodes sized to run several independent jobs with high occupancy.	How many jobs fit without memory or I/O interference? Does a larger node improve utilization or create stranded resources?
Capability cluster	Nodes with strong memory bandwidth, low-overhead I/O, and enough cores or accelerators to reduce inter-node communication.	Where does shared-memory scaling stop? Does adding resources to a node reduce total time or move the bottleneck into NUMA and PCIe?
Inference cluster	Accelerator-rich nodes sized around model weights, runtime state, batching, latency targets, and service redundancy.	Does the model fit in device memory? Which parallelism crosses device or node boundaries? What capacity is needed for failures and traffic bursts?

Heterogeneous clusters are reasonable when workloads have genuinely different requirements. They also increase the number of software images, scheduler constraints, spare parts, performance baselines, and failure modes. A node type should therefore represent a meaningful workload class, not a minor purchasing variation. Within each type, consistency makes performance easier to predict and faults easier to diagnose.

Physical design, power, and serviceability

The server design must deliver power, remove heat, hold devices without obstructing airflow or service access, and expose the management functions required at cluster scale. Rack density is limited by facility power and cooling as well as by the number of chassis that physically fit.

Thin air-cooled servers can provide high CPU density, but accelerators, large memory configurations, and high-power NICs increase thermal load. A device that repeatedly reaches a thermal or power limit can reduce clock frequency and produce variable job times. Liquid cooling can support higher heat density, but it adds facility interfaces, monitoring, service procedures, and leak-management requirements. Either design should be evaluated at sustained production load, not only during a short benchmark.

Power design includes the redundancy policy. Dual power supplies can be operated across independent A and B feeds, but the available power after one supply or feed fails must still support the node's intended load. N+1 and N+N policies describe different failure assumptions at server and rack scale. Power caps can protect the facility envelope and improve efficiency, though they must be tested because a cap can also change application performance and runtime.

Serviceability becomes more important as node count rises. Drives, fans, power supplies, memory modules, and accelerator trays should be replaceable with predictable procedures. Component labels reported by firmware and Linux should correspond to physical labels in the chassis. Otherwise an error message that identifies a DIMM or fan does not reliably tell the technician what to replace.

Out-of-band management

A server-class node normally includes a baseboard management controller, or BMC, with power control, sensor readings, event logs, firmware update mechanisms, and a console that remains available when the host operating system is down. Redfish provides a standard management interface, and OpenBMC is an open-source firmware stack used on some platforms. IPMI remains common on existing equipment. Serial-over-LAN carries the host's serial console through the BMC and is especially valuable for observing firmware, kernel boot, and crash output on a headless node. Actual feature coverage still varies by vendor and model.

The BMC belongs to the management and service scope described in Chapter 3's interconnect scopes. It should use a management network separated from application traffic. Credentials, firmware provenance, update procedures, audit logs, and recovery access belong in the node design and installation plan.

Inspecting a node from Linux

Purchase documents describe the intended machine. Linux shows what firmware and drivers exposed after assembly. Keep a topology record for every node type and compare delivered systems against it. The following commands answer different questions:

# CPU packages, cores, threads, caches, and NUMA assignment
lscpu
lscpu -e=CPU,NODE,SOCKET,CORE,CACHE,ONLINE

# NUMA capacity and relative distance
numactl --hardware
numastat

# Combined CPU, cache, memory, and I/O topology
lstopo-no-graphics

# PCIe hierarchy and negotiated device details
lspci -tv
lspci -vv -s <domain:bus:device.function>

# Memory modules and firmware-reported hardware
sudo dmidecode --type memory
sudo dmidecode --type system

# Local block devices and their topology
lsblk -o NAME,MODEL,SIZE,ROTA,TYPE,TRAN,MOUNTPOINTS

# Corrected and uncorrected hardware events
ras-mc-ctl --summary
journalctl -k

Program 2.1: A first-pass inventory of a Linux compute node.

On NVIDIA GPU nodes, nvidia-smi topo -m reports relationships among GPUs, NICs, CPUs, and NUMA nodes. AMD environments provide topology and status through ROCm tools and sysfs. Hardware-locality tools such as lstopo are especially useful because they place PCI devices beneath the CPU and NUMA objects to which Linux considers them local.

Inventory output should be stored with the node definition. It provides a baseline when one server performs differently from its peers and reveals silent substitutions or assembly differences. Firmware revisions, kernel, drivers, power policy, memory population, and PCIe link state belong in the same record.

Test one node before multiplying it

A cluster repeats its node design. An unnoticed problem in one prototype can become a fleet-wide limitation, so acceptance should begin with a single fully configured node and then compare a sample from the delivered population. Each stage narrows a different class of fault:

Verify identity and topology. Record processors, memory modules, NUMA domains, PCIe trees, devices, firmware, and negotiated links. Compare them with the approved design rather than only the purchase order.
Measure CPU behavior. Run representative scalar, vector, and threaded work. Record sustained frequency, package power, temperature, hardware errors, and scaling from one core through the intended production count.
Measure memory by locality. Run STREAM or another controlled bandwidth test within each NUMA node and across the complete machine. Test local, remote, and intended interleaved placement where applicable.
Exercise accelerator memory and peer paths. Verify device memory, host-to-device transfers, peer access, scale-up links, and the topology used by the production communication library.
Load PCIe devices together. A GPU, NIC, and NVMe drive may pass separate tests but contend when active simultaneously. Exercise the combinations that production jobs will create.
Test local storage through the intended file system. Use representative block sizes, queue depths, read/write mixtures, and data durability settings. Do not infer application behavior from one maximum-throughput test.
Hold a sustained thermal load. Short tests may end before cooling and power controls settle. Run long enough to identify throttling, unstable clocks, marginal cooling, and correctable hardware errors.
Exercise management and recovery. Confirm console access, sensor collection, event logs, orderly and forced power control, firmware inventory, and the procedure for returning a failed node to service.
Run the application. Compare useful work, power, and variability with the design target. This final test determines whether the subsystem results combine into a balanced node.

These are single-node tests. Once the node reaches the network adapter, Chapter 3's fabric acceptance sequence measures pairwise communication, topology-wide load, collectives, and failure behavior across nodes. Keeping the two stages separate makes a poor host path easier to distinguish from a poor network.

What the earlier architectures still teach

When I first wrote this chapter, IA-32 processor modes and registers, Itanium, the new AMD64 extension, AT and ATX motherboards, PCI, PCI-X, and an early vision of InfiniBand were current design choices. Their detailed specifications no longer provide useful purchasing guidance, but the transitions explain much of the system we now have.

From IA-32 and Itanium to x86-64

IA-32 brought the protected 32-bit x86 environment into widespread use, but its ordinary address space limited each process to 4 GiB. Physical Address Extension allowed a 32-bit operating system to manage more physical memory, without giving one ordinary process a flat address space larger than 4 GiB. Growing scientific data sets made a practical 64-bit transition increasingly important.

Intel and Hewlett-Packard pursued IA-64 and Itanium, built around Explicitly Parallel Instruction Computing. The architecture exposed substantial instruction-level parallelism to the compiler. AMD instead extended x86 with a 64-bit mode, more registers, and a larger address space while preserving a direct path for existing x86 software. That compatibility strategy became the x86-64 baseline used by both AMD and Intel systems.

The durable lesson is that processor architecture, compilers, operating systems, libraries, and the installed application base form one platform. Compatibility was decisive in this transition, while a future transition may weigh energy, memory, or another constraint more heavily. In every case, we evaluate the processor together with the software needed to use it.

From desktop form factors to cluster servers

AT and Baby AT motherboards arranged processors, memory sockets, connectors, and expansion cards for desktop computers of their time. ATX improved connector placement, airflow, power control, and access to expansion slots, and its descendants remain familiar in workstations and smaller servers. Large clusters, however, increasingly use purpose-built rack servers, blades, and sleds whose boards, risers, fans, power distribution, and management controllers are designed as one serviceable unit.

The historical lesson is physical rather than electrical. Placement determines cooling, cable access, usable slot length, service procedure, and the devices that can operate simultaneously. Current server boards may look very different from ATX, yet we still need the questions that motivated the original form-factor discussion: Will the parts fit, receive power, stay cool, and remain accessible after installation?

From PCI and PCI-X to PCIe

Conventional PCI was a shared parallel bus. PCI-X raised its rate and improved transaction handling for server I/O, but devices still shared a bus segment whose timing and loading became harder to scale. PCIe changed the physical and topological model to serial point-to-point links and switches. Generations and lane counts have advanced, yet the hierarchy of root complexes, switches, uplinks, and endpoints still determines contention.

The older discussion of bus mastering also survives in a more general form. High-performance devices use direct memory access so that the CPU does not copy every byte itself. RDMA adapters, NVMe controllers, and accelerators all build on this ability. We now ask which memory a device can reach directly, through which path, with what protection and software support.

InfiniBand moves into the cluster fabric

Early InfiniBand material described a broad system-area I/O architecture in which host channel adapters connected servers and target channel adapters could connect remote peripherals. The target-channel vision did not replace the server's internal PCI hierarchy. InfiniBand did, however, become an important switched interconnect for HPC and AI clusters.

The host channel adapter still belongs in this chapter because it occupies a node's I/O tree and accesses host or accelerator memory. The links, switches, routing, congestion, and communication behavior beyond that adapter belong to Chapter 3. This boundary between the host path and the fabric is clearer than calling InfiniBand a replacement for the PCI bus.