Cluster Interconnects

The network fabric that carries messages between the compute nodes is the cluster interconnect. A cluster may also have management, storage, service, and client-facing networks, but those networks serve different traffic and often have different failure and security boundaries. The compute interconnect is where tightly coupled applications exchange the data needed to make progress.

Choosing an interconnect is important because the compute and communication capabilities of a cluster must remain in balance. If the nodes can produce data faster than the fabric can move it, processors and accelerators will wait. Adapters, switches, optics, cables, software, and operational expertise can also form a significant part of the cost of a cluster. The design problem is to buy enough communication capacity for the workload, with a practical margin for growth.

The names and link speeds have changed considerably since this chapter was first written, but the important questions have not. We still need to examine latency, bandwidth, topology, host overhead, concurrency, reliability, and cost. Modern systems add several wrinkles: remote direct memory access (RDMA), GPU-to-NIC data paths, collective communication, and more demanding congestion control. These additions make careful measurement more important, not less.

Interconnect basics

Figure 3.1 follows a message from a process on node A to a process on node B. We have simplified the path in order to expose the decisions that matter. The sending process calls a parallel programming library such as MPI. The library selects a transport and arranges for the message to reach the network interface. On the receiving side, the corresponding layers deliver the message to the destination process.

At the physical level, bytes move from memory or accelerator memory to an adapter, across one or more links and switches, into the receiving adapter, and finally into the memory used by the destination process. Some modern paths permit the adapter to access registered user memory directly. Other paths stage data through the kernel or host memory. The difference affects latency, host CPU use, and the possibility of overlapping communication with computation.

The workload should determine which properties of this path deserve the most attention. A parameter sweep may exchange little data and care mainly about cost and reliability. A tightly coupled solver may be sensitive to every small-message delay. Distributed training may be limited by sustained collective bandwidth, while a mixture-of-experts model can put severe all-to-all pressure on the fabric. Even within one cluster, no single number describes all of these cases.

Latency

Latency is the time required for a message to travel from a sending process to a receiving process. A common measurement has two processes repeatedly exchange a small message. If the message completes N round trips in time T, the average one-way time is:

L = T / (2N)

In an MPI test this is usually called half-round-trip latency. Program 3.1 shows the essential ping-pong operation. A real benchmark should include warm-up iterations, use a monotonic high-resolution timer, and report variation as well as the average.

double half_round_trip_latency(int iterations)
{
    char message[SMALL_MESSAGE_SIZE] = {0};
    double start_time, end_time;
    int i, rank, rank_count;
    MPI_Status status;

    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &rank_count);
    if (iterations <= 0 || rank_count != 2)
        return 0.0;

    MPI_Barrier(MPI_COMM_WORLD);
    start_time = MPI_Wtime();

    for (i = 0; i < iterations; i++) {
        if (rank == 0) {
            MPI_Send(message, SMALL_MESSAGE_SIZE, MPI_CHAR,
                     1, 0, MPI_COMM_WORLD);
            MPI_Recv(message, SMALL_MESSAGE_SIZE, MPI_CHAR,
                     1, 0, MPI_COMM_WORLD, &status);
        } else if (rank == 1) {
            MPI_Recv(message, SMALL_MESSAGE_SIZE, MPI_CHAR,
                     0, 0, MPI_COMM_WORLD, &status);
            MPI_Send(message, SMALL_MESSAGE_SIZE, MPI_CHAR,
                     0, 0, MPI_COMM_WORLD);
        }
    }

    end_time = MPI_Wtime();
    return rank == 0
        ? (end_time - start_time) / (2.0 * iterations)
        : 0.0;
}

Measure the software path

Latency should be measured through the same API and software stack the application will use. A physical link can forward bits quickly while the application still sees a long delay. Buffer allocation, memory registration, copies, protocol processing, library choices, and progress mechanisms all contribute to the observed result.

This is one reason RDMA-capable transports are important in current HPC systems. On a configured RDMA path, a user-space library can submit work to the adapter and transfer data to or from registered memory without placing the operating-system kernel in the fast path for every message. That does not make all overhead disappear. Registration, synchronization, queue management, and completion processing still matter, and the result depends on the application library, driver, firmware, and platform configuration.

Latency is a curve, not one number

A small-message ping-pong result is useful, but it is only the beginning of a latency study. Applications send a distribution of message sizes. Protocols may change their behavior after an eager limit, a rendezvous threshold, or a transport-specific boundary. The useful result is therefore a curve over message size, accompanied by percentiles that expose occasional long delays.

Congestion also changes the answer. An idle pair of nodes can produce an excellent number even though simultaneous communication across the cluster creates queueing and tail latency. Synchronized applications are particularly unforgiving: one late rank can hold up an entire collective operation. Test the quiet fabric first, then test it while representative traffic is present.

Switching and distance

Messages that cross more switches may see greater delay than messages between nodes on the same leaf switch. Figure 3.2 shows a simple example. Nodes 1 and 2 have a shorter path between them than nodes placed on opposite sides of the uplink. The difference may be small on a well-designed modern fabric, but it can still matter when it is repeated across a large number of synchronized operations.

Scheduling can use this information. Applications with substantial nearest-neighbor communication may benefit when their ranks are placed on nearby nodes. Jobs that communicate heavily within groups can be mapped so each group stays within a locality whenever capacity permits. Topology-aware placement cannot repair an undersized fabric, but it can prevent avoidable traffic from crossing constrained links.

How switches forward messages

Three forwarding ideas help explain switch latency. With store-and-forward switching, a switch receives and checks a complete packet before forwarding it. With cut-through forwarding, the switch can begin transmitting a packet after it has read enough of the header to choose an output. Wormhole routing works at a finer granularity: a message is divided into flow-control units, or flits, which can occupy several links along a route at once. Cut-through and wormhole techniques are related, but they differ in forwarding and buffering granularity.

Current switches combine forwarding, buffering, virtual lanes or traffic classes, adaptive routing, and congestion-control mechanisms in implementation-specific ways. Cut-through behavior reduces the delay of an unloaded hop, but it does not remove contention. When an output link is busy, data must wait somewhere. Buffer design and congestion behavior therefore matter alongside the nominal forwarding mode.

Bandwidth and fabric capacity

Bandwidth is the amount of data that can move through the interconnect in a unit of time. Link rates are commonly stated in bits per second; application benchmarks usually report payload bytes per second. Encoding, framing, protocol headers, acknowledgements, and software overhead make the useful payload rate lower than the advertised line rate.

As with latency, measure bandwidth through the API used by the application. A large-message point-to-point test shows the best rate available to one communicating pair. It does not show what happens when every node communicates at once, nor does it predict the performance of a collective operation whose algorithm uses the topology in a particular way.

Bisection bandwidth and oversubscription

Bisection bandwidth is the minimum capacity across a cut that divides the endpoints into two equal halves, allowing a difference of one endpoint when the count is odd. Conceptually, place half of the nodes on each side and have pairs exchange data across the division. The worst such cut gives a useful indication of the fabric's ability to support many simultaneous flows.

A non-blocking switch can sustain any one-to-one permutation of traffic among its ports at line rate, subject to its stated duplex and packet-size conditions. A multi-stage fabric described as non-blocking should preserve equivalent full-bisection capacity between its endpoints. An oversubscribed fabric has less upstream capacity than the attached endpoints could demand at once. This lower-cost design works well when communication is sparse or localized; broad, synchronized exchange will expose its bottlenecks.

The distinction becomes important when a cluster outgrows a single switch. In Figure 3.2, the inter-switch links must carry all traffic between the two groups. Multiple uplinks or a leaf-spine topology can add capacity and path diversity. The design should be described by its endpoint speeds, uplink speeds, number of paths, and oversubscription ratio. Calling the fabric "100 Gb/s" or "400 Gb/s" without this topology information says very little about cluster-wide performance.

Path diversity must be usable as well as present. Conventional equal-cost multipath routing usually hashes each flow onto one path, so several large flows can collide on the same link while another link is idle. Adaptive routing reacts to current conditions, while Ultra Ethernet defines finer-grained multipath packet spraying for transports that can tolerate or manage reordering. These mechanisms can improve utilization and tail latency, but their behavior should be tested under the application's actual flow count and message pattern.

Topology must match traffic

Regular topologies are generally easier to route, schedule, grow, and diagnose. Irregular links can be useful when an installation repeatedly runs a known communication pattern, but the scheduler and application mapping must understand that asymmetry. Otherwise the extra capacity may sit unused while traffic follows a different path.

The same caution applies to modern multi-rail systems. Two adapters can improve bandwidth, resilience, or both, but only when the communication library distributes traffic across them and the physical paths avoid a shared bottleneck. Inventory the PCIe placement, NUMA relationship, switch paths, and library configuration before treating two links as twice the usable capacity.

Host overhead and communication progress

Network work competes with application work for processor time, memory bandwidth, cache capacity, and I/O resources. A conventional socket path may involve system calls, kernel protocol processing, and copies. An RDMA adapter can move data directly between registered memory regions and perform much of the transport work in hardware. Linux exposes this capability through the RDMA verbs interface, while libraries such as UCX and MPI select and manage suitable transports for applications.

Terms such as kernel bypass, zero copy, and offload each describe a particular part of the path. A path may bypass the kernel for steady-state data movement while still using it for setup and control. A library may copy small messages intentionally because a copy is faster than registering or coordinating a direct transfer. GPU communication can also fall back to host staging when the platform topology or software configuration does not permit a direct path.

The practical question is whether communication makes progress while the application computes. Some implementations need a CPU thread to poll completion queues. Others offload more progress to the adapter. Pinning, core allocation, interrupt placement, and NUMA locality can change results. Measure application throughput and CPU consumption together rather than optimizing either one in isolation.

Inspecting the path on Linux

Linux provides enough visibility to verify many assumptions before running an application benchmark. The rdma link command shows RDMA devices and ports, while ibv_devinfo reports userspace verbs capabilities. The ibv_rc_pingpong utility can confirm basic reliable-connected transport between two hosts. These checks establish that the device is visible and functional; they do not substitute for an MPI or application-level test.

Use lspci -tv, numactl --hardware, or hwloc-ls to inspect the relationship among CPUs, GPUs, and adapters. Adapter counters from ethtool -S, rdma statistic, or the vendor's fabric manager can reveal retries, discards, congestion notifications, and lane errors. Interrupt affinity and irqbalance policy are also worth recording. Tune them only after a controlled measurement shows that the default placement is interfering with the workload.

Concurrency and message rate

Earlier cluster designs often discussed this property as multithreading: could several processes use an adapter concurrently? The modern question is broader. A node may run many MPI ranks, CPU threads, containers, or GPUs, each maintaining communication contexts and queues. The adapter and software stack must sustain the resulting message rate without excessive serialization.

Small-message workloads can exhaust packet-processing or queue resources long before they reach the link's byte rate. Large numbers of flows can expose head-of-line blocking, poor traffic-class isolation, or uneven hashing across paths. For these workloads, messages per second and tail latency may be more useful than peak gigabytes per second.

Cost, reliability, and operations

The base cost of an interconnect includes adapters, switches, transceivers, cables, and software support. The rest of the bill appears in rack space, power, cooling, spare parts, firmware qualification, monitoring, training, and time spent diagnosing the fabric. A less familiar network may be entirely justified by application performance, but the operational cost should be part of the comparison.

Growth and failure behavior deserve the same attention as initial performance. Adding a leaf switch should not unexpectedly halve the bandwidth available to existing nodes. A failed link should be visible to monitoring, and the fabric should either reroute traffic or fail in a predictable way. Before purchase, ask how routing changes are applied, how congestion is identified, which counters are retained, and how adapter, cable, and switch faults are isolated.

Three interconnect scopes

Modern cluster discussions sometimes use the word fabric for several different networks. Keeping their scopes separate prevents misleading comparisons.

A cluster can contain all three. A fast scale-up link between GPUs does not eliminate the need for an inter-node fabric. Likewise, a high-performance compute fabric does not remove the value of an independent management network that remains available when the compute network is being serviced.

Modern interconnect choices

The comparison starts with the software path and workload, then expands to topology and operations.

Ethernet

Ethernet remains attractive because it is widely understood, supplied by many vendors, and useful for nearly every network role in a data center. For loosely coupled work, modern Ethernet with TCP/IP may be all that is needed. Faster hardware does not, however, erase software overhead or topology limits. A cluster still has to be tested through the socket, MPI, storage, or service interface that will carry production traffic.

This use of commodity network technology is part of the Beowulf lineage of Linux clusters. Commodity did not mean that every component had identical performance. It meant that broadly available hardware and open systems could be assembled, measured, and improved without committing the cluster to a proprietary computer architecture. Ethernet's continuing value follows from that flexibility as much as from its link speed.

RDMA over Converged Ethernet (RoCE) places RDMA transport on Ethernet. It can provide low latency and direct memory access while retaining an Ethernet physical and switching environment. Traditional lossless designs use Priority Flow Control (PFC) to pause a congested traffic class and Explicit Congestion Notification (ECN), often with DCQCN at the endpoints, to reduce the sending rate. PFC requires care because pause propagation and head-of-line blocking can spread congestion. Some current switches also support semi-lossless or ECN-only RoCE modes. The correct choice depends on the complete adapter and switch implementation, traffic mix, buffers, and recovery behavior.

The Ultra Ethernet Consortium is defining another Ethernet-based transport architecture for AI and HPC. Its specification addresses packet delivery, congestion management, security, and APIs as parts of one architecture. Products should still be compared by the exact features and specification version they implement; support for an industry label is less informative than an end-to-end interoperability and workload test.

InfiniBand

InfiniBand was designed as a system-area fabric and has a long history in HPC. It provides RDMA semantics, subnet management, and a software ecosystem used by MPI, storage, and GPU communication libraries. This integration is valuable for tightly coupled clusters, but it does not excuse topology analysis. Port speed, switch radix, oversubscription, routing, host placement, and software configuration still determine the result seen by the application.

An InfiniBand and RDMA-capable Ethernet comparison must cover the complete data path, operational model, vendor options, and measured workload behavior. Existing staff expertise and the surrounding network can reasonably influence the decision.

Libraries and transport selection

Applications should not have to contain a different communication implementation for every adapter. MPI implementations and communication frameworks such as UCX discover available transports and choose among shared memory, RDMA, TCP, and accelerator-aware paths. Automatic selection is convenient, but it also creates a verification requirement: confirm which path was selected. A functioning fallback can hide a missing RDMA device, an incorrect container mount, or a locality problem while delivering much lower performance.

GPU clusters and collective traffic

A GPU cluster brings the same latency and bandwidth principles into a more synchronized environment. Distributed training repeatedly uses collective operations such as all-reduce, all-gather, reduce-scatter, and all-to-all. These operations move data among groups of accelerators according to a collective algorithm. Their performance depends on the slowest relevant path and on how well the algorithm maps to the node and network topology.

Some fabrics can execute part of a collective operation in the network. NVIDIA SHARP, for example, builds aggregation trees that perform supported reductions in switches instead of sending every intermediate result back through CPUs or GPUs. This can reduce transferred data and endpoint work for suitable collectives. It is a specific fabric and software capability, so acceptance tests should compare both the enabled and fallback paths.

Dense data-parallel training often places sustained pressure on collective bandwidth. Mixture-of-experts models can create less regular all-to-all traffic and demand high bisection bandwidth. Distributed inference varies more widely: a service that assigns complete requests to independent replicas may exchange little data, while disaggregated inference or expert parallelism can make latency, message rate, and network predictability central to serving performance.

GPU-direct RDMA paths allow a compatible network adapter to exchange data with GPU memory without staging every transfer through a host buffer. The shorter path can reduce CPU involvement and avoid copies, but it depends on the platform's PCIe topology, device support, drivers, memory registration, IOMMU and access-control configuration, and communication library. Verify the active path on the intended servers rather than inferring it from the presence of capable components.

Rack-scale GPU systems add a scale-up fabric inside the rack and a scale-out fabric between racks. Our AMD Helios and NVIDIA Vera Rubin NVL72 comparison separates those networks, including their switches, NICs, and infrastructure processors. Appendix 1 examines how the same interconnect choices affect distributed inference.

Gauging interconnect requirements

Begin with the communication behavior of the application. Blocking remote operations and frequent small control messages usually emphasize latency. Streaming exchanges emphasize bandwidth. Global collectives expose bisection limits and congestion. A workload that runs mostly within one node may care more about scale-up links and memory placement than the inter-node network.

Application placement can reduce unnecessary use of constrained links. If most jobs occupy only a few nodes, the scheduler can keep them within a leaf switch or another close locality. This can make a moderately oversubscribed fabric economical. It should be an intentional scheduling policy, however, not an assumption used to excuse a design whose topology has never been measured.

Test the fabric before accepting it

A useful acceptance test proceeds from simple components to the complete application. Each stage answers a different question:

1. Record the intended path. Document adapter and switch models, firmware, link rates, PCIe and NUMA placement, routes, MTU, traffic classes, and the library versions inside the production environment.
2. Verify transport selection. Confirm that MPI, UCX, NCCL, storage, or the serving framework uses the expected devices and transport rather than a working but slower fallback.
3. Establish pairwise behavior. Measure latency, message rate, and unidirectional and bidirectional bandwidth over a range of message sizes, first within a switch and then across the longest normal path.
4. Load the topology. Run many communicating pairs across leaf and spine boundaries. Compare aggregate throughput and tail latency with the pairwise baseline.
5. Measure collectives. Test the operations and process counts used by the application. For GPU systems, include device buffers and the same rank-to-GPU mapping planned for production.
6. Run the application. Measure useful work, not only network counters. Include representative data loading, checkpointing, and service traffic if those networks or I/O paths are shared.
7. Exercise failure and recovery. Confirm that monitoring identifies bad links and that routing, jobs, and management tools behave as expected when a link or switch path is unavailable.

The OSU Micro-Benchmarks provide established MPI point-to-point, collective, one-sided, and accelerator-buffer tests. NVIDIA's NCCL tests measure the correctness and performance of GPU collectives and can run across nodes using MPI. These are useful baselines, but neither replaces the application test. A fabric can perform well on all-reduce and poorly on an all-to-all workload, or produce high aggregate bandwidth while leaving a latency-sensitive solver waiting.

References and further reading

1. Linux kernel documentation: Userspace verbs access
2. Linux RDMA core userspace libraries and tools
3. Open MPI documentation: TCP networking
4. OpenUCX FAQ: transports, devices, and protocol selection
5. InfiniBand Trade Association
6. Ultra Ethernet Consortium specification history
7. NVIDIA GPUDirect RDMA documentation
8. NVIDIA SHARP documentation
9. OSU Micro-Benchmarks
10. NVIDIA NCCL tests