How RDMA Bypasses the CPU to Sustain All-Reduce Throughput in GPU Cluster Training
Bypassing the CPU with RDMA and GPUDirect lets networks, not software, limit training speed.
RDMA sustains all-reduce throughput in GPU cluster training by bypassing the kernel, eliminating intermediate memory copies, and moving transport processing onto the NIC. GPUDirect RDMA extends that path into GPU memory, making the interconnect fabric rather than host software the determinant of collective performance. For high-throughput networked storage, the same bypass discipline extends through GPUDirect Storage, keeping CPU DRAM staging out of the storage-to-GPU path.
Why all-reduce is the communication bottleneck that determines training throughput
Distributed SGD requires every participating GPU to synchronize gradients after each forward-backward pass before weights can be updated. Even one late packet can hold up the collective. The synchronization barrier is structural because ring-allreduce passes gradient chunks through every rank before any rank holds the full reduced result. Its reduce-scatter and all-gather phases send inter-node messages across the fabric at every step, so communication latency and CPU overhead accumulate directly in training step time.
Adams & Bienz (IPDPSW 2026) characterize large all-reduce operations as dominating the cost of LLM training on heterogeneous multi-GPU architectures. Faster compute can make that imbalance more pronounced. On nodes without high-bandwidth switched interconnects, all-reduce can take up a large share of step time during full fine-tuning, and that share grows as models get bigger. Training throughput therefore depends on the data path that every gradient chunk must traverse as well as the amount of computation available.
Why the traditional TCP/IP software path cannot keep up with modern link speeds
The traditional TCP/IP software path routes application data through kernel socket buffers before it reaches the NIC, then reverses that process on receive. That copies the data four separate times across the two endpoints, and each copy uses CPU cycles and memory bandwidth. At hundreds of gigabits per second, the work of responding to interrupts and copying memory can overwhelm the host processor because the NIC is no longer slow compared with the CPU.
This contention extends beyond the collective itself. Packet processing competes with data loading, preprocessing, and checkpointing, all of which must proceed while GPUs compute. The software still has to do that work even with a faster physical link. Instead, the host can become the bottleneck feeding the link while also losing the CPU capacity needed to keep the training pipeline supplied.
The three mechanisms RDMA uses to remove the CPU from the data path
RDMA replaces that software-mediated path with kernel bypass, zero-copy transfer, and NIC-offloaded transport. For kernel bypass, applications post work requests directly to the NIC through user-space verbs libraries. The CPU still initiates the operation. But the transfer itself does not need the CPU to process the payload through the operating system’s networking stack.
Zero-copy needs pinned, page-locked user-space buffers that are registered with the host channel adapter, or HCA. The adapter reads from and writes to those buffers directly. On receive, it puts the arriving data into the destination application’s memory without a copy in kernel memory along the way or a host CPU wake-up to put it there.
NIC-offloaded transport completes the separation by placing reliability, sequencing, and retransmission logic in the HCA’s firmware and ASICs. Individual packet events therefore do not require CPU interrupts to run that logic. Together, these mechanisms free cores that would otherwise act as packet-processing engines for data pipeline work that overlaps with the backward pass.
How GPUDirect RDMA extends zero-copy discipline all the way into accelerator memory
Host-memory RDMA alone does not remove staging from GPU communication. Without GPUDirect RDMA, gradient tensors still travel from GPU HBM over PCIe into CPU DRAM, then cross PCIe again to reach the NIC. Host staging preserves a costly intermediate step. Although the network transfer bypasses the kernel, the GPU-to-network path still consumes CPU DRAM bandwidth and adds PCIe round-trips.
GPUDirect RDMA gives compatible NICs a direct peer-to-peer mapping into GPU memory over PCIe. The resulting path runs from GPU HBM through PCIe to the NIC, across the fabric, and through the remote NIC and PCIe into destination GPU HBM. The interconnect fabric, rather than host software or CPU DRAM bandwidth, becomes the determinant of collective communication performance. Its topology, bandwidth, and behavior under congestion now govern how nodes synchronize with one another.
The two-tier bandwidth architecture of a modern GPU cluster and where RDMA operates in each
The scale-up tier connects GPUs through NVLink and NVSwitch. NVLink 5.0 delivers 1.8 TB/s per GPU, while fourth-generation NVSwitch, or NVSwitch4, provides 130 TB/s of aggregate bandwidth inside an NVL72 rack. The 2026 Rubin generation NVSwitch5 extends aggregate fabric bandwidth to over 260 TB/s. Within NVL72, all-reduce takes only a small part of the time for each step. NVSwitch4 also includes SHARPv3 with FP8 and FP16 in-network reduction, improving collective efficiency for lower-precision data.
The scale-out tier connects nodes through InfiniBand or RoCEv2. InfiniBand NDR offers 400 Gb/s per port, while Quantum-3 at 800 Gb/s XDR is shipping into frontier deployments. RoCEv2 can reach much the same rates per port over Ethernet, but it has different needs for reliability. RDMA and GPUDirect RDMA operate in the scale-out tier, where inter-node bandwidth is roughly 30× lower per GPU than intra-node NVLink bandwidth, making host overhead especially expensive at the boundary between the two tiers.
InfiniBand versus RoCEv2: what each fabric requires to sustain lossless all-reduce at scale
InfiniBand and RoCEv2 both deliver RDMA semantics, but they impose different engineering obligations. InfiniBand uses a dedicated network with hardware-enforced reliability and lossless operation by design. Quantum-2 has NDR 400 Gb/s with SHARP in-network reductions, while Quantum-3 moves up to XDR 800 Gb/s. Most of the commitment comes when choosing the infrastructure upfront. Cost and a dedicated fabric buy consistent operation without operator-configured loss avoidance.
RoCEv2 instead runs RDMA over standard Ethernet and requires a lossless fabric configuration. Priority Flow Control, or PFC, prevents drops, while Explicit Congestion Notification, or ECN, works with DCQCN to manage queue buildup. Correct tuning becomes an ongoing operational responsibility. Under synchronized all-reduce bursts, poor PFC and ECN settings can produce head-of-line blocking and retransmission storms, undermining the latency benefit of CPU bypass.
A well-configured RoCEv2 fabric can approach InfiniBand’s all-reduce bandwidth, so Ethernet itself is not the decisive limitation. Scazzariello et al.’s SPILLWAY paper (arXiv, May 2026) reports that Meta disabled DCQCN for Llama 4 training and used library-level receiver-driven flow control instead.
In-network compute: how SHARP moves the reduction itself into the switch silicon
RDMA removes CPU involvement from transport execution, but the reduction arithmetic can still occupy the GPUs. SHARP, the Scalable Hierarchical Aggregation and Reduction Protocol, moves collective operations into switch silicon, so the network performs arithmetic as well as transport. By running all-reduce, reduce, and broadcast operations in the fabric, it can finish the reduction before the data reaches the destination GPU.
NVL72 measurements illustrate the effect of changing that execution location. A ring all-reduce of a 64 MB FP16 tensor across 72 GPUs achieves approximately 1.6 TB/s of effective bus bandwidth. SHARPv3 achieves approximately 3.0 TB/s on the same hardware. The improvement comes from relocating work into the switch. Rather than merely accelerating endpoint communication, in-network compute changes which part of the system performs the collective.
How cross-datacenter all-reduce exposes RDMA's assumptions about latency and buffer depth
RDMA’s tightly coupled, intra-datacenter design depends on short feedback loops and shallow buffers. Cross-datacenter latency grows roughly 5 ms per 1,000 km, replacing microsecond feedback with multi-millisecond delays. Distance changes the behavior of congestion recovery. The SPILLWAY paper describes geo-distributed training at Google and OpenAI and xAI’s expansion of Colossus 2 across nearby sites, placing this mismatch within the operating conditions of large training systems.
The collision occurs when cross-datacenter AllReduce reaches a destination leaf switch during bursty intra-datacenter AllToAll traffic. The local burst monopolizes the destination port, while remote packets accumulate and overflow shallow buffers within sub-millisecond timescales. Congestion feedback arrives too late to prevent loss. A dropped packet can then require retransmission at cross-datacenter RTT scale, turning a brief collision into a stall lasting tens of milliseconds.
SPILLWAY buffers packets that would otherwise be dropped in switch-disaggregated memory at the destination datacenter, then drains them after the burst subsides. It requires no changes to end hosts or training frameworks and reduces iteration time in simulation. The adaptation stays inside the network itself. Meta’s alternative uses deep HBM-backed switch buffers with library-level flow control, accepting costly worst-case provisioning for collisions that occur only intermittently.
Heterogeneous clusters and simulator gaps: why RDMA tuning cannot assume homogeneous hardware
Mixed GPU generations, interconnect architectures, and NIC speeds create uneven RDMA lanes that a homogeneous ring cannot treat as equivalent. Kumar et al. (Xsim, arXiv June 2026) describe transitional heterogeneous deployments that can persist for months or years because compute and interconnect bandwidth advance at different rates. A slower lane becomes a collective straggler. Standard simulators including SimAI, Multiverse, ATLAHS, and Meta’s Arcadia assume homogeneous compute and networking, leaving these tail-latency effects outside their model.
Xsim supports non-uniform workload partitioning, customized ring construction, and chunk partitioning adapted to uneven bandwidth. It validates within roughly 5% error across most heterogeneous data-parallel and tensor-parallel configurations. CPU bypass does not make unequal links equal. Tuning therefore requires algorithms and tools that represent the actual hardware, even as NIC offload preserves CPU capacity for the surrounding data pipeline.
How the same RDMA fabric becomes the critical path for KV cache movement in LLM inference
Disaggregated LLM inference uses the same transport machinery to move KV cache tensors between prefill and decode GPUs. On an H100 serving Llama 3.1 at 128K context, a single user’s KV cache occupies roughly 40 GB. HBM capacity quickly constrains concurrent serving at that size. Cold blocks must move to CPU DRAM or NVMe and return on demand over RDMA, making cache movement a latency-sensitive operation between machines.
Mooncake’s cache-centric disaggregated architecture, introduced in mid-2024 and awarded the FAST’25 best paper, treats the cache as shared system state. RDMA moves that state on the decode critical path. GPUDirect Storage extends the bypass discipline to NVMe by transferring data from SSDs into GPU memory without CPU DRAM staging. Local NVMe achieves 12–25 GB/s per server, while all-flash NFS over RDMA achieves 50–200 GB/s cluster-wide. Those figures describe different aggregation scopes, but both establish storage bandwidth as part of a viable tiered cache architecture.
What a real RDMA-optimized training run looks like end to end
Oracle + AMD MLPerf Training 6.0 (2026) pairs a RoCEv2 RDMA cluster network with RCCL tuning for multi-node synchronization. FLUX.1, an 11.9-billion-parameter diffusion transformer, reached convergence at 512 GPUs, with multiple runs completing within a narrow time window. The result anchors the mechanism in a complete workload.
NVIDIA Blackwell Ultra MLPerf Training v5.1 (2025) shows another layer of optimization by changing attention-block batched-matrix-multiply inputs from BF16 to FP8. That change made attention-kernel performance up to 1.3× better while still meeting the accuracy requirements. Collective traffic depends on which tensors are sent, so compute precision and communication volume are separate things to check when judging the full training stack.
An FPGA-based AI smart NIC study addresses transmitted volume directly through BFP compression of gradient tensors before they enter the RDMA fabric. It substantially reduced both exposed all-reduce time and total training time relative to its baseline. The optimization spans more than the transport protocol. RDMA and GPUDirect remove host overhead, SHARP relocates reduction arithmetic, and NIC compression lowers link demand, while storage bypass keeps data delivery from rebuilding the CPU bottleneck elsewhere. Storage and NIC-layer compression belong to the same optimization surface as the collective itself.
