RDMA Fundamentals for LLM Infrastructure Engineers

RDMA addresses specific data-movement bottlenecks in LLM infrastructure. Its zero-copy, kernel-bypass model removes CPU staging and kernel-copy overhead from critical transfers, including networked storage reads, KV cache handoffs, and training synchronization. Those gains depend on designing the transport, storage hierarchy, congestion controls, and cache representation to work together.

Why LLM data movement is a different problem from general-purpose networking

LLM inference is constrained by memory and I/O, so additional FLOPS alone cannot resolve its central bottlenecks. KV cache transfers between storage and compute nodes compete with the need to keep generation moving. Disaggregated prefill-to-decode handoffs add another latency-sensitive transfer across node boundaries. Training exposes a related problem through AllReduce gradient synchronization across GPU ranks. These workloads make data movement part of execution itself.

In all these workloads, large, latency-sensitive payloads cross node boundaries often. CPU-mediated transport adds kernel crossings and memory copies to those transfers, with overhead compounding across thousands of concurrent operations. A faster link can still leave CPUs staging data that accelerators need before they can proceed. The mismatch is structural because the transport consumes host resources on the critical path of work that depends on moving accelerator state.

What RDMA's zero-copy, kernel-bypass model actually does — and what it doesn't fix on its own

RDMA allows one server to read or write another server’s memory without involving the remote CPU. One-sided RDMA READ and WRITE operations give it that direct access, while two-sided SEND/RECV operations use both CPUs but still skip the kernel copy path. This removes three costs that recur with CPU-mediated TCP transport, which are per-message CPU work, kernel crossing latency, and memory copying. Message frequency drives per-message overhead, while payload size increases the cost of copying.

ByteDance’s ByteExpress demonstrates why these primitives are necessary but insufficient for production use. Variable-size messages, bursty incast, simultaneous many-to-one communication, and failure recovery remain system problems after RDMA is available. ByteExpress adds an RDMA-aware allocator, zero-copy transfers, QP multiplexing, receive-credit flow control, a software send window, and recovery assisted by remote procedure calls. Hardware does not replace these supporting controls.

ByteExpress’s 2026 analysis found that this combined design reduced P99 model parameter fetch latency by 43.7% relative to TCP in production inference. It also reduced mean fetch latency and total CPU usage. The result belongs to the full implementation. The evidence for RDMA with supporting software is specific to the measured implementation and cluster.

The three RDMA protocol families and their practical trade-offs for AI clusters

InfiniBand, RoCEv2, and iWARP span purpose-built RDMA fabrics and RDMA over commodity Ethernet. InfiniBand provides ultra-low latency of ~0.5 µs MPI/NCCL and purpose-built congestion control, and it is dominant in hyperscale training. It also comes with vendor lock-in and a higher cost per port. Both InfiniBand and RoCEv2 have reached 800G port speeds, so peak port bandwidth alone does not settle the choice.

RoCEv2 makes fabric configuration an explicit part of the deployment. The configuration described here requires PFC on the RDMA priority only, MTU 9000 end-to-end, and RDMA/TCP traffic isolation, including isolation by packet priority markings. Lossless operation and separation from TCP traffic establish the conditions under which low-latency RDMA transfers can remain predictable. The choice also depends on what the deployment can tolerate in operation and how the workload runs, as well as Ethernet compatibility.

Meta’s MetaRoCE redesigns RDMA transport for AI workloads on commodity Ethernet, with a specification, reference software implementation, and compliance test suite being contributed to the Open Compute Project. Its support for packet spraying across multiplane networks differs from standard RoCE, which discourages that approach because it assumes in-order frame delivery. The Ethernet ecosystem is also expanding in hardware. Chelsio’s seventh-generation AI Interconnect Platform, announced July 2026, provides native 400Gb Ethernet with Unified RDMA supporting iWARP and RoCEv2, storage acceleration, and AI infrastructure optimization.

How RDMA fits into the KV cache storage hierarchy and why its role changes across tiers

KV cache is the dominant consumer of GPU HBM in long-context and agentic inference, growing linearly with sequence length and batch size. As capacity runs out, the production hierarchy extends from GPU HBM to CPU DRAM, local NVMe SSD, and networked storage. Each transition adds latency and requires a transport. The fastest tier is also the smallest. Managing the cache means choosing where to put it and which transfer costs generation can bear.

Between GPU nodes in a disaggregated serving cluster, RDMA supplies the fast path needed to move KV blocks without stalling generation. Between NVMe-backed storage and GPU nodes, NVMe-over-Fabrics with RDMA, or NVMe/RoCE, removes CPU staging from the read path. The endpoint changes, but the principle does not. NVMe-oF over RDMA achieves substantially lower latency than iSCSI by avoiding SCSI emulation layers and provides meaningfully higher IOPS per CPU core.

NVIDIA’s Inference Context Memory Storage Platform, announced at CES 2026, standardizes this tiered architecture. Its NIXL protocol supports NVLink, InfiniBand RDMA, RoCE, NVMe-oF, and TCP across the hierarchy. Dynamo operates across HBM, DRAM, direct-attached NVMe, and networked external storage. These tiers need different paths to their data. Their shared design objective is to prevent the transport between tiers from consuming the benefit of cache reuse.

Disaggregated prefill-decode architectures and why RDMA is their load-bearing transport

Prefill-decode disaggregation assigns compute-intensive prefill and memory-bandwidth-intensive decode to different nodes. That separation improves overall throughput only if the KV cache handoff does not become the next bottleneck. The network becomes part of the architecture’s viability. Fast compute phases cannot compensate for a transfer path that leaves the next phase waiting for state.

DualPath, from Peking University, Tsinghua University, and DeepSeek-AI, identifies a precise utilization imbalance in standard disaggregated architectures. Its Figure 1 shows storage NICs on prefill engines becoming bandwidth-saturated while those on decode engines remain idle. All KV cache loading flows through the storage-to-prefill path. DualPath adds a storage-to-decode path, then transfers the loaded KV cache to prefill engines through RDMA over the compute network. This changes which interfaces can serve the storage workload.

DualPath’s optimized path avoids congestion and interference with latency-critical model execution communications, while its global scheduler balances load across prefill and decode engines. RDMA enables more than a faster individual transfer. It supports a topology that uses otherwise idle storage bandwidth rather than concentrating demand on prefill-side NICs. That distinction becomes acute for multi-turn agents, which accumulate context across dozens or hundreds of turns and produce very high KV cache hit rates. Loading cached state can then take up more of the work than fresh computation would.

AllReduce synchronization and the cascading cost of a single stalled GPU

Participating GPUs must synchronize at a barrier before the next iteration can proceed, so a single stalled GPU stalls the cluster. Tail latency governs progress more directly than average latency. With TCP, a dropped packet triggers retransmission that can delay an AllReduce operation across thousands of GPUs. With a barrier at each iteration, transport delays can keep interrupting the training loop.

In the lossless RDMA configuration described here, PFC and ECN work together to maintain zero-loss delivery while managing congestion without drops. With no packet loss, there is no delay from retransmission. However, PFC pause propagation can itself create cluster-wide stalls, so a lossless design still needs effective traffic control.

ByteExpress’s 2026 analysis found P99 model parameter push latency reductions of up to 74.0% relative to TCP in training, with no PFC pauses observed. Its relevance is the combination of improved tail latency and the absence of pauses in the measured workload. The structural goal for synchronization is similarly twofold, avoiding CPU-mediated delay without introducing a different mechanism that holds back participating GPUs.

KV cache compression as a complementary lever that changes what RDMA must transport

RDMA removes transport overhead but does not make wire bandwidth unlimited. Once the fabric is efficient, the size of the KV cache determines how much data still has to cross it. Quantization reduces bit-width, while eviction drops tokens entirely. These choices affect the same transfer budget. Treating them as separate choices misses the many tokens that can be kept at less than full precision.

RDKV treats eviction and quantization as parts of a joint bit-allocation problem. It achieves substantial decode speedup and peak memory reduction at long context lengths compared with full-cache FlashAttention-2 decoding while maintaining comparable performance. For infrastructure, the important consequence is a smaller cache to transfer. The compression decision therefore changes both demand on the RDMA fabric and memory consumption on an individual GPU.

Leyline, from Erlangen NHR, shows that agentic workloads also need policy-driven cache mutation, including splice operations that replace stale spans mid-conversation. Without that primitive, production systems have to do full re-prefill on every edit. An edit can then force the system to repeat costly cache-transfer work. Cache representation, mutation support, and transport design must be chosen together because they all affect how much state has to be rebuilt and moved.

What lossless versus lossy compression means when KV cache crosses the network boundary

Lossy compression at a prefill-to-decode handoff affects more than the transfer itself. Errors in the cache sent over the network affect every decode step that follows for that request. Judging a smaller payload takes more than counting the bandwidth saved, because the receiver keeps computing from the changed state.

Format-aware lossless compression uses the structure of BF16/FP8 to cut wire bandwidth while keeping model outputs the same. It does not introduce approximation into the transferred cache. In the boundary decision described here, that makes it the compression option that can be applied without per-deployment model-quality validation or compression-induced error accumulation across context turns. Joint eviction and quantization can reduce the cache before transfer, while lossless encoding reduces the wire representation without adding another source of error.

Putting it together — the infrastructure decisions that determine whether RDMA delivers its theoretical gains

Choose the protocol around the workload’s binding constraint. InfiniBand fits training clusters where barrier-latency sensitivity justifies per-port cost, while RoCEv2 or MetaRoCE fits Ethernet deployments where economics and operational flexibility weigh more heavily. For the RoCEv2 configuration described here, PFC on the RDMA priority only, MTU 9000 end-to-end, and RDMA/TCP isolation precede performance tuning. ByteExpress shows why allocation, flow control, send management, and recovery must then complete the hardware path.

DualPath’s storage-to-decode route addresses an imbalance that a faster prefill-side transport alone would not resolve. Joint eviction and quantization, paired with format-aware lossless encoding at the wire boundary, cuts how much data has to move and can lower the bandwidth requirements that drive hardware cost.

Storage is where these decisions converge into one system. NVMe-oF over RDMA, tiered KV offload architectures such as ICMSP, and compression pipelines govern the same journey from stored state to active computation. Treating them as independent defaults risks restoring bottlenecks elsewhere in that journey.