KV Cache Transfer Over RDMA Between Prefill and Decode Nodes

Moving KV cache tensors between prefill and decode nodes makes RDMA transfer engineering the load-bearing problem of disaggregated inference. GPUDirect RDMA removes host CPUs and system RAM from the tensor data path, but production performance still depends on buffer registration, transfer direction, aggregate bandwidth, and cache placement. A fast fabric cannot compensate for a protocol that serializes transfers or repeatedly moves state already held by decode.

Why prefill and decode cannibalize each other when colocated

Prefill and decode demand different things from the same GPU. Prefill saturates Tensor Cores through large GEMM operations, with arithmetic intensity that grows with input length. Decode instead streams model weights and the full KV cache from HBM to SRAM at each token step, driving arithmetic intensity toward single-digit levels. The interference runs in both directions. Long prefills block active decode streams and spike inter-token latency, while active decode workloads delay new prefills.

AMD documents unpredictable inter-token latency spikes on an 8-GPU MI300X node in its MORI-IO vLLM work, even when HBM is fully utilized and compute appears balanced. Prefill needs transient memory and maximal compute density, whereas decode needs persistent HBM capacity and maximal memory bandwidth. No single configuration serves both optimally, making spatial separation the clean solution to their competing resource requirements.

What disaggregation actually means for the data path

Separate GPU pools stop that fight for resources, but they need the network to move data between them. Prefill produces KV tensors, and decode consumes them before generation can begin. Alibaba Cloud’s FlowKV measured KV transfer at almost a quarter of total request latency in a baseline disaggregated system, making it the dominant non-compute cost.

GPU HBM holds active context at sub-microsecond latency, with capacity ranging from tens to approximately 288 GB per GPU. CPU DRAM holds warm evicted blocks, while local NVMe provides multi-TB capacity per node at tens to hundreds of microseconds for cold blocks. Object storage holds archived sessions and shared prefix caches. These tiers trade access latency for capacity, but none removes the immediate requirement to deliver usable KV state to decode.

Why RDMA is the required transport and how it removes CPU bottlenecks

Kernel-bypass, zero-copy RDMA WRITE lets prefill place KV tensors directly into pre-allocated memory on decode without CPU-side serialization. Kernel bypass keeps the host networking stack out of the transfer path, while zero-copy skips the copies made along the way. GPUDirect RDMA extends that direct path into GPU HBM. The NIC can read and write GPU memory without staging tensors through system RAM, keeping host copies from stalling the prefill compute pipeline.

NIXL delivers only a small fraction of its RDMA throughput when it falls back to TCP on the same hardware, directly increasing time-to-first-token. Shorter contexts can work with lower-speed Ethernet, while longer contexts need 100GbE or above. For 32K+ token contexts or latency-sensitive serving, InfiniBand or RoCE RDMA is the recommended transport. InfiniBand NDR provides the production same-datacenter benchmark, with sub-microsecond latency for moderate KV cache sizes.

How NIXL abstracts and implements RDMA KV transfer

NVIDIA Inference Transfer Library, or NIXL, provides a unified asynchronous API for moving KV tensors across GPU HBM, local NVMe, and remote storage. Open-sourced at GTC 2025, it lets inference frameworks request transfers without implementing each transport themselves. A framework can use the same transfer abstraction while experiencing very different latency and throughput underneath it.

NIXL supports UCX for RDMA/InfiniBand, RoCE, and TCP, together with Libfabric and storage backends including GDS, POSIX, OBJ for S3-compatible object storage, and Azure Blob. Depending on its backend choice, a request therefore takes the intended fast path or a much slower fallback. Zero-copy access also requires GPU HBM regions to be registered with the NIC. That setup cost must be amortized across transfers, so the library’s asynchronous interface does not eliminate the need for deliberate memory management.

Push versus pull: how transfer direction determines system latency

vLLM reports approximately 49 GB/s for push mode using NIXL WRITE and approximately 0.5 GB/s for pull mode using NIXL READ on the same hardware with a long prompt. In the documented pull path, decode waits for prefill to finish before requesting data, then pulls small blocks that underutilize NIC bandwidth.

The proxy dispatches to prefill and decode simultaneously, and prefill writes each layer’s KV data into pre-allocated decode pages as computation progresses. Transfer overlaps with compute. Decode can start generating as soon as the last prefill layer finishes, with no wait for a separate transfer phase.

By contrast, pull mode dispatches serially, adding a full round-trip of serialization overhead before decode retrieves the blocks. RDMA READ itself is not inherently responsible for that scheduling dependency. FlowKV independently identifies block-wise calls and discontinuous KV allocation as primary causes of baseline transfer latency. Its transfer coalescing cuts average transfer latency sharply, so submission granularity and memory layout need to be weighed alongside push versus pull.

Memory registration, buffer pre-allocation, and the hidden costs of RDMA setup

Push mode only overlaps work when its destination already exists. Decode must reserve physical HBM pages and register them with the NIC before transfer begins. That requirement reaches back into scheduling. The scheduler must predict or negotiate the incoming block count before prefill starts computing, or prefill must wait for decode to signal readiness and lose the intended parallelism.

Registration itself is not a kernel-free operation. Registering a new memory region needs a kernel call even when the RDMA transfers that follow bypass the kernel. Production systems pin and reuse registration handles across requests so that fast transfers do not repeatedly pay for slow preparation.

Connections introduce a separate scaling cost because every RDMA Queue Pair consumes NIC resources. With many prefill workers communicating with many decode workers, the QP count grows as the product of the pool sizes. Fan-out becomes an infrastructure constraint in its own right. Huawei CloudMatrix-Infer uses model-aware connection grouping to balance communication and a dedicated RDMA plane to isolate KV transfers from other cluster traffic.

How context length and model scale amplify every transfer constraint

Long context can make KV transfer take longer than prefill computation. At that point, the usual assumption reverses. Transfer is no longer a small surcharge on compute, and faster prefill does not remove the wait for state to reach decode. Model scale and context length consequently have to be evaluated against both the transfer path and GPU capacity.

Repeated transfers make the constraint more severe under conversational load. PPD disaggregation found that multi-turn KV movement can saturate network bandwidth and cause service degradation, extending the impact beyond individual request latency. Once concurrent transfers saturate the fabric, availability can suffer, making byte reduction as important as additional link bandwidth.

Lossless compression and format-aware encoding as a bandwidth multiplier

General-purpose lossless codecs often target offline weight compression, execute on CPUs, or use variable-length coding too slow for GPU prefill. Those designs do not fit an in-path KV transfer. Compression must keep pace with tensor production. Otherwise, less data goes over the network, but it has to wait longer before it can be sent.

Guo and Joshi’s SplitZip, introduced in May 2026, exploits redundancy in the floating-point exponents of KV activations. It encodes frequent exponent values with fixed-length codes and sends rare exponents through a sparse escape stream. Its compression and decompression throughput substantially exceeds a single NIC’s RDMA bandwidth, making GPU-side encoding fast enough to operate in the transfer path.

Although the measurements do not come from a shared benchmark, relative to the approximately 49 GB/s push-mode result, the governing criterion is whether compression and decompression can outpace the link on the actual deployment. SplitZip also provides additional compression over native FP8 E5M2 encoding, showing that a compact source format does not exhaust the opportunity. For BF16 and FP8 KV tensors, format-aware compression turns a bandwidth limit into a GPU compute trade-off that needs validation on the specific model and hardware.

The multi-turn problem: how the one-way P→D protocol breaks under conversation workloads

Canonical disaggregation transfers state strictly from prefill to decode, with no reverse channel. Generated response-token KV state consequently remains unreachable by prefill. This breaks reuse on subsequent conversational turns. Prefill must recompute the conversation history, including prior responses and the new prompt, then transfer it back to decode even though decode already holds most of that state.

PPD’s March 9, 2026 preprint documents recomputation accounting for up to 99% of multi-turn prefill cost in real chat workloads. Its proposed alternative routes append-prefill locally to decode, processing only new input tokens while reusing cached KV state. This avoids recomputation and the incremental P→D transfer. PPD reports much lower Turn 2+ time-to-first-token while keeping time-per-output-token competitive.

PPD finds that no fixed routing strategy satisfies all SLOs simultaneously, so decisions depend on load, SLO weights, and queue state. vLLM v0.21.0, released May 15, 2026, ships a bidirectional KV transfer protocol for disaggregated deployments. That protocol and the concurrent PPD proposal address the same architectural gap through distinct approaches that extend disaggregation.

What hardware-level integration changes about the transfer architecture

Dedicated silicon is starting to take over work on cache placement metadata, buffer management, and registration that host software used to handle. NVIDIA’s Inference Context Memory Storage Platform, announced at CES 2026, extends GPU KV cache into NVMe storage using BlueField-4. The DPU was announced at GTC Washington D.C. in October 2025, with early availability expected in 2026. Its planned hardware-accelerated cache placement targets metadata overhead and data movement while providing isolated access from GPU nodes, with up to 800 Gbps throughput.

NVIDIA Dynamo 1.0 went GA on March 12, 2026, coordinating disaggregated inference and KV block routing through NIXL transfers. EFAv3, using Nitro v5 and GPUDirect RDMA, also allows NIXL to move blocks between GPUs on separate nodes without host-RAM staging. These integrations reduce explicit transfer plumbing for serving frameworks. They do not make cache placement or network provisioning irrelevant.

CloudMatrix-Infer combines a separate RDMA plane with asynchronous scheduling so decode execution is not blocked by prefill scheduling or transfer, plus model-aware connection grouping. Storage integration extends the addressable cache beyond HBM. NVIDIA’s DOCA framework, Dynamo’s KV cache offload engine, and NIXL treat NVMe-oF as a storage backend, allowing KV blocks to move over RDMA to remote NVMe. The storage fabric thus gives access to the cache as well as a place to keep archived data.

What a production RDMA KV transfer stack requires from the infrastructure team

For 32K+ context or latency-sensitive serving, teams need InfiniBand or RoCE with GPUDirect RDMA support, provisioned for aggregate traffic across concurrent prefill-decode pairs. Peak per-node bandwidth is not sufficient. Decode regions must be pre-allocated and pre-registered before prefill begins, registration handles reused, and QP fan-out bounded through pooling or connection multiplexing.

For latency-sensitive serving, push WRITE should be the default. Pull READ can suit throughput-oriented or asynchronous work where serial dispatch is tolerable. Format-aware BF16 and FP8 compression should be benchmarked against the NIC rate on the target configuration to establish whether it helps. Multi-turn and agentic deployments should also evaluate dynamic append-prefill routing to decode nodes, where state reuse can eliminate both computation and transfer.

The storage layer matters because evicted KV blocks still need a way back into memory where serving can use them. RDMA access to remote NVMe expands capacity, but does not give that tier HBM’s access characteristics. KV transfer latency, NIC utilization, HBM registration state, and QP counts should be first-class metrics, because transfer bottlenecks often appear as time-to-first-token variance even when latency is not consistently high. That distinction helps teams avoid misdiagnosing a fabric or registration problem as a model or scheduler problem.