Distributed KV Cache Across Inference Nodes

Distributed KV Cache Across Inference Nodes. Distributing KV cache across inference nodes is a multi-layered infrastructure problem, spanning memory hierarchies, network bandwidth, routing, and eviction policy, that requires understanding the specific failure modes at each boundary before any architecture can be evaluated.

KV cache memory as the binding constraint before distribution

Every token that passes through a transformer during inference generates a key and value tensor, and those tensors accumulate in GPU HBM for as long as the session lasts. That's the KV cache: a running ledger of attention state that lets the model avoid recomputing context from scratch on every new token. The trouble is that the ledger doesn't grow along one axis. A 70B model with 8K context requires roughly 20 GB of KV cache, while a batch of 32 requests demands roughly 640 GB (more than the model weights themselves).

On a 70-billion-parameter model with an 8,000-token context, a single request needs around 20 GB of cache. Putting 32 of those requests in flight together brings the total to roughly 640 GB, more memory than the model's own weights consume. That inversion, cache outweighing weights, is the reason KV cache stopped being a footnote in systems design and became the main event.

Before the vLLM team addressed this directly, most of that memory was simply wasted. Fragmentation and over-allocation burned through 60 to 80 percent of whatever was set aside for cache https://dl.acm.org/doi/10.1145/3600006.3613165. That's a genuinely large win, and it solved the problem of waste inside a single GPU. It did nothing, however, for the problem of scarcity across many GPUs, which is a different question entirely, and it's the one this piece exists to answer https://arxiv.org/pdf/2607.28150.

The memory hierarchy an engineer is distributing across

Where KV cache actually lives in a production system depends on how much you're willing to spend, and how slow you can tolerate being. Four tiers have become the working model. GPU HBM is the fastest memory available, the most expensive per gigabyte, and where the actively-used cache blocks have to sit for any speed. Below it, CPU DRAM offers a warm overflow zone, quicker to reach than disk but still bounded in size and still not cheap once you're provisioning it at fleet scale. Below that is local NVMe SSD, cold or persistent storage with orders of magnitude more room than either of the tiers above it, and fast enough in practice that fetching from it can beat recomputing the cache outright for a lot of workloads.

The progression from HBM to CPU DRAM to local NVMe follows a shape any systems engineer would recognize instantly: each step down is slower but larger, and only one inference worker can see the cache. The fourth tier is where that pattern stops holding. The fourth tier is a structural break, not a quantitative step down the hierarchy. For the first time the cache is no longer a private resource glued to one GPU but something the whole fleet can share, which is precisely what makes distributed KV cache a different discipline from ordinary memory offloading.

The payoff is visible concretely in workloads with repeated long context: retrieval-augmented generation, document QA, multi-turn chat. Tiering that repeated context out to NVMe can cut per-node GPU memory requirements by 30 to 50 percent, with a time-to-first-token cost that's small enough to live with https://www.spheron.network/blog/nvme-kv-cache-offloading-llm-inference/. That tradeoff, giving up a little latency to buy back a lot of GPU headroom, is the basic currency of every decision described in the rest of this piece. Networked external storage forms a fleet-wide shared pool (S3-compatible object stores, RDMA-attached NVMe-oF, or dedicated KV stores), visible to every worker in the cluster.

Prefill–decode disaggregation and the restructuring of where KV cache lives and who owns it

Prefill, the phase where the model chews through the input prompt, is bound by raw compute and sets the time-to-first-token. Decode, the phase that generates output one token at a time, is bound by memory bandwidth and sets the time-per-output-token. Bolt both phases onto the same GPU and you're forced to compromise: tune for prefill's compute appetite and decode suffers, tune for decode's bandwidth needs and prefill suffers.

The fix that's become standard is to stop asking one GPU pool to do both jobs. Running both phases on the same GPU forces a tradeoff between optimizing for each; disaggregating prefill and decode onto separate GPU pools allows each to be provisioned for its own bottleneck. But that split has a cost the diagrams don't always show clearly: the KV cache generated during prefill on machine A has to physically move to the decode workers on machine B before a single output token can be produced. That transfer isn't background housekeeping. It sits directly on the critical path of every request the system handles, and how fast it happens decides whether disaggregation was worth doing at all.

The network bandwidth constraint that determines whether disaggregation is faster

The arithmetic gets unforgiving. For a large model with 80 layers, 8 KV heads, 128 dimensions per head, and FP16 precision, each token costs 327,680 bytes of cache https://jarvislabs.ai/blog/llm-optimization-disaggregated-prefill-decode. That works out to a transfer payload of 1.34 GB for a 4,000-token prompt https://jarvislabs.ai/blog/llm-optimization-disaggregated-prefill-decode. That's the amount of data that has to cross a wire before decode can start.

How fast that wire is turns out to be the whole question. Outside that domain, a cross-node RDMA link running at 400 Gbps drops effective bandwidth to roughly 50 GB/s. Over 400 Gbps RDMA (50 GB/s effective, cross-node), the same 2.6 GB transfer takes 52 milliseconds, added directly to the request's time-to-first-token https://arxiv.org/html/2607.28633v1. For a system targeting a 200-millisecond TTFT budget, that's 26 percent of the entire budget spent moving cache before a single output token has been generated https://arxiv.org/html/2607.28633v1.

And that's the good case, the one running on data-center InfiniBand. Most rented cloud GPU instances don't come close. Separate published work on Alibaba cloud infrastructure backs this up from another angle: with L20 GPUs and 25 Gbps RDMA, transferring a 48,000-token prompt's cache takes 6.5 times longer than the prefill computation itself, and cache transfer alone can eat up to 42.2 percent of total job completion time https://arxiv.org/pdf/2607.28150. Bandwidth, not compute, is what decides whether disaggregation pays off. The interconnect bandwidth hierarchy in a DGX H100 cluster illustrates the network bandwidth constraint that decides whether disaggregation is actually faster. NVLink 4.0 delivers 900 GB/s within the same NVLink domain only, so a 2.6 GB transfer takes 2.9 ms. Most cloud GPU instances offer only limited inter-node bandwidth (AWS L4/L40S instances and Google Cloud A100 instances provide only 10–35 Gbps), far below data-center InfiniBand, making full-cache transfer designs impractical for self-hosted deployments on rented instances.

KV-aware routing: how the system decides which node should handle a request

Routing is a non-question when there's one worker: every request goes to the only place it can go. Adding more workers makes routing decide whether a cache hit is possible at all.

Blind load-balancing means an identical prompt prefix lands on a different worker each time it arrives, forcing a full recompute or a remote fetch on every single request, which defeats the purpose of caching. Consistent-hashing schemes solve the common case by hashing a prefix to a specific worker and keeping it there, which holds up fine under evenly distributed traffic. Uneven traffic, which is most of the time in practice, causes the failure mode: a single popular system prompt or a viral document can concentrate load onto one worker and turn it into a hotspot while its neighbors sit idle.

Eviction policy and the cost of getting it wrong in a distributed context

On a single machine, eviction is a contained decision. Something like LRU decides what gets dropped, the decision is bounded by that one worker's HBM, and if it turns out to be the wrong call, the penalty is a recompute on the next request that needed it. Nothing propagates.

Distribute the cache across a fleet and that containment disappears. An eviction decision made on one node changes what the router elsewhere in the cluster believes is available, and there's no immediate signal when that belief turns out to be wrong. The mistake stays invisible until a later request misses and either eats a recompute or triggers a cross-node fetch, by which point the cost has already been paid. Granularity compounds the problem: evicting at the level of PagedAttention's memory blocks is far more efficient than evicting entire requests wholesale, but doing it well means tracking block-level access frequency across every worker in the cluster, not just the one making the eviction call.

LMCache's answer to this is to stop treating eviction as something that happens implicitly and expose it as a control surface instead: pinning, lookup, cleanup, movement, and compression, all available as first-class operations rather than buried inside an LRU policy nobody can inspect. That's a substantial infrastructure gain, evidence that eviction, treated as a control surface, carries far more leverage than a marginal tuning improvement would. It's evidence that eviction, treated as an infrastructure primitive rather than an afterthought, is one of the highest-leverage places to intervene in a distributed cache system.

Compression as a way to shrink the problem at every layer of the hierarchy

Every constraint covered so far, HBM scarcity, network bandwidth, eviction cost, gets smaller if the cache itself gets smaller. That's the appeal of compression, and it splits into two approaches. Token dropping removes cache entries at selected layers and attention heads outright, while quantization keeps every entry but stores it at lower numerical precision.

Correctness is the catch, and it surfaces only after some delay. Lossy compression introduces divergence from what a full-precision KV cache would have produced, and that divergence accumulates as output length grows. Short answers barely notice it. Long-form output, code generation, multi-step tool calling, degrades sharply even at compression ratios that looked perfectly safe in a short-answer benchmark.

VeriCache, from researchers at the University of Chicago, Tensormesh, Samsung Semiconductor, and Microsoft Research, tackles this by refusing to accept the tradeoff at face value. VeriCache drafts tokens using a compressed KV cache, then verifies them against the full, uncompressed KV cache stored outside GPU memory, correcting any wrong tokens so the final output matches full-KV inference exactly. It's a fitting place to end, because it makes plain what every layer of this stack has been demonstrating in its own way: the constraint never actually disappears, and it shows up as HBM capacity, interconnect bandwidth, or numerical precision. The engineering work is in deciding which layer absorbs it, and making that choice on purpose instead of by default. A 70B model with 8K context requires approximately 20 GB of cache per request https://dl.acm.org/doi/10.1145/3600006.3613165. A 70B model with 8K context requires approximately 640 GB of cache for a batch of 32 requests. vLLM's PagedAttention reduces KV cache waste to under 4% https://dl.acm.org/doi/10.1145/3600006.3613165. PagedAttention enables a 2–4× throughput improvement https://dl.acm.org/doi/10.1145/3600006.3613165. Mooncake can achieve up to a 525% increase in throughput in certain simulated long-context scenarios while adhering to SLOs compared to baseline methods https://arxiv.org/abs/2407.00079. NVLink 4.0 provides 900 GB/s of bandwidth for KV cache transfers within the same NVLink domain https://arxiv.org/html/2607.28633v1. At 900 GB/s under NVLink 4.0, a 2.6 GB KV cache transfer takes 2.9 milliseconds https://arxiv.org/html/2607.28633v1. An H100 GPU hour runs four to six dollars ($4–6) in 2026 cloud pricing https://medium.com/@sseshadri/why-llm-inference-is-disaggregating-its-memory-2d9d299d931a. VAST Data reported roughly 10× faster prefill times using ICMSP with VAST DataStore, an all-NVMe storage system https://www.blocksandfiles.com/ai-ml/2026/03/30/nvidia-and-its-partners-kv-cache-extenders/5209284. Combining LMCache with vLLM achieves up to a 15× improvement in throughput across workloads such as multi-round question answering and document analysis https://arxiv.org/pdf/2510.09665. Across eight typical workload profiles, the Unified AI Gateway yields modeled TTFT speedups of 1.25×–13.28× https://arxiv.org/pdf/2609.06940. The Unified AI Gateway provides input-cost benefits of 1.20×–6.16× across eight workload profiles https://arxiv.org/pdf/2609.06940. KV cache compression methods such as token dropping and quantization deliver 2–5× reductions in memory or transfer size https://arxiv.org/pdf/2605.17613. SmartGen reduces time-to-second-token by 4.3× compared with the typical full KV cache transfer approach https://arxiv.org/pdf/2607.28150. Recomputing the KV Cache on an AWS 8×B200 GPU cluster takes approximately 15 seconds https://arxiv.org/pdf/2608.01526.