Controlling KV Cache Memory Growth Without Sacrificing Throughput

Efficient KV cache management for production LLM serving requires four layers, applied in order and matched to the workload. Paged allocation recovers wasted memory, quantization shrinks each token’s footprint, eviction limits retained tokens, and offloading extends capacity beyond GPU memory. Controlling memory growth without sacrificing throughput depends on combining these layers with prefix reuse and an adequately provisioned data path, rather than pushing any single technique to its limit.

Why KV cache memory becomes the dominant infrastructure problem at scale

KV cache memory grows linearly with context length because each generated token appends a key-value pair. When contexts are long enough, cache memory can exceed parameter memory and become what limits serving. Every concurrent request also needs cache state, so longer contexts and more users compete for the same GPU memory budget.

Capacity is only half of the constraint. Each full-attention decode step reads the entire cache from HBM, and that read consumes more time as the sequence grows. Memory bandwidth, rather than FLOPs, then governs throughput. Xu, Khaira, and Singh’s March 2026 paper identifies efficient KV cache management as a first-order challenge for scalable deployment.

Why no single technique solves this across all workloads

The same linear growth produces different bottlenecks across workloads. High-throughput serving encounters fragmentation and concurrency limits, while long-context single requests strain the per-token memory budget. Multi-turn conversations offer a chance to reuse context because much of each request repeats what came before. Hardware and accuracy requirements further constrain the available response.

The technique families are paged allocation, quantization, eviction, architectural compression, and offloading. The first, second, third, and fifth form the operational layers, while architectural compression depends on the model. Their interactions rule out a universal winner. Quantization without paging still leaves fragmentation, and offloading without prefix reuse creates unnecessary I/O.

One aggressive technique may appear operationally simpler than several moderate ones. But how much accuracy aggressive compression loses depends on the workload, and those losses can be non-linear. FP8 and INT8, for example, both lower storage precision, but they differ in how much accuracy they lose on long-context retrieval. The decision is therefore not just how much memory to reclaim, but which representation or information can change safely.

Paged memory allocation as the required substrate, not an optimization

Static pre-allocation sets aside space that requests of different lengths cannot use well. As requests arrive and finish, wasted allocation consumes memory that could otherwise admit another request. PagedAttention addresses this by storing KV state in fixed-size, non-contiguous blocks instead of requiring a large contiguous allocation for each sequence.

The reported reduction in memory waste is substantial, from roughly sixty to eighty percent to under four percent. Throughput gains reach up to 24× over a HuggingFace Transformers baseline, although that comparison is not a forecast for every deployment. Paged KV management is standard across vLLM, SGLang, and TensorRT-LLM, with SGLang building RadixAttention on paged KV. Flexible allocation also makes continuous batching feasible.

Moving from static allocation to a paged serving engine can defer the need for quantization or offloading. Paging also enables prefix caching, which reuses stored state for repeated system prompts and shared context. Once allocation and reuse are efficient, the next lever is the size of each retained token.

KV cache quantization and shrinking per-token footprint with controlled accuracy loss

FP8 KV quantization is a practical starting point for most production workloads. It halves the memory each token needs, and reported accuracy regression is below one percent on the benchmarks tested. The smaller footprint leaves room for larger batches or longer contexts with the same GPU budget. This is both model compression and storage-format engineering because the representation directly affects how much the cache can hold and how it moves.

Reuse can compound the benefit of a smaller format. Reported benchmarks place BF16 KV without prefix caching at roughly 800 tok/s aggregate, compared with roughly 1,900 tok/s for FP8 KV with prefix caching.

More aggressive formats serve narrower constraints. KIVI achieves nearly lossless 2-bit quantization through separate per-channel and per-token strategies, while FlexGen uses group-wise 4-bit quantization to reduce offload I/O, with compression and decompression overhead. MLA-enabled DeepSeek models reduce cache size structurally before quantization is applied. Model choice therefore changes the starting budget.

Accuracy remains the condition for deployment. Q&A, retrieval, and code benchmarks may not expose compounded losses on multi-instruction long-context tasks. Teams with accuracy-critical workloads should validate the format on their own task distribution. Quantization preserves token representations at lower precision, but it does not stop their number from growing.

Token eviction and discarding low-salience KV pairs to cap cache size during long sequences

Eviction addresses that remaining growth by removing selected KV pairs. Policies differ in how they estimate usefulness, using position, attention, embeddings, or hybrid signals. The oldest token can differ from the one with low salience. H2O, SnapKV, and Ada-KV provide examples of intelligent selection beyond simple position-based removal.

KVpop’s July 2026 preprint reports retaining 98% of full-attention performance on a small Qwen3 model at seventy-five percent KV cache compression. MiniCache reports cutting memory use by up to roughly forty percent, with around five times the throughput. These results establish the potential of selective cache reduction within the models and tasks tested. Policy choice remains an engineering decision.

Unlike quantization, eviction removes the cached representation entirely. Recovering a discarded KV pair requires recomputation if it later becomes relevant. Attention at the current step may also be a poor predictor of future relevance, although hybrid signals reduce that risk. Eviction is a better fit for long documents or code where distant context really is low-salience than for reasoning tasks where any earlier detail may turn out to be needed.

For ultra-long contexts, eviction and quantization can compose. The operational framework establishes quantization first, while a combined processing pipeline can evict low-salience tokens and then quantize those retained. Even that combination cannot make every workload fit in HBM. Offloading addresses the residual capacity constraint without requiring further information loss.

KV cache offloading and extending capacity into CPU DRAM and NVMe when HBM is exhausted

Offloading extends the memory hierarchy from GPU HBM to CPU DRAM and then NVMe SSD. Each tier is roughly an order of magnitude larger and slower than the preceding tier. The capacity gain is substantial, but moving state back to the GPU can erase the throughput benefit. Interconnect bandwidth is often the binding resource.

Tool choice follows the required data path. LMCache extends vLLM with storage across GPU memory, CPU RAM, and disk, and reports latency reductions in benchmarks. NVIDIA Dynamo supports disaggregated serving with vLLM and KV transfer through NIXL using RDMA, NVMe-oF, and S3.

INF2 reports roughly 3.5× the throughput with Computational Storage Device hardware and no loss of accuracy, compared with a baseline where KV I/O takes over eighty percent of inference time. Scality ADI instead demonstrates an RDMA path from storage-server memory directly into reserved GPU memory. In its tests, first-token time stayed steady across a wide range of cache sizes while it held many times more KV state than HBM alone could hold.

Scality’s testing exhausted four 100 Gbit/s RDMA links before saturating NVMe, while four-GPU long-context serving can generate roughly 200 GB/s of cache store and refill traffic. Teams designing an offload tier should therefore size RDMA bandwidth first and NVMe capacity second. A capacity-rich system can still be throughput-limited.

Prefix caching and disaggregated serving as cross-cutting multipliers

Prefix caching reduces repeated work across requests rather than changing the representation of an individual token. Repeated system prompts and shared context become cache lookups after their KV state has been computed. Agentic and RAG workloads with long shared prompts offer substantial reuse, whereas unique single-turn questions offer almost none. The measured hit rate shows how much this choice helps.

Disaggregated prefill and decode separate two different resource demands. Prefill workers can be sized for compute throughput, while decode workers can be sized for HBM capacity and bandwidth. The transfer stack described above connects those workers, making cache movement part of the serving architecture. Compression reduces what must move, and prefix reuse reduces what must be computed again.

The transfer substrate also affects the payoff. GPUDIRECT RDMA removes bounce buffers by allowing the NIC to access GPU memory directly. CXL shared memory offers another approach, in which GPU DMA writes KV blocks into shared memory that decode workers can read without retransmission. RDMA’s memory pinning and queue pair synchronization impose several microseconds per transfer, which can become problematic for sparse attention. CXL is therefore promising for fine-grained sharing, but the production choice remains unsettled.

Mapping the four layers to seven deployment scenarios

Xu et al. map KV optimization to seven deployment scenarios. The infrastructure decision turns on context length, concurrency target, and accuracy requirement, with hardware determining which tiers are available. Five common deployment profiles expose most of the choices. Their combinations differ because each stresses a different part of the cache lifecycle.

Long-context single requests call for FP8 quantization and, where task sensitivity permits, eviction. GQA or MLA can provide the largest structural reduction at model-selection time, while the longest contexts may still require NVMe offload. High-throughput datacenter serving instead starts with paging, then adds prefix caching and FP8 to expand useful batch capacity. Eviction usually offers less incremental value for many short-to-medium requests.

Multi-turn conversation is primarily a reuse problem. Prefix caching preserves the shared history, while quantization manages growth in the non-shared portion. Edge deployments lack an offload tier, making 2-bit to 4-bit quantization and eviction relevant, subject to per-task validation. Accuracy-critical reasoning takes the conservative path of paging and prefix caching, avoiding eviction and quantization below FP8 unless the specific task validates them.

High-concurrency long-context serving combines the hardest constraints. It requires paging, FP8, selective eviction, and NVMe offload, plus disaggregated prefill and decode to avoid head-of-line blocking. Scality’s 8 TB per GPU result and NVIDIA ICMSP address this capacity regime. Neither removes the need to provision the transfer path.

For teams applying this framework, the sequence begins by confirming paged allocation in vLLM V1, SGLang, or TensorRT-LLM. Prefix caching comes next, with hit-rate measurement determining its actual value. FP8 comes next when context length pushes KV memory beyond parameter memory, if tests show that it meets the task's accuracy needs. Eviction enters only when quantization cannot contain persistent context growth within the budget. NVMe offload addresses the remaining HBM shortfall, with interconnect headroom assessed before storage capacity.

Where the research frontier is moving and what that means for infrastructure decisions made today

Model-side compression is changing how much data the storage tiers must hold. Ray, Feamster, and Jiang report that KV cache for 100,000 tokens fell from 9 GB in DeepSeek-V3.2 to 1 GB in DeepSeek-V4-Pro through Compressed Sparse Attention. TransMLA also demonstrates compression of LLaMA-2 with a small performance drop across six benchmarks. Architectural compression can also help models that were trained without MLA.

At the same time, reuse is pushing cache management beyond the node. Ray et al. propose a content-distribution model in which placement and replication follow model, infrastructure, and application metrics across datacenter and cloud boundaries. ObjectCache, from ETH Zurich and HPE Labs, demonstrates S3-compatible object storage as a runtime KV backend with acceptable long-context latency when protocol and transfer scheduling are co-designed. Storage need not always sit beside compute.

These developments build on the four layers, which remain in place. Adaptive pipelines may eventually select combinations per request, while RDMA and CXL remain open architectural choices for increasingly fine-grained sharing. Teams choosing infrastructure today should make topology, protocol, and interconnect key parts of the design.