KV Cache Offloading Stack Decisions From GPU DRAM to Object Storage

KV cache storage and management for production LLM serving requires deliberate decisions at each boundary from GPU HBM to object storage. Efficient serving reduces cache size before eviction, uses DRAM for reusable prefixes, and adds NVMe only when reloading beats recomputing. Shared storage serves a different purpose, enabling cache portability and disaggregated serving when the network can sustain the transfers.

Why KV cache offloading is not a simple memory extension

Planning around single-user peak VRAM understates the problem. KV cache grows per token, per layer, per attention head, and per concurrent request, so longer contexts and larger batches compound memory pressure. A single long-context session with a large model can require tens of gigabytes before concurrency enters the calculation. At that scale, cache can displace model weights as the dominant memory consumer.

Moving those bytes elsewhere substitutes transfer time for memory pressure, while recomputing discarded prefixes substitutes GPU work for storage. Repeated prefill becomes untenable for conversations, agentic loops, and RAG sessions revisited after interruptions. Meng et al. define κcrit as the critical cached-to-prefill token ratio beyond which execution becomes memory-bound. Their measurements show that with naive offloading, GPUs can use only a fraction of rated TDP while moving data takes up nearly all end-to-end latency.

How the bandwidth cliff between tiers sets the terms of every decision downstream

The hierarchy trades progressively slower access for capacity and sharing. GPU HBM holds the active cache, while CPU DRAM provides a warm tier whose return path is constrained by PCIe. DRAM access latency remains orders of magnitude below NVMe and remote storage. Local NVMe bandwidth is roughly another order of magnitude lower than DRAM, with access latency in the hundreds of microseconds to low milliseconds. A single drive supplies roughly 7–10 GB/s, so total storage capacity and the bandwidth of all the drives must be weighed separately.

Remote storage adds the outermost transfer boundary. NFS, S3-compatible storage, and RDMA-backed object stores make up the shared tier, which has the lowest bandwidth and highest latency in this working hierarchy. Their advantage is portability across the serving fleet. Consequently, a stored block earns its place only when expected reuse amortizes the cost of moving it.

The GPU HBM boundary — what leaves first and why PagedAttention alone is not enough

PagedAttention alone does not settle the offload decision. Even with in-GPU memory management, long-context, high-concurrency serving must determine which blocks leave HBM, when they leave, and in what format. FP8 KV quantization halves per-token cache size independently of model weight precision. NVFP4 stores KV tensors in 4-bit format on NVIDIA Blackwell hardware, including HGX B200, B100, and GB200, allowing meaningfully more concurrent users before offloading becomes necessary.

Other in-GPU levers provide headroom within narrower bounds. Increasing the GPU memory utilization fraction makes more memory available, while tensor parallelism distributes weight memory across GPUs to free space for cache. Available GPUs and network topology constrain that route. Reducing maximum context length is appropriate only when the application genuinely does not need the full window. Otherwise, the application saves memory but loses some of what it can do.

Keeping that headroom useful requires an eviction policy suited to the workload’s prefix reuse. Least-recently-used block eviction suits multi-turn and RAG workloads with high prefix reuse because recently displaced blocks may soon be requested again. Unique-prompt workloads offer no such return when their evicted blocks are never reused. Eagerly writing those blocks to DRAM consumes bandwidth without avoiding future computation.

The DRAM tier — why it is the highest-leverage upgrade and what limits it

CPU DRAM is the most cost-effective extension of GPU memory for KV cache, making it the first offload tier that most teams should develop. Its value depends on reuse. Completed conversation turns, repeated system prompts, and shared document prefixes can come back from warm storage with less effect on time-to-first-token than NVMe hits. Multi-turn chat, RAG, agentic sessions that pause during tool calls, and code assistants that reuse repository context all fit this pattern. Their cache traffic preserves work that would otherwise recur.

For one-shot unique prompts, batch classification, and single-question answering, the costs and savings go the other way. Blocks written to DRAM may never be read again, so they add write-path latency with no read-path savings. Even a high-hit-rate workload faces another limit because warm blocks must cross PCIe to reach HBM. Meng et al. show that transfers can dominate when cached tokens far outnumber new prefill tokens. Once DRAM capacity is saturated and PCIe is the bottleneck, the question becomes whether NVMe can help or merely add another constraint.

The NVMe tier — when reloading beats recomputing and when it does not

NVMe is justified when cold-block reload costs less than prefix recomputation for the actual model and workload. Large models at long context can favor reloading decisively because rebuilding the prefix requires substantial GPU work. Smaller models on fast GPUs can favor recomputation when storage is slow or congested. Meng et al. make this a workload-specific breakeven. A cache hit is useful only if retrieving it beats the alternative.

Model architecture can remove the need for this tier. DeepSeek V3.2 uses multi-head latent attention, while hybrid Mamba architectures can have minimal KV cache requirements. At typical context lengths, these smaller footprints may fit in DRAM without NVMe. Context length alone is therefore insufficient for sizing an offload stack. The representation of the cache determines how much storage pressure that context creates.

Where NVMe is warranted, sustained writes and endurance govern drive selection. KV spillover is continuous and write-heavy, so the practical design uses datacenter-class drives with high drive writes per day ratings and multiple drives for bandwidth aggregation. A single consumer SSD falls short on both dimensions. Tutti addresses a separate source of delay through a GPU-centric KV cache object store whose CPU only loads I/O kernels asynchronously once per layer. This lets transfers run without the CPU coordinating each one, which can cause GPU stalls.

NVIDIA cuFile and GPUDirect Storage remove CPU staging from the local NVMe-to-GPU path, addressing another transfer overhead in the spillover design. Avoiding a fetch is better still. NVIDIA Dynamo’s KV-aware router places requests on GPUs that already hold relevant prefixes, improving HBM and DRAM hit rates before NVMe is consulted. Routing and direct transfer thus solve different problems. One reduces how often the storage boundary is crossed, while the other reduces overhead when crossing is necessary.

Compression at the tier boundaries — lossless and lossy techniques that change the economics

Compression lets more cache cross the same link in the same time. Applied before eviction, FP8 and NVFP4 reduce the bytes that traverse PCIe and reach lower storage tiers. The smaller cache also lets DRAM hold more warm blocks. At NVMe, reduced write volume eases endurance demands and leaves more bandwidth for concurrent users. Compression therefore changes the reload-versus-recompute breakeven rather than merely improving an already chosen configuration.

SplitZip demonstrates how lossless compression can extend those gains. It uses GPU-friendly parallel decoding to reconstruct the original cache bit-exactly, improving BF16 KV transfer speed, time to first token, and request throughput. It also works on an already-quantized cache such as FP8, saving more bandwidth without any further risk to accuracy. General-purpose algorithms do not compress BF16 enough to be useful, so purpose-built approaches rely on the structure of attention tensors.

The OSDI 2026 zero-copy KV offloading paper addresses intermediate buffer-copy overhead in long-context serving. Its approach complements compression because eliminating a copy and reducing the bytes transferred attack different sources of delay. Both alter the effective price of crossing a bandwidth cliff. That price must include both the data path and the storage device’s advertised bandwidth.

The networked storage tier — when shared KV cache enables disaggregated serving and what it costs

Shared KV storage changes the architecture of serving. It allows prefill and decode to run in separate clusters while cache moves between nodes. Mooncake, from Moonshot AI / Kimi, implements this approach across cluster-wide CPU DRAM and SSDs. Its Transfer Engine integrates with TensorRT-LLM and vLLM v1 as a KV connector. Cross-node movement nevertheless remains bounded by RDMA bandwidth per NIC.

NVIDIA NIXL supplies transport. Its supported paths include NVLink, RoCE, InfiniBand RDMA, and TCP via UCX for GPU and DRAM transfers, with GPUDirect Storage for local GPU-to-NVMe movement and backends including NVMe-oF and S3. RDMA can keep inter-node transfers in memory without routing them through NVMe. ObjectCache extends the architectural options further. Its high-bandwidth RoCE prototype brought S3-compatible runtime KV storage within a small percentage of optimal local DRAM performance at long context, showing that object storage can serve as more than an archive under those conditions.

LMCache provides the persistence and integration layer for configurable backends including local NVMe, Redis, and remote object stores. It plugs into vLLM, SGLang, and NVIDIA Dynamo, with reported cache-hit-heavy latency gains from retaining blocks across restarts and sharing them across replicas. Those shared-backend benefits go beyond node-local NVMe. Integration remains an engineering constraint, however. Mooncake Store and DeepSeek 3FS have faced gaps in efficient small-tensor movement across tiers and full EFA support, although preliminary or partial EFA support has emerged in some systems.

CXL and emerging near-memory architectures that may reshape the tier boundaries

CXL-attached memory and processing-near-memory accelerators challenge the separation between HBM capacity, CPU DRAM, and transfer bandwidth. These approaches could introduce a tier with more capacity than HBM and higher effective bandwidth than conventional PCIe offloading. These approaches could change the earlier breakeven calculations. They do not remove the need to account for movement and cache-management overhead.

CXL-SpecKV (2026) moves cache management and selection toward near-memory accelerators through a disaggregated FPGA speculative KV-cache system for datacenter serving. This targets scheduling overhead that accumulates at high concurrency. TraCT (2025) instead targets rack-scale disaggregated serving with CXL shared-memory KV cache. Rack-scale sharing could let GPUs reuse prefixes without the RDMA bandwidth constraint of network-transfer designs. When planning infrastructure for the next several years, these systems call for another look at where cache lives and which device manages it, though they have yet to show that they can replace existing tiers in every case.

How should production teams manage KV cache across the full stack?

The practical decision sequence starts with prefix reuse. Multi-turn, RAG, agentic, and code-assist workloads justify engineering a reuse-oriented offload stack, while unique-prompt and batch-classification workloads generally do not. For workloads with reuse, apply supported FP8 or NVFP4 quantization before sizing lower tiers. Then provision DRAM generously before adding NVMe. System memory of 1–2 TB on a multi-GPU node is defensible for most long-context serving scenarios, subject to the PCIe transfer limit already established.

NVMe should follow only when DRAM is saturated and measured reload cost is below recomputation cost. That comparison must take into account the model architecture, context length, and the bandwidth available across the drives, which need to be high-DWPD datacenter drives for sustained spillover. Shared storage requires a different justification. Persistence across restarts, fleet-wide portability, or disaggregated prefill and decode can warrant it, provided InfiniBand, RoCE, or EFA supports the required transfer rates. Lossless compression belongs across these boundaries, stacked on quantization.

Tool selection follows the boundary being constrained. Routing, local transfer, shared-cache management, and inter-node transport each have their own job, so each is needed for the work it does. FAST 2026’s SolidAttention work on memory-constrained PCs and the OSDI 2026 zero-copy work show storage research addressing KV offloading directly, giving open-source serving teams developments to track. MLPerf Storage v3.0’s September 2026 inclusion of KV cache I/O tests adds a shared measurement baseline. Practitioners can now benchmark tier decisions against a reproducible standard rather than relying only on internal workload proxies.