KV Cache Offload to NVMe Under Memory Pressure

KV cache does not have a fixed footprint. It grows along five dimensions at once: layer count, attention head count, head dimension, sequence length, and batch size. A model that fits comfortably in memory at short context can blow past a single GPU's capacity the moment context length or batch size moves up, and most capacity planning still treats KV cache as a side cost instead of the main event. That habit is backwards, and it is why serving stacks stall out under load: KV cache offload is not a fallback you bolt on when memory runs low, it is the default condition of serving long-context models in production.

Take Llama-3-8B serving a 1-million-token context window. That alone needs at least 256 GB of KV cache memory, more than three times what a single 80 GB H100 holds, before the model weights even enter the picture. At a more modest 8K context, a 70B model's KV cache runs around 20 GB per sequence. Put 32 of those in a batch, a normal ask for any production serving stack, and total demand hits roughly 640 GB. At that point the cache is not sitting next to the model weights as a minor cost. It is the dominant consumer of memory on the box, and treating it as an afterthought is how teams end up debugging a slowdown that was designed into the system from day one.

Long-context windows, 128K tokens and up, reaching into the millions of tokens for some workloads, are what production traffic looks like now. Past that line, HBM alone cannot carry the load, full stop. What backs it up, and how much that backup costs in speed, is the real question, and it is the one this piece works through.

What each of the three storage tiers delivers

The standard fix is a hierarchy: GPU HBM at the top, CPU DRAM in the middle, NVMe SSD at the bottom. Each step down is cheaper per gigabyte. Each step down is also a lot slower, and that second part is the one teams underestimate, usually by an order of magnitude or three.

An H100 SXM5's HBM moves data at roughly 3.35 TB/s. Dropping to CPU DRAM over a PCIe 5.0 x16 link cuts bandwidth to around 63 GB/s, a drop of well over an order of magnitude. Dropping again to NVMe over PCIe 4.0 causes bandwidth to fall to around 7 GB/s. Each tier down is a cliff, not a gentle slope, and no amount of clever scheduling turns a cliff into a ramp.

Latency tells the same story, and it hurts serving more directly than bandwidth does. HBM operates in sub-microsecond time. DRAM is in the 10 to 100 microsecond range. NVMe spans 100 microseconds up to a full millisecond, so the bottom tier is not just thinner on bandwidth, it belongs to an entirely slower latency class. Pulling KV cache back from CPU DRAM already adds 10 to 50 milliseconds per cache retrieval, and NVMe adds more on top of that. None of this rules out using the tier. It means offload has to be designed around the cliff from the start, not treated as a background detail some other team's framework will quietly absorb.

The Clash Between KV Cache I/O Access Patterns and NVMe Drive Tuning

Enterprise NVMe drives have been tuned around the access patterns that databases and virtualized workloads produce, not the sub-kilobyte access patterns KV cache inference generates. Controllers, wear-leveling logic, and read-ahead heuristics are all built around that assumption. That assumption is wrong for KV cache, and pretending otherwise is where a lot of offload performance quietly disappears.

KV cache inference runs on embeddings a few hundred bytes each, with individual KV blocks sitting well under a kilobyte. Serve that traffic straight off a drive tuned for 4K blocks and you get read amplification: the drive pulls several times more data than the request actually needs, burning bandwidth and flooding controller cache with data nobody asked for.

Frameworks compensate by batching before the request reaches the drive. A study out of Vrije Universiteit Amsterdam and IBM Research looked at KV cache offload under DeepSpeed and FlexGen and found the dominant I/O size hitting the block layer was 128 KiB, far larger than the tiny per-vector accesses happening at the serving layer above it. Coalescing is happening. The mapping from logical access to physical I/O still is not clean, and that gap is where performance leaks out.

The same study found average read bandwidth reached 2.0 GiB/s, while write bandwidth measured just 11.0 MiB/s. That is not a rounding difference, it is nearly three orders of magnitude apart. Reads and writes behave like two different workloads entirely, and any write-path design that treats them the same will underperform on one side, usually the write side, since that is the one nobody bothers tuning for.

The CPU as Bottleneck Even When the Drive Is Fast Enough

Even when the drive itself can move data fast, the path leading to it often cannot. GPU memory allocation for KV cache tends to be fragmented, which produces a large volume of small, scattered I/O requests. Somebody has to initiate each one of those requests, and when that job falls to the CPU, a component with far less parallelism than the GPU it is serving, the CPU becomes the bottleneck. Most offload designs assume a faster drive fixes the problem. The drive was rarely the constraint to begin with.

GPUDirect Storage was supposed to fix a piece of this. It builds a direct DMA path between the SSD and GPU memory, skipping the CPU bounce buffer that used to sit in the middle of every transfer. That is a real improvement, but GDS still needs the CPU to kick off each I/O operation. GDS can deliver meaningful read bandwidth for prefetching large blocks. Under high concurrency, though, the per-operation initiation overhead piles up and starts eating the gain. The fix works only until the load pattern that actually matters in production occurs.

GDS-enabled LMCache, one of the existing SSD-backed KV cache approaches, has been measured with 70 to 80% GPU bubble time in production. The GPU sits idle three-quarters of the time or more, waiting on I/O that a CPU too slow to keep pace cannot deliver on schedule. A fast drive behind a slow orchestrator is still a slow system, and no amount of NVMe tuning changes that math.

GPU-Centric I/O and the Removal of the CPU From the Critical Path: the Tutti Approach

Tutti starts from one architectural bet: pull the CPU out of both the data path and the I/O control path between HBM and SSD. The CPU still exists in the system, but its job shrinks to a single act, loading I/O kernels once, asynchronously, at the start. After that, the GPU issues and manages its own I/O without asking the CPU for anything. Given that the CPU was the bottleneck in the first place, that is the correct call, not a helper worth keeping around out of habit.

Three mechanisms carry the design. A GPU-native object store uses Scatter Gather List addressing, so the GPU can describe a fragmented, non-contiguous transfer and start it without CPU coordination. A submission and completion queue mechanism running on the GPU mirrors the asynchronous I/O model Linux uses on the CPU, letting the GPU submit and complete I/O operations without blocking. A slack-aware scheduler, built on offline profiling, fits I/O transfers into the gaps in GPU compute rather than letting I/O and the forward pass fight over the same bandwidth.

Tested on Llama3-8B with a 64K sequence and a 75% cache hit rate, this design cut time-to-first-token by 78.3% compared to GDS-enabled LMCache. That is what happens when the CPU stops being asked to do a job it was never fast enough to do well.

What py-kvcache reveals about transfer granularity and scheduling placement

py-kvcache is a vLLM KV Offload connector built to study and improve NVMe-backed prefix caching, with its design driven by findings about how transfers should be sized and scheduled. Three design choices carry the weight here, and the scheduling one matters more than the other two put together.

Asynchronous direct I/O minimizes CPU staging overhead on the read path. Shared staging memory on the CPU is explicitly bounded rather than left to grow with request size, so it cannot quietly bloat DRAM usage under load. Preloading is scheduler-aware: disk reads start while a request is still sitting in the queue, so I/O overlaps with compute before the request has formally begun.

At 80K tokens, py-kvcache loads from disk roughly twice as fast as LMCache, and preloading by itself accounts for most of that gain. When a transfer starts, relative to the request's lifecycle, matters nearly as much as how fast the underlying drive is. That single fact undercuts the argument that faster NVMe alone solves this problem, because scheduling placement, not raw drive speed, is doing the heavy lifting. With GPU, CPU, and disk caching switched on together, py-kvcache runs noticeably faster than LMCache and lands within about 4% of vLLM's own native KV Offload implementation, close enough that the remaining gap is a rounding error next to the gains upstream of it.

Head-wise offloading as an alternative granularity for CPU-tier offload

HEADINFER takes a different cut at the problem. Instead of offloading whole layers of KV cache to CPU RAM, it offloads at the level of individual attention heads, keeping only a small working set of heads resident on the GPU at any given moment.

The math holds up cleanly here, which is what makes the approach defensible rather than merely clever. Attention heads are mathematically independent of each other, so attention output can be computed one head at a time, with only that head's KV cache resident on the GPU at that moment. No approximation is involved. Computing heads one at a time produces exactly the same result as computing all heads at once, just spread out over time instead of crammed into a single pass.

On Llama-3-8B at a context length in the millions of tokens, that head-by-head approach drops GPU KV cache footprint from 128 GB down to 1 GB, and total GPU memory demand from 207 GB to 70 GB, a 92% cut in KV cache memory against a standard BF16 baseline. The payoff: 4-million-token inference becomes possible on a single consumer GPU with 24 GB of memory, without touching model accuracy. That changes what class of hardware long-context serving actually requires, and it deserves more credit than the usual "efficiency win" framing gives it.

NVIDIA's CMX Platform and NIXL: Rearchitecting the Hardware Path

NVIDIA announced its Inference Context Memory Storage Platform at CES 2026, since renamed Context Memory eXtension, or CMX. The goal is to make KV offload to NVMe a first-class capability built into the platform itself, rather than something each serving framework hacks together on its own, which is more or less what Tutti, py-kvcache, and HEADINFER have each been doing independently.

CMX runs a four-tier hierarchy where NVMe-resident KV cache becomes part of the context memory address space directly. Context size is no longer bounded solely by a single node's HBM capacity. NVIDIA's own figures claim up to 5× better power efficiency compared to traditional storage approaches.

The detail that matters most: BlueField-4 DPUs take over KV cache movement entirely, pulling it off the GPU's compute path. That is the same failure point Tutti and py-kvcache were both built to work around, a CPU too slow and too serialized to orchestrate fragmented I/O at the scale modern context windows demand. Handing that job to a DPU built for it, instead of a general-purpose CPU pressed into service it was never suited for, is the shift that actually closes the gap between HBM and NVMe. Every framework fix covered above was a workaround for the CPU's limits. CMX is the first attempt to remove the limit at the hardware level instead, and that distinction, workaround versus removal, is what separates it from everything that came before it.