KV Cache Tiering Across DRAM NVMe and Remote Storage for Inference Latency

KV cache tiering across DRAM, NVMe, and remote storage is a latency engineering problem. For LLM inference throughput, the useful architecture keeps active blocks in GPU HBM, reusable prefixes in DRAM, and colder caches on NVMe or shared storage. NVMe can reduce recomputation and improve cost per token when prefix reuse is high, but only if transfer latency and I/O overhead remain below the cost of rebuilding the cache.

Why KV cache size makes GPU memory the wrong place to store it all

KV cache size grows linearly with context length, layer count, KV heads, head dimension, and batch size. Dell's March 2026 survey, KV Cache Optimization Strategies for Scalable and Efficient LLM Inference, describes production context windows expanding from thousands to millions of tokens. With million-token windows, a single user's cache can need more space than all the HBM on an H100. GPU memory alone is therefore not a workable retention strategy at that scale.

LMCache's paper, An Efficient KV Cache Layer for Enterprise-Scale LLM Inference, reports that users' accumulated caches have grown far beyond GPU memory capacity. Eviction and offload are normal operating conditions. Discarding a prefix and recomputing it was viable at shorter contexts, but the long-context H100 example puts full-prefix recomputation at roughly eleven seconds, a delay incompatible with interactive response expectations. Once the GPU cannot retain the cache, reuse accelerates inference or introduces a stall depending on the cache's destination and return path.

What the three storage tiers actually deliver in bandwidth and latency

GPU HBM is the hot tier, with roughly 3.35 TB/s bandwidth on H100 SXM5 and sub-microsecond latency per block. It is the only tier fast enough to keep pace with token generation directly. CPU DRAM serves as the warm tier, accessible at roughly 63 GB/s over PCIe 5.0 x16, with latency in the 10–100 µs range. Its bandwidth is about 50× lower than HBM. Still, DRAM can bring back warm blocks for multi-turn conversations without a visible stall if the blocks move within the request's latency budget.

NVMe serves a different role as cold storage. A block fetched from NVMe can add milliseconds to decode, while restoring an entire long-context cache can take seconds. Neither should be confused with the HBM bandwidth available once the blocks reach the GPU. The boundaries between these tiers are therefore set by transfer volume and access time.

How the hot-warm boundary between HBM and DRAM is managed in practice

The HBM-to-DRAM boundary offers substantial capacity relief without the access delay of NVMe. LMCache extracts caches from vLLM and SGLang and stores them outside GPU memory, using batched movement and compute/I/O pipelining to limit transfer overhead. vLLM's PagedAttention supplies the block-management foundation. It allocates fixed-size blocks on demand, treating KV cache like OS virtual memory, so promotion and demotion can follow a page-eviction discipline without making the model storage-aware. Memory pressure can move retained blocks out of HBM, while a reusable prefix brings them back.

SGLang's RadixAttention makes shared prefixes central to reuse alongside recency. That distinction is useful when many RAG requests repeat system prompts or document prefixes. Multi-turn assistants and agentic loops can also reuse warm blocks as conversations go on or sessions resume after tool calls. A DRAM cache hit avoids repeating the full prefill. For this warm tier to be economically useful, the notes' 2026 sizing target is on the order of 1–2 TB of DRAM on an 8-GPU node, replacing repeated GPU work with comparatively inexpensive memory capacity.

What changes when NVMe enters as the cold tier

In the long-context H100 comparison, it takes roughly eleven seconds to rebuild a prefix and roughly six to eight seconds to reload the same KV blocks from NVMe. Reload wins, but it is not instantaneous. The economics come from avoiding more expensive recomputation, with the gap widening at longer contexts. LMCache's disk backend on vLLM enables this trade by retaining cold blocks on NVMe so that a subsequent hit can skip prefill.

The I/O path then becomes part of inference engineering. NVIDIA's cuFile API removes CPU staging from NVMe-to-GPU transfers, while DPU-managed I/O can reduce host involvement in storage traffic. These address different sources of overhead. NVIDIA's ICMSP, announced at CES 2026 and rebranded CMX at GTC March 2026, combines those approaches in its STX reference architecture. NVIDIA claims up to 5× token throughput versus traditional storage for that design, a system-level claim rather than a result that applies to any NVMe installation.

Drive selection introduces another constraint because KV writes are continuous and write-heavy. The appropriate specification includes datacenter-class high-DWPD drives and multiple drives for I/O parallelism. Even with faster I/O, a cold tier needs hits to pay off. For short-context, single-turn traffic, storage and DPU overhead can add cost without saving work, so NVMe should be used only when it helps.

How compression at the tier boundary changes the arithmetic

Moving from BF16 to FP8 halves KV memory, while NVFP4 reduces it to a quarter of BF16. Each tier can consequently retain more users' caches before spilling. The same reduction also lowers transfer demand, so a warm-to-cold movement that would saturate a PCIe link at BF16 may fit within the available bandwidth at FP8.

That gain has a quality constraint, however. RDKV and HqeKV document workload- and task-dependent trade-offs from aggressive quantization, so negligible quality loss is not a universal assumption. NVIDIA's nvCOMP provides another storage-boundary tool, offering GPU-accelerated compression and decompression for BF16 and FP8 through data-type hints in its ANS codec. CacheGen, presented at ACM SIGCOMM 2024, treats compression and streaming as a joint optimization. With suitable FP8 or FP4 storage, more blocks remain in each tier, reducing slower-tier accesses and therefore tail latency.

What disaggregated prefill-decode architectures add to the tiering problem

Prefill is compute-intensive and more tolerant of batching, while decode is memory-bandwidth-intensive and latency-sensitive. Running them together can create interference that raises time-to-first-token. Separating them removes that particular coupling but requires the cache to cross a network. The hierarchy does not disappear with disaggregation. Instead, every local promotion and eviction decision remains, with fabric bandwidth added to the path between the workers that produce and consume KV blocks.

Mooncake's transfer engine integrations span TensorRT-LLM, vLLM v1, and SGLang in late 2025 through 2026, including RDMA-based disaggregated operation. LMCache supports cross-engine transfer over Ethernet, RDMA, and NVLink. NVIDIA's NIXL, the NVIDIA Inference Xfer Library, handles movement across NVLink, InfiniBand RDMA, PCIe, and TCP, and Dynamo uses it between prefill and decode workers. RDMA cuts protocol overhead, but one-sided verbs can move only as much data as the physical link allows.

At high concurrency, the limiting resource can therefore shift from compute to interconnect. Local retention policy still influences which blocks need slower storage access within each cluster. IMPRESS, presented by Weijian Chen et al. at USENIX FAST '25, uses importance-informed decisions in a multi-tier prefix KV storage system. It uses block importance to decide which blocks to evict from the warm tier to the cold tier. Disaggregated serving consequently requires storage placement and network transfer to be reasoned about together, rather than treating the fabric as an unlimited connection between independent caches.

CXL and remote object storage as emerging tier extensions

CXL extends the hierarchy between DRAM and NVMe, offering DRAM-like byte-addressability without requiring capacity on the GPU's local DRAM bus. Its appeal is additional capacity without NVMe's access-latency penalty. The 2026 SAC paper studies disaggregated KV caching for sparse-attention LLMs with CXL. Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management examines the adjacent CXL-SSD boundary. These proposals add another placement option alongside the hot tier.

ObjectCache proposes layerwise object-storage retrieval for persistent cross-request reuse beyond a single serving instance. DeepSeek's 3FS, open-sourced in 2025 under the MIT License, provides an example of distributed storage built for AI training and inference, while LMCache already supports Redis-based shared prefix caching. The distinction is between deployed shared-cache backends and newer proposals that extend the scope of sharing.

Prefill-as-a-Service, a 2026 proposal, takes that scope toward cross-datacenter KV sharing. An Internet for the KV Cache argues for a standard protocol layer across infrastructure boundaries, drawing an analogy with HTTP. Such extensions broaden where a reusable prefix can live. They also add access paths that need to be assessed beyond what local memory specifications can show. Each new tier adds another source of latency, so a larger hierarchy needs more care in setting the boundaries between tiers.

Where workload type determines which tier engineering decisions actually matter

With sufficient prefix reuse, saved computation repays transfer overhead. Multi-turn chat repeatedly uses conversation prefixes, RAG shares system prompts and document prefixes, and agentic loops resume contexts after tool calls. Code assistants similarly reuse repository context across keystroke bursts. These are strong candidates for warm and cold retention. Single-turn, unique-prompt traffic, including batch classification and one-shot question answering, gains little from NVMe reuse.

Production request handling can weaken even an apparently favorable workload. LMCache's paper reports that context truncation substantially reduces prefix cache hit ratio in enterprise deployments. Wang et al.'s USENIX ATC 2025 study characterizes KV cache hit rates and tier utilization at a large cloud provider, supplying production-scale grounding for workload-dependent decisions. Model architecture alone cannot show how prompt diversity, session length, and truncation affect how much of the cache is reused.

The practical implication is to benchmark the actual request distribution. NVIDIA's AIPerf is oriented toward inference infrastructure, making it a more appropriate instrument for this task than general storage benchmarks. Measured reuse and transfer behavior should then determine DRAM capacity, NVMe endurance and parallelism, and the fabric class needed for cross-node movement.

How to reason about tier boundaries when specifying an inference storage system

An inference storage specification should join three variables, namely the bandwidth gap between tiers, the colder tier's access latency, and the workload's achieved hit rate. A larger cache is useful only when retaining and retrieving its contents improves the serving path. The comparison must include reload against recompute. For disaggregated serving, it must also include the network transfer between prefill and decode, rather than ending the analysis at local disk performance. This is how storage selection connects to throughput and cost per token without assuming a universal NVMe saving.

DRAM targets set how much the warm tier can hold, DWPD and multi-drive parallelism describe how well the cold tier can keep up over time, and fabric class describes how much data can move between nodes. The I/O tools are not interchangeable either. GPU-direct transfer removes CPU staging, DPU offload reduces host I/O work, and RDMA reduces network protocol overhead. Their useful combination depends on which path limits the target workload, while precision and quality requirements determine how many bytes that path must move.

As CXL and remote storage mature, the same test can accommodate additional tiers. Their capacity is not sufficient evidence of their usefulness. A tier earns its place when the measured reuse benefit exceeds retrieval and management costs within the workload's latency budget.