Prefix Caching and Prompt Deduplication in vLLM

Every LLM inference request splits into two phases with very different resource needs: prefill, which chews through the prompt and is bound by raw compute, and decode, which generates one token at a time and is bound by memory bandwidth. That split matters most when many requests share the same opening tokens. A system prompt, a chunk of retrieved documents, a batch of few-shot examples: these repeat token-for-token across thousands of concurrent calls, and recomputing them every time wastes prefill cycles and drags out time-to-first-token for no good reason. Prefix caching exists to stop that waste, and prompt deduplication exists to catch what caching misses. Skipping either one means paying GPU hours to re-derive the same block of boilerplate tokens, over and over, for no return.

The scale of the problem gets worse once you look at where the memory actually goes. KV cache size grows with both context length and batch size, and at production volume that cache routinely outgrows the model weights sitting in VRAM. Naive allocation makes it worse: reserve a contiguous block sized for the maximum possible context, and most of that reservation sits empty for the life of the request. Fixing that waste requires engineering below the application layer, in how the GPU hands out memory.

PagedAttention: turning GPU memory into a shared pool for prefix caching

PagedAttention, introduced by Woosuk Kwon and coauthors at SOSP 2023 (doi.org/10.1145/3600006.3613165), is the piece of infrastructure that makes prefix caching possible. The idea borrows straight from operating-system virtual memory: instead of one contiguous slab of VRAM per request, each request's KV cache splits into fixed-size blocks, and those blocks live wherever there's room in physical memory. Nothing has to sit next to anything else. Blocks get handed out as tokens get generated and freed the moment a request finishes.

That one design choice removes the need for a contiguous VRAM reservation sized to worst-case context length, and it means the physical block pool can be shared across requests instead of locked to one. PagedAttention cut KV cache waste to under 4%, a dramatic improvement over the much higher losses that plagued earlier static-allocation schemes. That's the reason higher batch sizes became viable at all, not a marginal tuning win. It's the reason higher batch sizes became viable at all, and prefix caching rides directly on top of that: without a shared, block-addressable pool, there's no place for a cached prefix to live that a second request could actually find.

vLLM's Automatic Prefix Caching: identifying and reusing matching blocks

vLLM turned this shared block pool into a caching layer called Automatic Prefix Caching. It ships on by default in vLLM's V1 engine, a call the vLLM team made once the gains held up consistently enough across workload types that treating it as optional stopped making sense.

The identity mechanism is a chain hash. Each 16-token block gets a hash computed not just over its own tokens but over every block that precedes it in the sequence. That chaining matters: two blocks with identical content but different histories hash differently, because the hash captures position in the chain, not just content sitting in isolation. There's no tree structure sitting on top of this, no parent-child relationship to maintain. Every block stands alone, addressable and freeable by its own hash, which keeps the bookkeeping simple even if it costs some flexibility, and that tradeoff is exactly where vLLM and SGLang part ways.

When a new request comes in, vLLM walks its token sequence 16 tokens at a time, computes the chain hash for each stride, and checks the physical block pool for a match. Find one, and that block's KV values get reused wholesale, no prefill computation required for those tokens. If it misses, the block gets computed and hashed fresh, ready for whatever request needs it next.

Where vLLM's block-hash design diverges from SGLang's RadixAttention

vLLM's APC and SGLang's RadixAttention both solve prefix reuse across requests, but they reach for different data structures, and the choice has real consequences for hit rate. Anyone picking between the two on "they both do prefix caching" is missing what actually determines performance on long, messy contexts.

vLLM keeps a flat hash table of fixed 16-token blocks. A match requires exact alignment: the incoming sequence has to line up on the same 16-token boundaries as whatever's already cached. SGLang's RadixAttention instead organizes cached sequences into a radix tree, where prefixes are matched at finer granularity than fixed block boundaries. That structure lets RadixAttention match at any token boundary.

The practical gap appears at the edges. RadixAttention can reuse a prefix of any length, down to the token. vLLM can only reuse whole blocks, so the last partial block, whatever doesn't fill out a clean 16-token stride, gets recomputed every time regardless of how much of it might already sit cached elsewhere. On short prompts that overhead barely registers. On long, block-heavy contexts it adds up fast, and this is the clearest architectural tradeoff between the two systems: vLLM gives up some hit-rate precision in exchange for a flatter structure that's cheaper to maintain and easier to reason about at scale.

GPU memory limits and the CPU and NVMe offloading tiers

GPU HBM runs out fast. Serving LLaMA-70B across four A100s leaves room for something like 36,000 cached prefix tokens, which sounds like a lot until you weigh it against the volume of reusable KV state a real production workload throws off in a single afternoon. Once HBM fills, the system needs somewhere else to put blocks it isn't ready to throw away. That's the memory hierarchy kicking in: GPU HBM at the top, CPU DRAM below it, NVMe SSD below that. Each step down trades latency for capacity, and cost per gigabyte drops accordingly.

vLLM's native answer is the OffloadingConnector, shipped starting in v0.11.0, which moves KV cache blocks to CPU memory without blocking the GPU while the transfer happens. The vLLM team's own figures put the payoff at a range spanning roughly double to over twenty times the reduction in time-to-first-token, with throughput gains reaching several times over under concurrent load and high cache hit rates. Those aren't small numbers, and they're the reason offloading has become a default expectation in serving stacks rather than some niche optimization bolted on for edge cases.

Below CPU DRAM sits NVMe. When a request needs a prefix that's already been evicted all the way down to disk, the system pulls the KV blocks back from NVMe instead of recomputing them from scratch during prefill, and that's still cheaper than a full prefill pass even with disk latency factored in. The tier exists for one reason: retrieving cached blocks from NVMe can avoid the overhead of a full prefill recomputation.

How RDMA moves KV cache between disaggregated prefill-decode engines

Prefill and decode have opposite resource profiles, so splitting them onto separate hardware pools, letting each scale and get scheduled on its own terms, is the obvious next move. This is prefill-decode disaggregation, and it's now running across vLLM, SGLang, and Dynamo.

Disaggregation only pays off if the KV cache tensors computed by the prefill workers reach the decode workers fast. Slowing that transfer down doesn't remove the bottleneck, it just relocates it. The interconnect is the whole game here, full stop.

NVIDIA's answer, open-sourced at GTC 2025, is NIXL (NVIDIA Inference Transfer Library), a point-to-point transfer library that supports multiple backends: UCX (covering RDMA over InfiniBand, RoCE, and plain TCP), GPUDirect Storage, NVMe, S3-compatible object storage, Azure Blob Storage, and others. In Dynamo, NIXL moves KV cache directly from the prefill engine's VRAM into the decode engine's VRAM, and the transfer is designed to minimize pipeline stalls as the data moves between engines. RDMA read and write operations let the prefill worker reach into remote KV blocks, or write to them, moving cache data at network speed with minimal interruption to either engine. The cache moves at network speed, with minimal interruption to either engine, instead of stalling the pipeline every time work crosses from one side to the other.

Prompt deduplication as a complementary layer that operates before caching sees the request

Caching and deduplication get talked about as if they're the same idea wearing two names. They aren't, and confusing them costs you savings on the table. The distinction is straightforward: byte-exact pre-prompt deduplication operates on repeated chunks before the prompt is assembled, while caching operates on the assembled prompt's prefix, or on its KV-tensor representation, after the fact.

Mechanically, the two exploit different kinds of repetition. Prompt caching exploits the fact that many separate calls share the same prefix. Deduplication exploits the fact that a single call might contain repeated chunks within itself: the same retrieved passage appearing more than once within a single context window. Feeding a deduplicated prompt into a caching backend makes the savings stack. You get both, not just whichever one happens to be larger.

That has a direct operational consequence. Deduplication shrinks the token count that ever reaches the prefill phase. Caching then avoids recomputing whatever tokens do reach prefill. Neither one covers for the other: skip deduplication, and caching still processes the duplicate chunks fresh every time they occur in a new arrangement.

RAG is the workload where this appears most clearly. The same retrieved document chunks appear across a huge share of requests, especially with a small, high-traffic document set. Deduplication collapses the repeated chunks before the prompt gets assembled, and caching then reuses the KV blocks for whatever's left. Running only one of the two leaves money on the table, and in a RAG pipeline hitting the same handful of documents thousands of times a day, that's not a rounding error.

Where prefix caching breaks down: agentic workloads, branching, and position shifts

Prefix caching and RadixAttention both rest on an assumption that's easy to miss until it stops holding: prompts arrive once, the cache only grows, and content sitting in the cache stays exactly where it was first written until the request finishes. That assumption holds fine for single-turn completions and even for straightforward chat. Agents break it.

Two distinct failures happen here, and they call for different fixes. First, content shifts to a new absolute position in the sequence: a multi-turn agent re-renders its entire conversation history on every turn, and as new tool outputs land, content that was already cached moves to a new spot. Exact-prefix matching, which is what both vLLM and RadixAttention rely on, can't recognize that shifted content is identical to what it already has cached, even though the underlying KV values would be perfectly reusable if the position simply lined up. Second, an agent sometimes needs to actively edit its own context: drop a failed tool call's output, summarize a block that's gone stale, and keep going without re-running prefill on everything downstream of the edit. No existing vLLM primitive supports that. Production agentic harnesses fall back to full re-prefill whenever an edit like this happens, which is exactly the computation prefix caching was built to eliminate.

Leyline (arXiv:2606.01065v1) proposes a serving-side fix aimed squarely at letting agents edit their own context without re-running prefill on everything downstream: a declarative directive, expressed as a (span, replacement) pair, paired with a splice kernel that reuses whatever prefix work the edit would otherwise throw away. The paper reports that splicing lifts replay cache-hit rate by a substantial number of percentage points and cuts latency by up to 241 milliseconds. A ten-line truncation rule routed through the same interface reportedly lifts agentic solve rate by a substantial margin on the debug-gym benchmark, a meaningful jump for a change confined entirely to the serving layer rather than the model itself.

A separate proposal, KV Packet, tackles a related but distinct issue specific to RAG: standard KV caches are context-dependent, so a document's cached KV state, computed once, can't get dropped into a new prompt surrounded by different retrieved chunks without recomputing it, since the attention values were derived relative to whatever else sat in context at the time. KV Packet wraps frozen document caches in trainable soft-token adapters, aiming to make a cached document's KV state usable across different surrounding contexts without a fresh prefill pass each time. Both proposals point at the same gap: prefix caching, as it exists today, is built for append-only, static contexts, and agentic serving is neither.