Prefill and Decode Disaggregation in LLM Serving

Prefill and decode are not two flavors of the same job. Prefill reads an entire prompt at once and runs dense matrix math across every token in parallel, so it lives and dies by how many FLOPs the GPU can push. Decode generates one token at a time, and each step has to read back the full key-value cache of everything generated so far, so it lives and dies by memory bandwidth, not compute. That split is the entire reason prefill-decode disaggregation exists as a serving architecture, and understanding it, along with the cost of moving KV cache between the two phases, separates a real production design from a diagram on a whiteboard.

The asymmetry appears directly in hardware choice. H100 and B200-class GPUs are built for the compute density prefill wants: high tensor core throughput, dense matmuls, everything running hot. H200-class GPUs, with roughly 4.8 TB/s of HBM bandwidth, are built for decode, where the bottleneck is how fast you can pull KV cache off memory, not how many multiplications you can do per second. Time to First Token (TTFT) is a prefill metric. Inter-Token Latency and Time Per Output Token are decode metrics. Keep that mapping straight, because it drives every SLO decision later in this piece.

Continuous batching, the scheduling approach most serving stacks use today, interleaves prefill and decode steps inside the same loop to keep the GPU busy. It's a genuinely good trick for utilization, and it's also the wrong long-term fix, because it papers over the resource mismatch instead of solving it. Dropping a 32K-token prompt into a batch that's mid-generation for a dozen other requests stalls every one of those decode steps behind that prefill's compute demand. That's head-of-line blocking, and no amount of clever scheduling inside a single node makes it go away. It just moves the pain somewhere else in the stack.

What the KV cache costs and why it becomes the load-bearing object in disaggregation

The KV cache is not a rounding error, and treating it as scratch space is the mistake that sinks most first attempts at disaggregation. For a large model with 80 layers, 8 KV heads, and 128 dimensions per head at FP16, each token costs 327,680 bytes of cache. Running that out across a 4K-token prompt means holding 1.34 GB of state before generation even starts. At FP16, a 70B-class model runs roughly 300 to 350 KB per token, and a single 128K-token session can approach 40 to 45 GB, and that's one session, before any concurrency gets layered on top.

In a lot of production deployments today, the KV cache eats more memory than the model weights do. The thing serving frameworks used to treat as scratch space is now, in most large-context deployments, the bigger memory citizen on the GPU, and any capacity plan that still budgets for weights first and cache second is planning against the wrong constraint.

Left unmanaged, that memory gets wasted badly. Traditional inference systems burn 60 to 80% of allocated KV cache memory on fragmentation, the same problem that plagued naive memory allocators for decades. PagedAttention, introduced by Kwon et al. at SOSP 2023, fixes this with virtual block tables borrowed straight from a technique used in operating-system memory management, cutting waste to under 4% and delivering 2 to 4x throughput gains in the process. It's one of the few ideas in this space that's aged into being simply "how it's done," and any serving stack that skips it is leaving throughput on the table for no good reason.

Once prefill and decode split onto separate nodes, the KV cache computed by the prefill node is the payload. It's large, it's structured, and the decode node cannot generate a single token until enough of it arrives. Moving the phases apart without thinking hard about that transfer does not remove a bottleneck. It relocates it from GPU contention to a data movement problem, and that new bottleneck can be worse than the one it replaced.

The request flow end to end in disaggregated serving

The architecture itself is simple to state: two separate pools of GPUs. Prefill nodes take the incoming request, run the full forward pass over the prompt, and produce a KV cache. Decode nodes take that cache and generate tokens until the response is done. A request moves from the router to a prefill node, gets its KV cache built, that cache transfers to a decode node, and the decode node streams tokens back out.

The router in this setup does real engineering work, not load balancing on autopilot. It has to track KV cache state across the fleet, not just which node has a free slot, because routing decisions decide whether an existing cache can be reused and how much transfer cost a given request is about to take on.

A handful of research systems established this pattern well before it showed up in production stacks. DistServe (Zhong et al., OSDI 2024) frames the problem as optimizing "goodput," tuning resource allocation and parallelism separately for each phase and placing nodes to minimize communication overhead based on available bandwidth. Splitwise (Patel et al., 2024) tested both homogeneous and heterogeneous GPU configurations, exploring the trade-offs between throughput, cost, and power across different deployment scenarios. Mooncake (Qin et al., 2024, later published in ACM Transactions on Storage in November 2025) centered its approach on the cache holding key-value pairs, treating idle CPU, DRAM, and SSD capacity as a distributed cache store and adding early rejection under heavy load so the system fails cheap instead of burning compute on requests that won't finish in time.

The payoff beyond raw latency is independent scaling. Long-prompt workloads get more prefill nodes. Long-generation workloads get more decode nodes. Nobody has to scale the two together as one undifferentiated GPU pool anymore, which is exactly the constraint that made single-node serving so wasteful at scale. Reporting from Jarvislabs found that Meta, LinkedIn, Mistral, and HuggingFace were already running vLLM with disaggregated serving in production by 2026, not as an experiment.

Chunked prefill as the intermediate option before committing to full disaggregation

Full disaggregation is not the only rung on this ladder, and treating it as the automatically "correct" answer is a mistake teams make when they read the throughput numbers and skip the operational cost. Chunked prefill, the technique behind Sarathi (Agrawal et al., 2023), breaks a long prompt into fixed-size chunks and interleaves them with ongoing decode steps on the same GPU. Decode never gets fully blocked the way it does under naive continuous batching, but the two phases still share one GPU's compute and memory bandwidth. The contention gets bounded, not eliminated.

vLLM exposes this directly through --enable-chunked-prefill and --max-num-batched-tokens, and the appeal is obvious: single node, low operational overhead, nothing new to deploy. The throughput gain is real but modest, roughly 20 to 40%, against the 1.5x to 2.5x that full disaggregation can deliver, and that gap widens as prompts get longer and concurrency climbs.

So the decision comes down to workload shape, not sophistication, and most teams get this backwards by defaulting to whichever option sounds more advanced. Chunked prefill fits prompts generally under 8K tokens, single-node setups, and teams that aren't ready to stand up cross-node KV transfer infrastructure. Full disaggregation earns its complexity once prompts consistently run past 8K tokens and concurrency is high enough that chunking stops moving the needle. Neither one is the serious choice with the other as training wheels. They sit at two different points on the same complexity-versus-performance curve, and the right one depends on what's actually hitting the servers.

The KV transfer problem: what has to move, over what, and at what cost

The thing moving between prefill and decode nodes is not a small handshake message. For a 128K-token context on a 70B model, it's tens of gigabytes of KV state, and none of it is optional: the decode node cannot produce a first output token until enough of that cache has landed. That transfer has to be non-blocking on both ends, so the GPU forward passes on the prefill and decode sides keep running while the data moves in the background. A blocking transfer defeats the entire point of splitting the phases.

Whether this works comes down to the transport layer. NVLink gives the highest bandwidth and lowest latency but only inside a single node or an NVLink-connected multi-node setup. InfiniBand RDMA is the standard cross-node path for this kind of traffic, moving data GPU-to-GPU without pulling the CPU into the loop. TCP is available everywhere, which is its only real advantage here: When NIXL falls back to TCP the latency penalty is steep, and workloads that care about TTFT feel it immediately.

NIXL, the NVIDIA Inference Xfer Library used inside NVIDIA Dynamo, gives serving stacks one consistent API for non-blocking, non-contiguous data movement across memory and storage tiers, picking the best transport available on its own. Spheron reports that it's the standard mechanism both vLLM and Dynamo lean on for this exact transfer.

Storage is not a bystander in any of this either. When KV cache can't fit entirely in GPU HBM during a transfer, it has to spill to CPU DRAM or local NVMe, and at that point the storage tier sits directly on the critical path for latency, not off to the side as a background concern. The whole architecture rests on one condition: if the network or storage layer can't sustain the bandwidth the prefill side produces, disaggregation just trades GPU contention for a data movement bottleneck. The transfer layer has to be engineered to match compute throughput, full stop, or nothing else about the design matters.

Production implementations of disaggregation: NVIDIA Dynamo, vLLM, SGLang, and LMDeploy

NVIDIA Dynamo, announced at GTC 2025 and reaching general availability as Dynamo 1.0 at GTC on March 16, 2026, sits as an orchestration layer above inference engines like vLLM, SGLang, and TensorRT-LLM rather than replacing any of them. It coordinates those engines into a multi-node disaggregated system. NVIDIA's own announcement claims up to 30x more requests served running DeepSeek-R1 on Blackwell GPUs under Dynamo, compared to serving without it.

Its KV Block Manager runs a three-layer design: a model integration layer connecting into TensorRT-LLM and vLLM, a memory management layer, and a storage and transfer layer built on NIXL that reaches CPU memory, SSD, file systems, and cloud storage. The router evaluates KV cache overlap and per-worker load, supporting both aggregated routing and disaggregated routing where requests get split between prefill and decode pools based on cache reuse, not round-robin assignment. Kubernetes-native distributed inference built on vLLM, introduced at Red Hat Summit 2025, addresses the same cross-node KV transfer challenge.

vLLM exposes a KVConnector interface that lets its paged KV cache talk to external storage systems, with a NixlConnector implementation handling the disaggregated transfer path. LMCache sits between vLLM and external storage, whether that's host memory, local drives, or a distributed store, so the inference backend never has to know or care what's underneath it.

SGLang supports disaggregation through its router using a --disaggregation-mode flag, with a topology of one prefill worker, one decode worker, and a router process tying them together. Its RadixAttention design gives it a real edge on prefix-heavy workloads, where shared prompt prefixes can be cached once and skipped on later prefill passes. As of v0.5.15, SGLang defaults to breakable CUDA graphs, cutting graph build time on the variable-length requests a dedicated prefill node tends to see.

LMDeploy added prefill-decode disaggregation in v0.9 through DLSlime and Mooncake, landing on an architecture similar in shape to vLLM's NixlConnector approach.

None of these four is strictly better than the rest, and picking one because it's the most talked about is how teams end up with the wrong tool. They differ in routing sophistication, how deep the storage integration goes, and how native they are to operations on a container-orchestration platform. The right pick depends on the priority: single-cluster performance, multi-cluster scale, or squeezing value out of prefix reuse.

Sizing the prefill-to-decode resource ratio for a given workload and SLO

Getting the ratio of prefill nodes to decode nodes wrong is not a minor inefficiency. Too few prefill nodes and TTFT blows past its SLO. Too few decode nodes and TPOT does the same, or you swing the other way and end up paying for idle GPUs sitting on the wrong side of the split. A paper out of Kingsoft Cloud (arXiv:2603.04716) states that the industry doesn't yet have a widely agreed-on method for getting this ratio right, a striking admission given how much production traffic already runs through disaggregated stacks.

The ratio depends on total throughput demand (input and output tokens per second combined), average input length against average output length, and the two latency SLOs pulling in different directions. The Kingsoft Cloud paper models the TTFT constraint using M/M/1 queuing theory to work out achievable prefill throughput under a given latency target, while the TPOT constraint, which caps the usable decode batch size, gets derived empirically instead. NVIDIA's AIConfigurator takes a search-based approach to find good tensor-parallel, data-parallel, and expert-parallel settings for a given SLO, though per the same paper no widely established method yet solves the ratio problem for throughput and SLO jointly.

The practical heuristic is simpler than the math behind it: long-prompt, short-output work like document summarization is prefill-heavy and wants more prefill nodes, while short-prompt, long-output work like code generation is decode-heavy and wants more decode nodes. Even at the frontier, the exact ratio stays a closely held detail. SGLang's production deployment of DeepSeek V3.1 runs on H200 nodes, eight GPUs per instance, but per the Kingsoft Cloud paper, the specific prefill-to-decode instance ratio has never been disclosed. A deployment running at that scale doesn't guard a number like a trade secret unless the math behind it is genuinely hard.

The breakdown of static prefill-decode separation in multi-round and agentic workloads

Everything above assumes a clean, one-time boundary: prefill runs once, the cache moves, decode runs to completion, done. That model holds for single-shot inference, and it starts to crack under multi-round and agentic workloads, where the model is calling tools, reading results back from an environment, and generating across many such rounds. It's calling tools, reading results back from an environment, and feeding those results into another round of generation. Each round can look like its own miniature prefill-decode cycle, and the KV cache from the previous round doesn't just get thrown away. It needs to persist and stay attachable to whatever comes next.

Static disaggregation was built around a request with a clear beginning and end. Agentic loops don't offer that. The boundary between "this is prefill" and "this is decode" gets redrawn every time a tool call comes back with new tokens to process, and a serving architecture built only for the one-shot case has no clean way to represent that repeatedly. Treating this as a minor edge case is the wrong call: it's a sign of how far the architecture still has to stretch to keep up with how models actually get used in practice, and any team betting on agentic workloads at scale needs to plan for that stretch now rather than patch for it later.