KV Cache Lifecycle Orchestration Across Request Queues and Preemption Boundaries

Production LLM serving needs KV cache lifecycle orchestration across request queues and preemption boundaries. Efficient management couples allocation, residency, eviction, offload, and reuse through shared scheduling signals. A tiered path from HBM through CPU DRAM and NVMe to shared storage preserves useful state only when routing, transfer bandwidth, and retrieval timing support the next request.

Why treating KV cache as a passive buffer creates production failures

Documentation often calls KV state “the cache.” At runtime, it is a shared, contended, expiring resource. The simple model allocates a contiguous buffer per request, fills it during prefill, reads it during decode, and discards it afterward. But autoregressive decoding grows that buffer while new requests compete for the same GPU memory. Execution progress and memory consumption are inseparable.

Agent loops make that coupling harder to ignore. UNISON, from Fudan University, documents growing live KV sets across coding and web-agent sessions that retain prefixes through tool waits and repeated model invocations. Idle sessions therefore occupy memory that other requests need to advance. At long-context windows, one user’s KV state can approach or exceed a single GPU’s HBM capacity, turning residence duration and placement into architectural decisions.

The six lifecycle phases every KV block passes through

An explicit state machine takes the place of the passive-buffer model. A block enters phase 1, allocation, when admission reserves memory for its KV state. It enters phase 2, active residency, when the GPU reads and writes it during decode. Phase 3, idle residency, begins when the request waits for a tool result, network response, or scheduler capacity while its block remains in HBM.

Memory pressure moves blocks into phase 4, eviction candidacy, where the scheduler selects state for removal. Phase 5, eviction or offload, begins when a selected block is dropped or moved to another tier. Phase 6, reuse, begins when a session resumes or a later request finds matching saved state and skips recomputation. A saved block is useful for reuse only if it can be retrieved in time for the request.

Why allocation policy determines everything downstream

Allocation decides which requests can make forward progress. Consider a low-priority request that arrives first and takes up HBM before a high-priority agent needs that memory. First-come-first-served allocation ignores their different roles, allowing the agent’s KV state to be evicted and stalling the downstream pipeline. This is critical inversion. The system prompt, tool-call graph, or task priority can expose criticality at admission, but that information has no effect unless allocation consumes it.

UNISON’s Survival-Penalty Eviction for Agent Return-gap, or SPEAR, uses a gap average and turn-indexed hazard rate to inform allocation and eviction jointly. Huawei’s Unified AI Gateway framework extends the same concern to routing, where ignoring cache state can discard accumulated prefill work and force full-context recomputation. A locally available execution slot is not necessarily the cheapest place to continue a request.

An incoming request that shares a resident prefix can use that block through copy-on-write, avoiding allocation of fresh state. The allocator must consult the reuse index first. With chunked prefill, it must also track partially allocated blocks as incomplete so that interleaving prefill with other requests’ decode steps does not expose unfinished state for reuse.

Eviction policy fragility under real workload patterns

Even with correct allocation, a poor eviction policy can cause problems. Least recently used, or LRU, assumes that recent access predicts near-term demand. Consider two sessions, one waiting on a tool and another touched more recently but not about to return. Under pressure, LRU can remove the waiting session just before its tool result arrives.

UNISON reports that its hazard-rate-aware policy lowers time to first token by 58%–89% on long-horizon traces by correcting this treatment of live waits as coldness. Multi-tenant and multi-model workloads introduce further tension because load balancing and per-model cache-hit maximization need not favor the same placement. Adaptive Replacement Cache, or ARC, separates recently accessed and frequently accessed blocks into two lists and is integrated into vLLM. SPEAR instead uses what is known about agent return gaps.

LRU is still simple, has low overhead, and is well understood. That is a legitimate advantage, but policy overhead is only one part of the comparison. A wrong eviction forces recomputation whose cost can exceed the original prefill for a long-prefix agent session. Choosing a policy thus depends on how the workload behaves and what it costs to lose state, as well as how fast it can select a victim.

How preemption boundaries force recomputation and what to do about it

Preemption interrupts an in-progress request to reclaim GPU resources, while eviction selects cache state for removal. Good eviction policy cannot prevent every scheduling interruption. Without explicit preservation, the interrupted request must restart prefill when rescheduled. Longer contexts make that loss cost more, and the request’s priority shapes how much the delay hurts downstream work.

UNISON’s Tiering in Idle-window DMA Events, or TIDE, turns observed tool waits into a migration budget. The scheduler signals that a session is entering an idle window, and DMA moves its KV blocks from HBM to a lower tier. Preemption during that wait can then lead to offload-and-restore, avoiding recomputation. Timing is part of the preservation mechanism. The memory controller needs enough bandwidth headroom to migrate the state before the session returns.

The engineering response is to treat preemption as routine. Its cost depends on how long the context is, which offload tiers are available, and whether migration began before the need for memory became urgent. At a prefill-to-decode handoff, transfer failure or decode-node memory pressure can also discard prefill work. The gateway framework treats cache availability as a routing input so that scheduling accounts for this boundary before committing the request.

Prefill/decode disaggregation as a lifecycle boundary, not just a performance split

vLLM’s MORI-IO connector makes the handoff cost concrete. Its default Write Mode sends layer-by-layer RDMA writes into the decode instance’s pre-allocated KV blocks as each prefill layer completes. Chunked prefill is an exception, with blocks accumulating until the last chunk before writing begins. Transfer can overlap computation, but only within the interconnect’s bandwidth budget. A slower path can turn that handoff into the bottleneck.

A one-directional P→D channel also limits state movement after handoff. Generated-token KV and accumulated multi-turn context on the decode node cannot flow back through that channel, so agentic turns re-traverse the pipeline. Mooncake, from Moonshot AI / Kimi, addresses the lifecycle through a global KVCache and scheduler shared across prefill and decode clusters. It also uses underutilized CPU, DRAM, SSD, and NIC resources to build the disaggregated cache. NVIDIA Dynamo uses an event-driven Mooncake shared-cache index for SGLang HiCache routing, tracking reusable pages through object events.

Disaggregation nevertheless adds transfer cost and architectural complexity. DuetServe and Nexus pursue isolation without the network hop, while TaiChi treats aggregation and disaggregation as choices governed by SLO constraints. Any multi-GPU design still needs a clear policy for transfer and placement, even when the boundary falls somewhere other than between dedicated prefill and decode pools.

The memory hierarchy as a tiered offload path, not a fallback

The storage horizon extends beyond the receiving GPU. HBM holds hot, active KV, CPU DRAM holds warm DMA-accessible pages, NVMe holds cold prefixes, and object storage or distributed cache supports persistent cross-request reuse. Each tier has its own limits on latency, capacity, and bandwidth. Placement must match the expected return time. Sending an imminently needed block to NVMe can stall a request that DRAM residency would have served promptly.

TIDE uses idle windows to reclaim HBM without destroying state. NVIDIA’s ICMSP, or CMX, extends cold-block offload to Ethernet-attached NVMe managed by BlueField-4 DPUs over a Spectrum-X fabric. At long-context scales where one user’s footprint exceeds HBM, cold prefixes require residence beyond GPU memory. NVMe supplies that capacity without making HBM the sole repository of reusable work.

ObjectCache, from ETH Zurich / HPE Labs, demonstrates a more distant runtime tier. It stores fine-grained, hash-addressed chunks in S3-compatible object storage and delivers them in layer-major order over RDMA, overlapping retrieval with GPU computation. ObjectCache reports only 5.6% added latency over local DRAM for 64K-token contexts. That result depends on protocol and transfer co-design. The University of Chicago’s Internet for the KV Cache vision extends the placement question across clouds and datacenters when transfer is cheaper and faster than recomputation.

Prefix reuse as a lifecycle outcome, not a caching trick

Stored blocks produce value only when routing and indexing can find and serve them. Exact prefix matching over token sequences lets requests sharing a system prompt or conversation prefix skip prefill for those tokens. Long, frequently reused prompts offer especially valuable targets. Agent edits disrupt this simple reuse model. Dropping a failed tool call or summarizing stale output can invalidate a prefix match even when useful computation remains.

Leyline, from Erlangen NHR, gives agent policy a serving-side control mechanism. Its declarative (span, replacement) directive specifies an edit and its mode, including in-place splice or prefix-trimmed re-prefill. The serving stack applies a closed-form positional-embedding rotation correction to re-anchor downstream cached slots without full re-prefill. Leyline reports a +14.3 percentage-point lift in agentic solve rate on debug-gym.

ObjectCache makes distributed reuse a hash-keyed lookup independent of the producing node. LMCache similarly demonstrates the need for distributed indexing and placement when sharing cache across expert-parallel workers. When routing switches models, cross-model mapping is still needed as a separate step. Lookup introduces overhead, but hash computation, indexing, and tier-appropriate retrieval can be bounded and parallelized, unlike the full recomputation on which later work depends. Correctly separated state makes compute workers fungible, providing a basis for horizontal scaling.

Compression as a lifecycle-phase multiplier, not a storage optimization

Smaller blocks need less storage and transfer bandwidth, which can allow migration to finish within an idle window. A compressed block that fits in CPU DRAM rather than spilling to NVMe also changes the cost of resuming its session. Compression changes the placement and retrieval decisions available to the scheduler.

NVIDIA’s NVFP4 format, introduced with Blackwell on B200, B300, and RTX 50-series hardware, makes hardware-accelerated 4-bit floating-point KV storage viable where prior architectures did not. Quantization at 4 bits is harder than FP8, so choosing a format is still part of the design. Lossless compression has a different opportunity in the statistical structure of BF16 and FP8 values, including exponent concentration. SplitZip, a 2026 research system, targets fast lossless compression for disaggregated serving, where compression and decompression must fit within the transfer window.

Too much compression can add needless decompression work on the hot path, while too little can leave blocks too large for the tier they are meant to use. The same dependency applies to geographic reuse, where shrinking a transfer can change whether preservation is economically feasible. Compression cannot be selected independently of the offload and transfer schedule it is meant to support.

Orchestrating the full lifecycle without isolating its policies

The common scheduling signals are session state, expected return time, prefix hash, block size, tier occupancy, and available migration bandwidth. SPEAR-style eviction can still lead to critical inversion when used with first-come-first-served allocation. Fast NVMe offload without idle-window detection can begin too late. Content-addressed indexing without cross-model mapping can lose reuse value at routing switches.

Mooncake couples global placement, prefill/decode disaggregation, and SLO-aware scheduling around KV state. UNISON couples SPEAR and TIDE through one live ranking and reports a 22%–51% reduction in average memory access time when they are jointly optimized. Leyline brings application intent into reuse decisions, while the gateway framework jointly selects a model, execution site, and cache action. These systems implement complementary portions of lifecycle orchestration. The remaining gap is that allocation, eviction, and offload often communicate through memory state and lack shared scheduling signals.

Storage infrastructure sets the physical limits of that coupling. A correct placement decision is unusable if the chosen tier cannot deliver the block before the request resumes. The engineering direction is to instrument each lifecycle transition, surface its scheduling signals, and connect them to the policies that consume them. Those policies must operate within measured storage throughput, latency, and bandwidth constraints.