Data Residency Strategies for Multi-Tenant Inference
Prefix caching in multi-tenant LLM inference leaks prompts through timing side channels.
Data residency in multi-tenant LLM inference is not settled by which region handles a request. That model holds for request bodies and response payloads, which exist for the duration of a call and then disappear. It stops holding the moment prefix caching is turned on, because the KV cache is stateful, and it persists tenant data across requests and across sessions, long after any individual call has finished.
The KV cache stores intermediate attention states, the key and value tensors computed from every token in a prompt, so that a later request sharing the same prefix can skip recomputing them, and those tensors are a derivative encoding of confidential input that behaves accordingly.
If you want residency right, you have to start by giving the cache its own architecture, not the request path.
The Timing Side Channel as a Prompt Reconstruction Tool
A prefix cache keyed only on the content of the tokens, which is the default behavior in vLLM's automatic prefix caching and in SGLang's RadixAttention, functions as a timing oracle. Repeated probing across a vocabulary of guesses lets that co-tenant reconstruct another tenant's prompt, block by block.
The mechanism rests on a simple latency gap. On an NVIDIA A100 running Qwen2.5-7B-Instruct with a shared 2,119-token prefix, KVGov's authors measured a cold TTFT of 149.6 milliseconds against a cached TTFT of 32.8 milliseconds. That gap is large enough that any observer watching the API, without any special access, can tell hit from miss with confidence.
Three independent research efforts have turned that gap into a working attack. PROMPTPEEK, from Wu et al. in 2025 and demonstrated at NDSS 2025, fingerprints a victim tenant's cached prompts by probing TTFT across a known vocabulary, and KVGov reports it reaching 99 to 100 percent attack success rate against SGLang's radix-tree cache. EarlyBird, from Song et al. in 2024, reconstructs prompts token by token using TTFT timing at single-token cache granularity, reported at 100 percent success against both vLLM and SGLang; vLLM's 0.26.0 V1 engine raised the cost of this particular attack to infeasibility for free-text prompts by dropping block_size=1, though that protection does not extend to template-structured prompts where the attacker only needs to recover a bounded field. InputSnatch, from Zheng et al. in 2024, attacks both prefix caching and semantic caching, and against prefix caching in vLLM, TGI, and TensorRT-LLM it reconstructs structured prompts field by field, reaching 62 percent success on disease extraction under realistic rate and time limits.
All three attacks trace back to the same design choice. Timing obfuscation through TTFT padding or injected jitter does little to stop this, because the attacker only needs a statistically reliable separation between hit and miss distributions, not precise latency values. SafeKV notes that padding hit responses toward the miss-latency tail erases most of the benefit of prefix reuse in the first place and drives up P95 and P99 TTFT for every tenant, not just the targeted one.
The exposure scales with prompt length. That scaling is what pushes the conversation from a security footnote toward a capacity and architecture question, because the same long-context trend driving these attacks is also the trend straining GPU memory.
Long-Context Workloads and the Limits of Single-GPU KV Storage
KV cache memory grows linearly with context length and with the number of concurrent users, and at the context lengths now common in production, a single tenant's cache can take up a substantial share of a high-end GPU's entire HBM capacity.
The physics here shows why nobody can just turn caching off to sidestep the problem.
Multi-tenant serving adds further pressure on top of long context alone. The conclusion that follows is structural rather than optional: offload tiers in CPU DRAM and NVMe SSD are not a performance tuning choice for long-context multi-tenant inference. They are the only mechanism you have for keeping many concurrent sessions alive without constant, expensive recomputation.
The three-tier memory hierarchy that multi-tenant KV offload introduces
It creates several persistent copies of a tenant's prompt state scattered across a cluster, and because the NVMe tier survives a server restart, isolation has to be enforced where the data is stored. The three-tier hierarchy of GPU HBM, CPU DRAM, and SSD is now a standard pattern in production serving, and hybrid memory management is treated as its own distinct category of optimization in the broader survey literature on KV cache systems.
When a cache hit occurs, the ideal read path bypasses the CPU entirely, using GPU Direct Storage to land data straight into GPU memory, which minimizes host overhead but depends on the storage tier correctly identifying which tenant's block is being requested.
That identification step is where the leak originates. vLLM and LMCache both identify cache blocks by hashing the token sequence itself, so the hash depends only on content, not on who sent it. Two tenants sending an identical system prompt will, by default, share the same cached block, which is the entire reason prefix caching delivers the efficiency it does at scale. That same mechanism is what creates cross-tenant leakage the instant the hash is not scoped to a principal.
Disaggregated prefill extends the hierarchy one tier further. The cache key is the identity assertion that has to travel through every one of these tiers. If that key is not scoped to a principal at the moment it is created, no access control applied downstream, at the CPU tier, the NVMe tier, or the network tier, can recover the isolation that was never built in at the source.
Cache key namespacing for tenant isolation without abandoning prefix reuse
The root cause behind cross-tenant KV leakage is the absence of any binding between the cache key and the principal who issued the request. Fixing that binding at the key level, rather than disabling caching outright or layering on timing obfuscation, is the approach that preserves most of the performance benefit prefix caching was built to deliver.
KVGov's core mechanism is HMAC-keyed namespace isolation. A per-principal salt, defined as σ_p = HMAC_K(secret, principal_id), seeds the block-hash chain, so that cache keys become cryptographically disjoint across principals. Under this scheme, a hit on tenant A's block is mathematically unreachable by tenant B's queries, which removes the timing signal at its source.
Where the salt gets injected matters enormously. KVGov's engineering insight is to inject the salt at the divergence boundary, the point where two tenants' prompts first differ. That placement keeps an estimated 93 percent of the prefix-cache benefit, and it leaves no cross-principal timing signal exposed. The framework also finds a tipping point in adversary prevalence below which global caching stays stable, giving operators a principled threshold for deciding when cache scope needs tightening.
SafeKV approaches the same problem from a different angle, with a three-tier asynchronous detection pipeline that separates privacy classification from the inference path itself, a radix-tree memory manager with sensitivity-aware eviction, and a runtime safeguard guided by a Reuse Diversity Ratio that detects and bounds any residual leakage. Whichever mechanism is used, the scoped key has to survive the full trip through the hierarchy described above. A block correctly namespaced in GPU HBM has to carry that same namespace when it migrates to CPU DRAM, when it is written to NVMe, and when it is transferred across the network. The namespace is a property of the block's identity, not a filter applied at lookup time.
What per-user cache isolation costs in practice
Full per-user isolation, in which no KV block is ever reused across tenants, closes the timing side channel completely. SafeKV's design starts from this trade-off directly: isolating per user removes the root cause of leakage by preventing one tenant from observing another's cache hits, but it eliminates cross-user prefix reuse entirely, and for workloads where tenants genuinely share long common prefixes, that efficiency loss is substantial.
SafeKV's answer is to classify blocks. A unified radix-tree memory manager with path compression handles both the shared and the isolated namespace within a single data structure, and sensitivity-aware eviction prioritizes keeping the shareable blocks in cache, recovering hit rate that a blanket isolation policy would otherwise sacrifice.
Offload tiers and disaggregated prefill extending the isolation perimeter beyond the GPU
Once a KV block leaves GPU HBM, whether through offload to NVMe or through transfer to a decode worker over RDMA, it enters infrastructure built for capacity and throughput. A correctly scoped cache key is necessary at that point but not sufficient on its own. Credible residency claims also require encryption at rest, TTL policies tied to session lifecycle, and explicit deletion guarantees.
The NVMe tier changes the nature of the obligation. Disaggregated prefill adds a second concern: KV tensors produced during prefill have to reach the decode worker before generation starts, and that transfer moves across shared network fabric, either RDMA or NVMe-oF, that other tenants' traffic also uses. The physical path a block takes matters for isolation as much as for performance: NVLink within a domain, InfiniBand across nodes, and TCP across data centers carry very different bandwidth characteristics, and that path determines what shared infrastructure the isolation boundary actually has to protect.
Hardware-accelerated cache placement, such as NVIDIA's BlueField-4 DPU, used in CMX (formerly known as ICMSP) to eliminate metadata overhead and enforce secure, isolated access from GPU nodes, can enforce this namespace scoping at the data path level, reducing the chance that a software bug in cache-key computation translates into actual cross-tenant access at the storage layer.
Cache-aware load balancing and data residency constraints in multi-region deployments
Residency regulations that confine a tenant's data to a particular region are a real constraint on cache design, but routing is not the mechanism that satisfies the constraint. A cache hit is only possible when a request lands on the node holding the matching prefix, so a prefix-hash consistent routing layer, one that consumes cache events from the inference engines and tracks which prefixes live on which nodes, is a necessary piece of infrastructure for realizing the benefit of isolation-preserving caching across a cluster.
Sending a request across a regional boundary does not produce a partial cache miss, it produces the same outcome as if the cache did not exist for that request, because the KV state generated on one region's GPUs is not reachable from another region's nodes without an explicit cross-region transfer, and that transfer may itself violate the residency requirement it was meant to respect. The sound engineering response is to make the regional cache tier as efficient as it can be within its boundary, which in practice means sizing NVMe offload capacity per region large enough to extend the effective cache pool well past what GPU HBM alone could hold, narrowing the performance gap between a constrained regional deployment and an unconstrained global one.
A deliberate KV cache residency architecture end to end
A multi-tenant KV cache residency architecture is a set of design decisions that stay consistent across key computation, block placement, offload tier management, network transfer, and load balancing, all of them anchored to the same principal identity. A gap anywhere in that chain produces either a leakage path or a performance penalty, and there is no tier downstream that can repair a mistake made upstream.
At the cache key layer, cache keys should bind to an authenticated principal identity using HMAC-keyed namespacing, following KVGov's approach, with the salt injected at the divergence boundary rather than the chain root to preserve shared-prefix efficiency. Blocks should be classified as sensitive or non-sensitive, following SafeKV's selective isolation model, so that genuinely shared prefixes are not forfeited for the sake of uniform caution. That principal identity and sensitivity classification need to travel as metadata through every tier the block passes through.
At the GPU memory tier, per-tenant memory quotas prevent one tenant's large batch from exhausting HBM and triggering cascading preemption for every other tenant on the same hardware, and fair-share scheduling should extend to tensor core access, not just to memory capacity. At the offload tier, NVMe-resident KV blocks should be encrypted at rest with keys derived from the principal identity, fusing storage-layer access control and cryptographic protection into a single artifact. Where hardware-accelerated placement is available, as in DPU-backed architectures, it should enforce namespace isolation at the data path level rather than leaving that enforcement to software alone.
At the network transfer tier, scoped block identity needs to survive RDMA and NVMe-oF transfers, since disaggregated prefill moves tenant data across shared fabric and the transfer library has to treat principal identity as a first-class attribute of the payload, not an afterthought. At the routing layer, a prefix-hash consistent router that consumes cache events and maximizes hit rate within the permitted regional boundary keeps residency and performance from working against each other, and you close the efficiency gap residency constraints impose by sizing regional NVMe offload capacity to compensate for a smaller cache pool.
None of this architecture works if principal identity breaks down at any single point. Block placement, encryption key derivation, TTL enforcement, and regional bandwidth capacity are themselves storage problems, and whether the isolation guarantees described throughout this piece are real in production or merely nominal on a design document depends on getting these right.
