KV Cache Memory Layout on Multi-GPU Hosts

KV cache is not overhead you pay once and forget. It grows with every token generated, every layer in the model, every attention head, every request running at the same time, and it sits in memory for as long as that request stays alive. Get the layout wrong across a multi-GPU host's memory tiers, from HBM down through CPU DRAM, CXL, and NVMe, and the serving system spends more time moving bytes than computing anything useful. Get it right, and the same hardware serves more users at longer context lengths without adding a single GPU to the rack.

The math behind this is not complicated, but it does not forgive mistakes. KV cache size follows a fixed formula: 2 × L (layers) × H_kv (KV heads) × D (head dimension) × S (sequence length) × B (batch size) × bytes per element. Every variable is a knob someone turns, and every turn costs something somewhere else. A single Llama 3.1 70B request at 128K tokens of context needs roughly 40 GB of KV cache. The model's own weights at FP16 run about 140 GB. Neither number fits on one 80 GB H100 by itself, and they certainly don't fit together.

Published scaling figures show how fast this compounds once concurrency enters the picture. At 4K tokens with one user, expect around 1.3 GB of KV cache; push to eight concurrent users at that same context and it climbs to about 10.7 GB. At 16K tokens, one user costs roughly 5.4 GB, eight users about 43 GB. By 32K tokens with eight users, the cache alone hits around 86 GB, already past what the model weights take up. At 128K tokens with eight users, it's around 343 GB, nearly two and a half times the size of the model itself. None of this levels off, because the formula scales with sequence length and batch size at the same time, and the workloads pushing hardest right now, agentic loops and multi-turn chat sessions, drag both variables up together. A faster GPU does nothing to fix this. The fix is understanding what shapes these tensors in the first place and deciding, on purpose, where each byte sits.

The determination of KV tensor geometry by attention architecture before a single byte is allocated

Layout decisions start with the attention mechanism itself. Three common variants produce wildly different memory footprints per token, and no amount of clever tiering rescues a model that stores more than it needs to. Picking the wrong attention architecture means solving a hardware problem that a software choice already created, and no amount of tiering after the fact makes up the difference.

Multi-Head Attention, the original formulation, keeps a separate key and value vector for every attention head. It's the most expensive option by design, and at this point it belongs in the "don't do this" column for anything running at real context length. DeepSeek-67B, built on MHA, costs roughly 400 KB of KV cache per token, which sets something close to a ceiling for what "unoptimized" looks like at that model size.

Grouped-Query Attention cuts that cost by letting groups of query heads share a single set of K/V parameters, saving memory in proportion to the ratio of query heads to KV groups. Llama 3.1 70B runs GQA with 8 KV heads against 64 query heads, a steep reduction, though that ratio belongs to this model's configuration, not to some guarantee GQA makes across the board. In BF16, Spheron's 2026 numbers put per-token KV cost at about 0.327 MB for the 70B model and roughly 0.131 MB for the 8B version. GQA shrinks the tensor, but it doesn't compress the information sitting inside it. Running large batches at long context still saturates HBM. It just takes longer to get there than it would under MHA.

Multi-Head Latent Attention goes further still: instead of storing full K and V vectors, it compresses the hidden state into a low-rank latent vector and caches only that, reconstructing the full heads through a projection step at attention time. DeepSeek-V2, which uses MLA, brings per-token cost down to about 27 KB, a 93.3% cut from the MHA baseline set by DeepSeek-67B. The list of models shipping with MLA has grown fast: DeepSeek V3 and a growing number of recent models use it, with broader adoption tracked across the field.

The open-sourced DeepSeek-V2 modeling code actually caches the full, decompressed KV tensor rather than the compressed latent vector, because reconstructing K and V from the latent on every retrieval adds latency that's hard to hide during decode. MLA's memory savings look great on paper. Capturing them at inference time means engineering around the decompression cost, not flipping a switch and walking away, and skipping that work is how a team ends up running MLA in name only, with none of the memory benefit to show for it.

That gap creates a quieter problem for anyone running a fleet across multiple model families. Ganjihal's analysis shows that general-purpose serving frameworks sizing memory off assumptions built for an older attention design, without accounting for MLA's real footprint, can over-provision memory by a large multiple. Running MHA, GQA, and MLA models under one static allocation rule is a mistake, full stop: the tensor geometry differs too much between them for a single rule to fit all three. A sizing engine aware of the underlying architecture is essential here. It's the thing standing between a deliberate memory layout and pure guesswork.

PagedAttention and block sizing: the foundational layout mechanism and its architecture-specific tuning

Before PagedAttention, serving systems reserved memory for the maximum possible context length as soon as a request arrived, regardless of whether the request ever used it. The work that became the foundation of vLLM replaced that with on-demand block allocation, borrowing directly from how operating systems handle virtual memory paging. Blocks get allocated as tokens actually accumulate, not before.

The detail that matters for everything downstream: these blocks don't need to sit next to each other physically. That single design choice is what makes moving KV data between memory tiers, evicting cold blocks, and prefetching warm ones workable in practice, instead of demanding a rewrite of the memory layout every time something shifts.

Block size is not a one-size-fits-all setting, and treating it as one wastes bandwidth in one direction or bookkeeping cycles in the other. Ganjihal (arXiv:2604.26968) lays out architecture-aware defaults: 512 tokens per block for MLA, 128 tokens per block for GQA and MQA, and 64 tokens per block for MHA. That granularity sets the unit for eviction, prefetch, and transfer across the whole system. Too small a block, and the bookkeeping overhead of tracking millions of tiny units eats the savings. Too large a block, and partial reuse wastes bandwidth, since a request might only need a fraction of what got fetched.

MLA's larger 512-token block is not an arbitrary round number. It falls straight out of the math: because each token costs so much less under MLA (that 27 KB figure from DeepSeek-V2), packing more tokens into a single eviction or transfer unit still keeps the block a manageable size in absolute bytes. PagedAttention's block model is the substrate everything else in this piece rests on. Tiering, prefetching, and inter-GPU KV transfer all operate on blocks, not on raw tensors, so understanding how blocks get sized and moved comes before reasoning about anything further down the memory hierarchy.

The GPU memory hierarchy a multi-GPU host provides, and what each tier costs in latency

On-chip SRAM is the fastest memory a GPU has, but it holds active compute tiles during kernel execution, not the accumulated KV cache. It's technically part of the memory hierarchy and irrelevant to residency decisions all the same.

HBM is where the real decisions start. An H100's HBM3 offers 80 GB of capacity at 3.35 TB/s of bandwidth, and it's the reference tier against which every other option in Ganjihal's analysis (arXiv:2604.26968) gets measured. The H200 steps that up to 141 GB of HBM3e, meaningfully more headroom, and That extra capacity makes it a natural choice for long-context serving on a single node.

Below HBM sits CPU DRAM: pinned host memory reached over PCIe DMA, with GPU-observed latency in the 1 to 5 microsecond range. That's roughly an order of magnitude slower than HBM, but it buys around five times the capacity at a much lower cost per gigabyte, exactly the trade a tiering strategy wants for data that isn't needed on every single decode step.

CXL 3.0 memory pools sit in an odd middle spot: low-latency enough to be useful for GPU memory expansion. Research cited in arXiv:2511.00321 shows that offloading to CXL can cut GPU memory usage by a meaningful fraction while still meeting latency targets, with sub-100 nanosecond access latency demonstrated using a CXL memory controller built for the job.

NVMe, reached through GPUDirect Storage, trades more latency (tens of microseconds) for large capacity at low cost. That makes it the right home for KV blocks that run warm or cold and can be prefetched with enough lead time to hide the wait. RDMA fabric is a movement path between nodes, and its role in KV transfer for disaggregated serving comes up later in this piece.

None of these numbers are trivia. The latency hierarchy decides which blocks a system can afford to read synchronously (hot, HBM-resident), which need asynchronous prefetch scheduled ahead of time (warm, living in DRAM or CXL), and which have to be treated as out-of-band retrieval entirely (cold, sitting on NVMe). Within a single node, NVLink 4.0 on the H100 moves data between GPUs at 900 GB/s, the fastest path available for inter-GPU KV movement, and the one tensor parallelism should lean on before any request touches the network.

Tensor parallelism's distribution of KV tensors across GPUs and where the layout choices concentrate

Ganjihal's guidance (arXiv:2604.26968) is blunt on this point: run tensor parallelism across NVLink-connected GPUs inside one bare-metal node before crossing any network boundary. Nothing across a network comes close to competing with NVLink's latency and bandwidth. Teams that lean on network-connected tensor parallelism because a node ran out of NVLink-connected GPUs are trading throughput for a convenience they will regret at scale, and that trade rarely gets revisited until latency numbers force the issue.

Tensor parallelism splits both the model's weights and its KV cache across devices, so each GPU ends up holding a slice of the attention heads along with the KV blocks that correspond to them. Reference configurations across common model families show what that looks like in practice. DeepSeek-V3, at a massive parameter count and built on MLA, carries an extreme weight footprint, but its compressed KV tensor takes some of the pressure off the distributed cache budget. Llama-3-70B and Qwen-2.5-72B, both GQA models, see their KV heads divided across GPUs after sharding. GQA mixture-of-experts models add a further per-expert dimension to the per-GPU head count.

8-way tensor parallelism cuts each GPU's share of the KV cache, but it also cuts the HBM budget available on each GPU once model weights are accounted for. A long-context request can still accumulate KV blocks faster than tensor parallelism divides them down, especially once sequence length climbs into six figures.

Head distribution and block size interact in a way that's easy to overlook. GQA's 128-token block, paired with 8 KV heads per GPU, fixes the eviction and prefetch granularity by architecture, not by anything an operator dials in after the fact. Tensor parallelism doesn't make memory pressure disappear, it spreads it across more devices, and that spreading brings its own cost: GPUs now have to synchronize and track KV block ownership across the group, coordination overhead that simply didn't exist on a single device.

MLA models add one more wrinkle under tensor parallelism. The latent vector, not the full K/V tensor, becomes the unit that gets parallelized, so the compressed cache is sharded across GPUs while the projection back to full attention heads runs independently on each GPU. The compression benefit gets shared. The decompression cost gets duplicated everywhere it's needed.

Read amplification: how memory access granularity mismatches degrade throughput when KV tensors spill beyond HBM

LLM inference runs on tiled matrix multiplication, and those compute tiles are two-dimensional, typically somewhere between 64 and 512 bytes wide, according to TileLens (arXiv:2607.04031). HBM's minimum access granularity is 32 bytes, close enough to that tile width that amplification barely registers under normal HBM operation.

The trouble starts once data moves to Large-Granularity Memory Systems, things like High-Bandwidth Flash and RoMe (which bumps access granularity up to 4 KB), because these systems demand kilobyte-scale minimum reads. Lay a two-dimensional compute tile out as a flat one-dimensional strip in one of these systems, and every memory request pulls in far more data than the tile actually needs.

The direction of the waste depends on layout, and the two common options fail in different ways. Depending on the memory layout chosen, the overfetched data may offer little opportunity for reuse, making the wasted bandwidth unrecoverable. In row-major layout, the overfetched data leads to its own access pattern inefficiencies that degrade parallel execution. Georgia Tech's measurements in the TileLens paper put the damage at roughly one and a half times to several times slower matmul performance on these large-granularity systems, using kernels drawn from Qwen-3 30B and Llama-3.1 70B as test workloads.

TileLens's fix is a tile-major layout: reshape each contiguous block in memory into a 2-D rectangle that lines up with the compute tile's own boundary, so the mismatch never occurs. TileLens-SW extends existing GPU domain-specific languages, letting kernels built on CUTLASS or FlashAttention adopt tile-major layout by changing a layout descriptor rather than the kernel logic itself. TileLens-HW does the equivalent at the hardware level, extending the Tensor Memory Accelerator so that TMA-based kernels, cuBLAS and DeepGEMM among them, get tile-major support with no code changes. Paired with an adaptive hardware prefetcher, the combined approach gets HBF-augmented GPUs (running with a 5 microsecond HBF NAND read latency) within 1% of pure HBM performance on a geomean basis.

Moving KV tensors into NVMe-class or HBF-class storage is a layout decision before it's a capacity decision, and treating it as pure capacity planning is how teams end up paying for a fast tier that behaves like a slow one. Row-major or column-major tensors migrated to these systems without a layout change will hit read amplification that quietly erases whatever bandwidth gain the tier was supposed to deliver.

Six-tier KV cache management across a multi-GPU host: placement, eviction, and predictive prefetch

Ganjihal's paper (arXiv:2604.26968, with a first version in April 2026 and a revision in August 2026) organizes the entire memory hierarchy into six tiers, each exposed through the same Allocate, Read, Write, Evict interface, so the system managing KV cache doesn't need a separate code path for every storage type it touches.

GPU HBM makes up Tier 0, holding the KV blocks actively in use during decoding, managed through PagedAttention's block system. CPU DRAM makes up Tier 1, pinned and reached through asynchronous DMA at 1 to 5 microsecond GPU-observed latency, offering roughly five times HBM's capacity. CXL 3.0 memory pools make up Tier 2, coherent and byte-addressable, with capacity above DRAM and latency between DRAM and NVMe. NVMe, accessed through GPUDirect Storage, makes up Tier 3. Tier 4 is RDMA fabric, the network path used for KV transfer between nodes in disaggregated serving setups. Tier 5 is parallel filesystem storage.

Six tiers, one interface, a clear ordering by latency and capacity: that's the shape of the problem once a serving system outgrows a single GPU's HBM. Which attention architecture is running, how PagedAttention sizes its blocks, how tensor parallelism splits the load across a node, whether the tensor layout survives the trip to a slower tier without amplifying every read: these decisions all feed into where a given KV block should live at any given moment, and skipping any one of them just moves the bottleneck somewhere else in the stack. None of them can be made in isolation from the others, and none of them get fixed by bolting another GPU onto the rack.