GPU Memory Hierarchy for LLM Practitioners

Every consequential decision in LLM inference, from batching policy to quantization format to how a serving team provisions GPUs, answers to one physical fact: the GPU memory hierarchy, not raw compute, is what limits how fast tokens come out. Most engineers still picture a GPU as a compute device, a pile of arithmetic units waiting for work. During autoregressive decoding that picture is backward. The model produces one token per forward pass, and each pass has to pull every model weight out of VRAM before it can do anything with it. While that transfer runs, the arithmetic units sit mostly idle because the memory bus, not the Tensor Cores, is what's saturated. A 2026 inference guide puts the point directly: during decode, GPU compute capacity runs at only a fraction of its ceiling, and the thing actually holding up each token is the time spent moving data through HBM, not the time spent running tensor operations.

The hardware numbers make the imbalance concrete. An H100 SXM5 has 3.35 TB/s of HBM bandwidth, and its FP16 Tensor Core throughput sits far above what that bandwidth can feed. The distance between those two figures is where inference performance disappears: the chip can compute far faster than it can be supplied with weights, so compute waits on memory for most of the decode step. That asymmetry also explains why adding GPUs doesn't fix the problem by itself. Putting a second H100 next to the first adds more FLOPS, but FLOPS were never the scarce resource. Per-token latency barely moves under naive horizontal scaling, because the memory bus on each device is still the thing being saturated, and no amount of parallel compute capacity relieves that saturation on its own.

The practical consequence is a reframe that governs the rest of this piece. The question worth asking when diagnosing a slow inference deployment is how much data is moving, from which tier of memory, and at what speed. Every technique covered below, architectural choices made when the model is designed, eviction and compression policies applied at serving time, and tiered offloading when HBM runs out, is an answer to that one question.

The three data structures competing for GPU HBM

VRAM holds three kinds of data during inference: model weights, activations, and KV cache. Only one of them grows without bound as a workload scales, which is what makes it the central target of nearly every optimization technique that follows.

Model weights are the easiest of the three to plan around. They're static for a given model and quantization format, and the relationship between formats is simple: a model's footprint at FP16 is roughly double its footprint at FP8. Once a team picks a quantization format, the weight budget is fixed and known in advance. Activations are the intermediate values produced during a forward pass, and they're transient and small next to weights and cache at the batch sizes most production systems run.

The KV cache behaves differently from both. Its size grows linearly with context length and with batch size at the same time, so it scales along two dimensions at once rather than one, and at sufficient scale it routinely exceeds the size of the model weights themselves. A scaling law presented in a Dell Technologies survey (arXiv:2603.20397) states the relationship directly: KV cache memory equals 2 × L × H × D × T × P, where every term in that product grows as context windows move from thousands of tokens to millions. Running the arithmetic on a large LLaMA-3 model with a very long context window shows the KV cache alone exceeding the memory capacity of many high-end GPUs, before a single model weight is loaded.

This is why the piece treats the KV cache as the dangerous occupant of HBM. Eviction policies, compression schemes, offloading systems, and architectural redesigns of attention all exist to manage a data structure that the basic Transformer design allows to grow without any natural ceiling.

The memory hierarchy a practitioner operates within

HBM is the fastest tier of a much longer hierarchy, and every tier beneath it carries its own latency and bandwidth cost that a serving design has to account for. The effective order for LLM workloads runs from Tensor Cores to L2 cache to HBM (the on-die VRAM) to LPDDR or CPU DRAM to NVMe SSD, with each successive tier roughly an order of magnitude slower and, in exchange, an order of magnitude larger than the one above it.

HBM capacity itself varies by hardware generation. An H100 SXM carries 80 GB of HBM3, while an H200 SXM carries 141 GB of HBM3e. That's a meaningful range for capacity planning, but KV cache growth at long context lengths can exhaust either end of it, which is why the lower tiers matter even on the newest accelerators.

Movement within and between machines has its own physical limits. Inside a single node, fifth-generation NVLink delivers 1.8 TB/s of bidirectional bandwidth per GPU, more than 14 times what PCIe Gen 5 offers. That difference is why GPU pools connected over NVLink can share KV cache across devices far more efficiently than pools connected only over PCIe. Across nodes, RDMA, running over InfiniBand or RoCEv2, lets a network adapter read directly from a remote server's GPU memory without involving that server's OS or CPU, which keeps inter-node KV transfer latency low enough to be workable in production. Lower in the stack, PCIe Gen 5 NVMe drives reach roughly 14,000 MB/s each, and four such drives in RAID 0 deliver roughly 45 to 50 GB/s of sequential read, a real tier for offloading KV state but still orders of magnitude below what HBM delivers.

None of these figures are specs to admire. They're constraints that determine where a piece of KV state should sit at any given moment. Data residency, which tier a given slice of the cache lives on right now, is the variable that decides whether a serving system runs fast and cheap or slow and wasteful.

Architecture-level techniques that reduce KV cache pressure before it reaches the memory hierarchy

Some of the most effective fixes for KV cache growth happen before a request ever reaches a serving stack, at the point the model itself is designed. These decisions set the baseline a serving team inherits, and understanding which problems are already solved at this layer keeps practitioners from reinventing fixes that architecture has already handled.

Grouped-Query Attention (GQA) shares key-value heads across multiple query heads. Llama 3 uses 8 KV heads shared across 64 query heads, an eightfold reduction in KV cache size compared to standard multi-head attention. That's a choice made when the model is built, and it sets the capacity baseline every downstream optimization has to work from. DeepSeek's Multi-Head Latent Attention (MLA) goes further, re-parameterizing keys and values as linear projections of a low-rank latent space, shrinking the KV footprint beyond what GQA achieves without a corresponding loss in model quality.

PagedAttention and prefix caching work at the serving layer to fix structural waste. PagedAttention, introduced by vLLM, borrows the idea of virtual memory paging from operating systems: KV cache is allocated in fixed-size blocks and mapped through a page table instead of as one contiguous allocation per sequence. That change drops memory waste from a range of 60 to 80 percent down to near zero and allows 2 to 3 times more concurrent requests on the same GPU. Prefix caching computes the KV state for a shared prompt prefix once and reuses it across every request that shares it. For a long system prompt served to many users, that eliminates billions of tokens of redundant computation, and the effect is especially pronounced in retrieval-augmented generation pipelines where the same retrieved documents recur across many queries.

Prefix caching has a real limit: a 2025 vLLM issue asking for generalized KV cache reuse across non-prefix-aligned token subsets, not just shared prefixes, was closed as "not-planned" in 2026, with a batching workaround using dummy requests offered in its place. Reuse works cleanly when token sequences align from the start. It breaks down when they don't, and that gap becomes a serious problem once workloads stop moving strictly forward, a situation this piece comes back to later.

Serving-layer strategies for managing KV cache within HBM: eviction and compression

Once the KV cache starts to fill HBM during an active serving run, the first tools reached for are eviction and compression, techniques that reclaim space without ever leaving the GPU. Their trade-offs between output quality and throughput depend heavily on the workload rather than holding universally, which is the section's central argument.

Eviction rests on an empirical property of attention: it's sparse. Most sequences depend on a small set of tokens, often called Heavy Hitters, that get disproportionate attention weight, while the rest contribute little, so you can drop them without a large hit to output quality. Heavy-Hitter Oracle (H2O) builds a cache policy directly on this observation: it maintains a fixed-size KV budget and, once that budget fills, evicts whichever token has accumulated the lowest total attention score. No model weights change under this policy. Only the contents of the cache do.

Compression reduces the bit-width of what's stored. KIVI applies per-channel quantization to keys and per-token quantization to values, compressing down to 2 bits by exploiting the fact that keys and values have different statistical properties and tolerate compression differently.

The 4-bit configuration of KIVI gives the most stable quality across both models tested. SnapKV delivers the strongest throughput on long-context workloads. CaM produces large gains on some QA tasks but swings substantially on others, both in quality and in the compression ratio it actually achieves in practice. No single technique wins across every context length, task type, and hardware setup. This argues for pipelines that choose a technique based on the workload in front of them.

The Dell Technologies survey organizes this entire space into five categories: cache eviction, cache compression, hybrid memory solutions, novel attention mechanisms, and combination strategies. The next two sections take up hybrid memory and the distinct problems that agentic workloads introduce.

Tiered residency across CPU DRAM and NVMe when HBM is exhausted

Eviction and compression can only reclaim so much. Once they're not enough to keep KV state inside HBM, for long-context or high-batch workloads especially, the only remaining option is to spill state to slower tiers, and the real engineering problem becomes managing that spill's latency cost without collapsing throughput.

LMCache is the main open-source system built for this purpose. It supports spilling KV state to CPU memory, NVMe, Redis, S3-compatible object storage, and remote servers, while keeping a fast HBM tier reserved for whatever tokens are actively being attended to. The raw cost of this approach is the latency added per cache miss when a read has to reach NVMe, a real cost but not necessarily a fatal one if the system can anticipate which tokens will be needed before they're requested.

Prefetching systems attempt that anticipation: InfiniGen fetches KV cache entries from CPU memory to the GPU while the GPU is still processing the current layer, anticipating which tokens layer i+1 will need and moving them ahead of time. FlexGen takes a related but distinct approach: it overlaps GPU computation with prefetching of model weights for the next layer, hiding part of the transfer latency behind ongoing compute.

DASH takes a more structural position on the same problem. Rather than treating High-Bandwidth Flash (HBF) as a fallback to spill into when HBM runs out, DASH treats HBF as a co-equal main memory tier alongside HBM. A dedicated HBM-HBF path lets HBF-resident KV data reach the GPU either directly or by routing through HBM, with placement decisions and prefill/decode-aware write scheduling designed specifically to hide HBF access latency.

At cluster scale, a further option opens up. RDMA-capable interconnects can fetch KV cache that lives in a remote GPU's HBM over the network, and that fetch can arrive faster than a local NVMe read would, making remote HBM a legitimate cache tier for distributed serving. GPUDirect Storage (GDS) complements this by removing the CPU from the I/O path for the NVMe tier entirely, which matters for large-scale checkpointing and data loading: at the throughput a large GPU cluster demands, routing storage I/O through the CPU becomes the bottleneck, so bypassing it is necessary.

What ties all of this together is that the quality of the underlying storage infrastructure is directly load-bearing. Drive generation, RAID configuration, and fabric connectivity for the NVMe tier set the floor on how much offloading degrades serving performance. A serving team that skimps on this layer pays for it in every cache miss that has to travel through it.

Agentic workloads break the monotonic-cache assumption that most serving infrastructure is built on

Every technique covered so far, prefix caching, PagedAttention, eviction by attention score, tiered offloading, rests on one shared assumption: a conversation moves forward. Tokens get appended, the cache grows, and nothing already written needs to change. Agentic LLM workloads violate that assumption directly. An agent retries a failed tool call, discards an output it decides was wrong, and pivots to a different trajectory mid-task, and each of those actions requires the KV cache behind it to be mutated, not just extended.

That requirement exposes the limit already visible in the vLLM prefix-caching issue closed as "not-planned": existing serving primitives are built to reuse and extend a forward-moving sequence, not to rewrite or selectively discard pieces of one that no longer belong. No serving primitive in current production use can handle that kind of active cache mutation cleanly. The techniques this piece has covered, from GQA at the architecture level to tiered NVMe offloading at the infrastructure level, were all built for caches that only grow. Agentic workloads need a cache that can change its mind.