HBM Bandwidth Saturation in Autoregressive Decoding

During a decode run on an H100, the utilization graph shows a number that stands out: how low it sits. The tensor cores, the part of the chip built to do trillions of floating-point operations per second, often register single-digit-percent usage while the GPU is unmistakably busy generating text. That gap between "the chip is working" and "the chip's compute units are working" is the entire story of autoregressive decoding, and it comes from how the token-by-token generation loop is built.

Each decode step produces exactly one token. To produce it, the accelerator has to pull the full set of model weights out of HBM, along with the accumulated key and value state from every prior token in the sequence, and feed all of it into what amounts to a single matrix-vector multiply. That multiply is small. The data movement required to feed it is not. The ratio between the two, known as arithmetic intensity, collapses to roughly O(1) per cached token: one unit of compute for every unit of data read. TokenStack's introduction states the mechanism: "the arithmetic intensity of this step drops to O(1), making decode-phase attention fundamentally bandwidth-limited.

Prefill, the phase where a model processes an entire prompt at once before generating its first token, looks nothing like this. Prefill handles many tokens in parallel, so it can reuse the weights it just streamed in across a large batch of simultaneous computation, and it runs compute-bound as a result. Decode and prefill run on identical silicon, yet one keeps the tensor cores fed and the other starves them. An H100 needs on the order of 300 FLOPs per byte moved before its compute units become the limiting factor. Decode delivers roughly one FLOP per byte, a ratio about 300 times below that threshold. The tensor cores stall waiting on memory because the autoregressive constraint, one token in, one token out, makes high arithmetic intensity structurally unreachable at this step of the pipeline.

The roofline model as a diagnostic tool for decode-phase behavior

The roofline model gives this observation a precise, testable shape. Plot a chip's performance with arithmetic intensity on the horizontal axis and achieved throughput on the vertical axis, and two ceilings appear: a flat line marking peak FLOPS, and a rising diagonal line marking peak memory bandwidth multiplied by arithmetic intensity. A kernel's achievable performance is bounded by whichever ceiling is lower at that point on the horizontal axis: you take min(peak FLOPS, peak bandwidth times arithmetic intensity). The point where the two lines cross is the ridge point, the arithmetic intensity at which compute and bandwidth become equally limiting.

Datacenter GPUs carry a very high FLOP-to-bandwidth ratio, which pushes their ridge point far to the right on this chart. Almost any kernel with low arithmetic intensity, including decode attention, lands on the diagonal bandwidth roof rather than the flat compute roof, and no amount of added FLOPS moves it upward. On a roofline plot, decode-phase attention draws as a near-vertical line hugging the left axis, and that position holds regardless of model size, batch size, or how long the generated sequence runs. Adding compute capacity to a chip already sitting on the bandwidth roof does nothing for throughput, because the bandwidth roof, not the compute roof, is what caps the kernel.

Patterson and Ma's work on LLM inference hardware quantifies how far that ridge point has already moved. From 2012 to 2022, NVIDIA GPU 64-bit FLOPS rose 80-fold, while memory bandwidth grew only 17-fold over the same decade. The ridge point has shifted rightward with every generation, enlarging the memory-bound region of the roofline chart.

Empirical profiling backs this reading at the kernel level. SLIM's measurements trace throughput saturation to the attention kernels specifically during decode, where arithmetic intensity stays nearly constant as the active context grows longer, driving DRAM bandwidth to its ceiling while measured compute throughput remains far below what the hardware can do. A natural objection follows: doesn't batching more requests together raise arithmetic intensity and move the kernel rightward, off the bandwidth roof? For the linear projection layers, yes, batching helps directly. For attention kernels, SLIM's data shows arithmetic intensity holds nearly flat even as batch size and context length increase, because KV memory traffic grows in step with context length but the compute work does not scale at the same rate. Batching helps parts of the model. It does not rescue decode attention from the bandwidth roof.

KV cache growth and bandwidth pressure

The roofline chart explains where decode sits structurally. So the KV cache explains why that position gets worse the longer a conversation or a reasoning trace runs. Every new token generated appends a fresh key vector and a fresh value vector to the running cache, one encoding the identity of the token and the other encoding its content, and both have to be read in full at every subsequent step to compute attention correctly over the whole sequence. There is no shortcut inside the standard transformer architecture that lets a model skip reading old tokens to attend to new ones. So each decode step has to stream a state that grows linearly with context length, and that compounds the O(1) intensity problem one token at a time.

Reasoning workloads push this to its limit. Traces generated by reasoning models can run past 10,000 tokens, and every token generated along the way needs the entire model weight set plus the full active KV cache read out of HBM. Inference scaling research has named a distinct regime for this behavior: capacity-bound, where reasoning workloads produce thousands of intermediate thinking tokens that build a KV cache footprint large enough to saturate HBM capacity before compute utilization ever peaks. Bandwidth saturation and capacity saturation are different failure modes in principle, but reasoning workloads routinely trigger both at the same time, which makes serving them a harder problem than serving short, single-turn responses.

Engineers watching a production deployment have a direct instrument for this, Time-Per-Output-Token, or TPOT. A rising TPOT signals that HBM bandwidth is running out, or that tensor-parallel configurations are burning time on communication overhead. The same inference scaling research documents a related failure mode in data-parallel deployments running vLLM: KV cache fragmentation spread across replicas forces the system to throttle incoming requests prematurely even while compute utilization sits low, a pattern researchers call stranded capacity. Memory that exists on the chip but cannot be assembled into a usable contiguous block is, for practical purposes, memory that was never there.

Why the hardware roadmap widens the gap

Datacenter GPUs are built chiefly for training, a compute-bound workload, and each new generation adds FLOPS faster than it adds bandwidth as a direct consequence of that design priority. That choice pushes the roofline ridge point further right with every release, which makes the gap decode sits in larger, not smaller, from one hardware generation to the next. Patterson and Ma trace this pattern across a full decade: the FLOP-to-bandwidth ratio has expanded sharply, and nothing in the current trajectory reverses it, because no GPU on the market today was designed solely to serve LLM inference.

A 2026 study of edge inference (arXiv:2605.30571) adds a sharper wrinkle to this picture. On lower-bandwidth GPUs such as the L4, the fraction of peak HBM bandwidth a decode kernel actually achieves falls as peak bandwidth rises across the product line, so the choice of kernel implementation has a direct effect on throughput on hardware with a lower bandwidth ceiling. Substituting one quantization kernel for another on the default kernel path cut step time by more than 3-fold in that study, a result achieved without touching the accelerator.

So the conclusion is a hard one for procurement decisions built around the assumption that newer hardware fixes decode. Buying a faster GPU raises the bandwidth ceiling in absolute terms, and that is a real gain. But it does not move the decode kernel relative to the ridge point, because the kernel's arithmetic intensity has not changed. An H200 serves more tokens per second than an H100 because its bandwidth ceiling is higher. The hardware roadmap raises the floor. It does not close the structural gap this piece opened with, so the remaining work falls to software and system design.

Software mitigations that raise the effective bandwidth ceiling without changing the hardware

None of the mitigations below change the arithmetic intensity of decode attention itself, which stays pinned near O(1) regardless of software. What they change is the volume of data that has to move per token, or the number of tokens a given byte of bandwidth can serve.

The least invasive starting point is memory management. PagedAttention, implemented in vLLM, splits the KV cache into small pages allocated on demand instead of reserving one large contiguous block per request. That eliminates the fragmentation that otherwise wastes HBM capacity and forces requests to throttle early, so more requests fit into memory at once and tokens-per-second per GPU rises, without touching model weights or numerical precision.

Quantizing the KV cache goes a step further by reducing how many bits represent each stored value. FP8 halves memory per parameter compared to BF16 or FP16, and a model that needs two H100s running at BF16 can fit on a single B200 at FP8. Google's TurboQuant, announced March 2026, pushes this further still, compressing the KV cache to approximately 3.5 bits per value with no measured accuracy loss at that level, though degradation appears once compression passes below 2.5 bits per value. That cuts KV cache memory by a factor of 6, which is the most aggressive published result among the approaches cited here.

Semantic-aware eviction adds a layer of judgment about which parts of the cache matter most. RocketKV runs a two-stage pipeline, a coarse eviction pass that discards low-value cache entries first, followed by fine-grained compression applied only to what survives. Newer architectures target a different source of pressure entirely: long-running, multi-turn agent sessions that keep KV state alive across many conversational turns or multiple cooperating agents. CommitKV (arXiv:2608.07855) compresses KV cache around commit transitions in multi-turn agent lifecycles, and GraniKV (arXiv:2608.15584) pages KV state at asymmetric granularity for agents that share a common prefix, addressing residency pressure that didn't exist in single-turn serving.

A separate category sits apart from all of the above: lossless compression, which reduces the bytes BF16 or FP8 weights occupy without changing a single value they represent. ZipServ reports up to 30% model size reduction with bit-exact numerical precision and a substantial kernel-level speedup over cuBLAS on GDDR-based GPUs. The NWC project reports BF16 weights compressed smaller and run faster than cuBLAS, with FP8 weights compressed smaller still while running at native FP8 speed. SplitZip, accepted at NeurIPS 2026, applies this same lossless approach specifically to KV cache transfer in disaggregated serving setups. The distinction between quantization and lossless compression carries real operational weight: agentic applications where small errors compound across many sequential steps, and fine-tuned domain-specific models where quantization measurably degrades task performance, need dense BF16 inference, and lossless compression is the only option on this list that preserves numerical fidelity exactly while still reducing bandwidth demand.

Memory hierarchy tiering as a capacity and bandwidth expansion strategy

When the KV cache and the model's weights together exceed what HBM can hold, the obvious instinct is to prefetch data from host memory into HBM ahead of when it's needed. That instinct backfires: prefetching consumes HBM write bandwidth, and that's the exact resource the decode kernel needs free for reads, so prefetching intensifies the saturation it was meant to relieve.

BOOST addresses this with concurrent proportional access: a wave-aware page placement scheme that routes memory accesses to HBM and host memory at the same time, split in proportion to each tier's bandwidth, without requiring any change to the decode kernel itself. In high-throughput serving conditions, BOOST delivers substantially higher throughput than prefetching does, because it never asks HBM to spend write bandwidth on data the read path needs.

Extending the hierarchy to NVMe storage opens up a further tier below host memory. FlexGen schedules computation and tensor placement across GPU, CPU, and disk to maximize throughput for single-GPU generation. DeepSpeed Inference and ZeRO-Inference stream model layers in from CPU or NVMe storage as needed. One system provides asynchronous host-to-device tensor transfer alongside direct storage-to-GPU transfer interfaces, and the LLM in a Flash research looks at how you execute models efficiently when DRAM is scarce and flash storage has to carry part of the load.

DASH (arXiv:2608.14333) pushes this co-design further by connecting HBM and High-Bandwidth Flash as two tiers of main memory joined by a dedicated HBM-to-HBF path. DASH combines dual-path access with placement decisions that account for the workload, early determination of which experts a mixture-of-experts model will need, and KV write scheduling aware of the prefill and decode phases, all aimed at hiding HBF access latency without interfering with the latency-sensitive reads decode depends on. TokenStack goes further still, proposing a vertically heterogeneous HBM-PIM architecture that splits each HBM4 stack into dense capacity layers and separate processing-in-memory compute layers, coordinated by a base-die controller that handles cross-layer data movement and performs inline quantization as data migrates between layers. Production-derived traces show this design achieving meaningfully higher geometric-mean token throughput and better compliance with serving latency targets than the AttAcc baseline, along with a substantial cut in per-token energy use.

What unites every system in this section is a single design principle, that storage tier selection and data placement decide whether the decode kernel sees bandwidth close to HBM's peak or bandwidth closer to whatever slower tier sits beneath it, and tiering is therefore an active bandwidth-management decision with direct throughput consequences.

Prefill/decode disaggregation and RDMA as the system-level response to phase mismatch

Every mitigation covered so far operates inside a single node serving both prefill and decode for a given request. Prefill/decode disaggregation abandons that assumption. Because prefill is compute-bound and decode is bandwidth-bound, running both phases on the same GPU means each phase competes with the other for resources it doesn't actually need, a compute-heavy prefill request stealing memory bandwidth from a bandwidth-starved decode step running alongside it, or the reverse. Disaggregation assigns prefill and decode to separate pools of hardware, each provisioned and scheduled for the resource profile its phase actually demands.

That separation creates a new problem in its own right: once a prompt finishes prefill on one set of GPUs, the KV cache it produced has to move to a different set of GPUs before decode can begin, and that transfer is itself a bandwidth-bound operation running over the network. This is the role fast, purpose-built interconnects play in disaggregated serving, moving KV cache state between nodes fast enough that the transfer doesn't erase the throughput gains disaggregation was meant to deliver. SplitZip's lossless compression, built specifically for KV cache transfer in disaggregated serving, exists because this inter-node transfer step has become its own bandwidth bottleneck, distinct from the HBM bottleneck inside each individual GPU.

Disaggregation, in this light, is the system-level acknowledgment that prefill and decode are two workloads with opposite resource signatures, and that serving them well means building infrastructure, compute pools, memory tiers, and interconnects, around that difference.