NVLink vs. PCIe Bandwidth Impact on KV Cache Transfers

Every token a large language model generates during autoregressive decoding has to read the full key-value tensor for every token that came before it. That dependency makes the KV cache grow in step with sequence length, and in long-context or multi-turn workloads, it commonly outgrows the model weights themselves as the largest consumer of GPU memory. Once high-bandwidth memory fills up, a serving system has three choices: shrink the batch, cut the context window, or move KV data somewhere else. The third option, moving data, always means crossing some interconnect between chips, and that interconnect sits on the critical path of every KV offload, every cross-GPU cache share, and every handoff between prefill and decode workers. Its bandwidth sets a hard ceiling on what any amount of clever scheduling or caching logic can claw back.

What NVLink and PCIe actually are and how their bandwidth has evolved

NVLink is NVIDIA's own direct GPU-to-GPU connection, built to skip the shared PCIe bus. It gives GPUs a dedicated path with no CPU arbitration and no competition from storage controllers or other devices sharing the bus. PCIe, by contrast, is the general-purpose highway that the CPU, storage devices, and every GPU in a server all share. GPU-to-GPU traffic sent over PCIe has to pass through a shared root complex and relies on DMA; on paths without peer-to-peer support, the CPU gets involved, though GPUDirect P2P can route around it.

The bandwidth gap between the two has only widened with each hardware generation. NVLink 1.0, shipped with Pascal-generation P100 GPUs in 2016, carried 160 GB/s bidirectional across four links. NVLink 2.0 arrived with Volta's V100 in 2017 at 300 GB/s across six links. NVLink 3.0, paired with Ampere's A100 in 2020, reached 600 GB/s across multiple links. NVLink 4.0, built for Hopper's H100 and H200 in 2022, pushed each link to 25 GB/s per direction for a total of 900 GB/s bidirectional. NVLink 5.0, introduced with Blackwell's B200, B300, and GB200 in 2024, doubled per-link speed to 50 GB/s while holding the link count steady, landing at 1.8 TB/s bidirectional. The next generation, NVLink 6.0 on the Rubin R100 roadmap, projects around 2.4 TB/s, but it hasn't shipped yet and hasn't been measured in production. Set against this, PCIe Gen5 x16 tops out at 128 GB/s bidirectional, meaning NVLink runs about 7 times faster than PCIe Gen5 on Hopper systems and about 14 times faster on Blackwell.

One structural limit matters more than any of these numbers: NVLink only works inside a single server or rack. The moment communication needs to cross from one node to another, it travels over InfiniBand or Ethernet instead, no matter how fast the NVLink fabric inside each node happens to be. On an eight-GPU H100 SXM server, four NVSwitch chips turn that intra-node NVLink fabric into a full non-blocking mesh, letting every GPU talk to every other GPU at full speed at the same time, a topology PCIe-based systems simply cannot reproduce. That intra-node boundary is the seam the rest of this argument turns on.

How Severely PCIe Bandwidth Constrains KV Cache Offloading

In most long-context or multi-turn serving, the number of cached tokens from prior turns vastly outnumbers the new tokens in the current request. Under that condition, offloading KV cache over PCIe doesn't just slow things down, it changes the nature of the workload: prefill, which should be compute-bound, turns memory-bound instead, and the GPU's compute units sit mostly idle.

Research analyzing KV cache offloading gives this effect a name: κ_crit, the hardware-derived ratio of cached to prefilled tokens at which a workload tips over into being memory-bound rather than compute-bound. The actual ratio observed in production, κ_ratio, blows past κ_crit by orders of magnitude in typical deployments: conversational workloads show a median κ_ratio of 100, and document-query workloads run far higher still. The same analysis found PCIe transfer overhead reaching 86 times the prefill computation time, with nearly the entire execution window spent moving data. GPUs in that state draw only a fraction of their rated thermal design power because the chip is stalled waiting for bytes to arrive over the bus.

This is the ordinary condition of conversational serving, not a rare edge case, and it carries a direct consequence for anyone planning infrastructure spend: buying a faster GPU to improve long-context throughput does nothing if PCIe remains the data path, because the GPU in that scenario is already waiting idle most of the time. Open Lake's work on KV cache management across GPU and NVMe tiers addresses this bottleneck directly; its involvement in industry storage benchmarking has produced measurements of exactly where memory-bound regimes appear and how the choice of interconnect decides whether extra GPU capacity turns into real throughput or just more idle silicon.

What NVLink's bandwidth actually unlocks: peer-GPU HBM as a KV cache tier

Once GPU-to-GPU bandwidth is fast enough to stay off the critical path, something becomes possible that PCIe could never support: the unused high-bandwidth memory sitting on a neighboring GPU can act as a fast intermediate cache tier, closer in effect to local memory than to a remote resource. The gain isn't that memory gets shared; it's that the bandwidth gap makes a physically distant pool of memory behave, from a latency standpoint, almost as if it were local.

A Columbia University preprint from February 2026 describes a system called Harvest that puts this idea into practice. Harvest opportunistically treats unused HBM on NVLink-connected peer GPUs as a transient cache tier for both KV cache entries and mixture-of-experts weights, placing and revoking allocations dynamically as memory availability shifts, all without touching model code. The results are substantial: Harvest cuts KV cache transfer latency by a wide margin compared to CPU offload over PCIe, speeds up MoE expert retrieval considerably on smaller models and even more so on larger ones such as Mixtral, and raises end-to-end throughput meaningfully on models including Qwen and Phi-3.5. The opportunity is already sitting there unused: in a large production trace, most machines consume well under half of their available GPU memory at any given moment, so Harvest is exploiting idle capacity operators already have. None of this would hold over PCIe. Without NVLink's bandwidth, a peer GPU's memory offers no real latency advantage over host DRAM, since both require the same costly trip across the slower bus.

How Disaggregated Prefill-Decode Serving Exposes the NVLink Boundary

Splitting prefill and decode across separate pools of GPUs has become a standard way to scale inference elastically, letting each phase be provisioned and optimized on its own terms. But that split comes with an obligation: the KV tensors generated during prefill have to physically travel to wherever decode is running, and the interconnect carrying that handoff decides whether the split pays off or backfires.

On a node where GPUs are connected by NVLink, this handoff is cheap enough that disaggregation stays architecturally clean, and the benefits of separating the two phases come through largely intact. On a node still relying on PCIe for that same transfer, the overhead of moving KV tensors between prefill and decode workers can eat into or even cancel out whatever parallelism gain the split was meant to deliver. The interconnect, in other words, decides whether disaggregation is a genuine architectural improvement or a self-inflicted bottleneck dressed up as one.

When NVLink's Advantage Disappears: Cross-Node Disaggregation and the RDMA Tier

NVLink's reach stops at the edge of the server or rack. When prefill and decode workers run on separate physical machines, a setup increasingly common in elastic or cost-optimized cloud deployments, KV tensors have to cross an RDMA fabric no matter how fast the NVLink inside each individual node might be.

Systems built with this boundary in mind treat the network, not NVLink, as the real path the KV cache travels. The NIXL library, for instance, moves KV data across nodes using RDMA and overlaps that transfer with ongoing computation to hide latency, rather than depending on raw interconnect speed to make the handoff fast. An emerging pattern called "Prefill-as-a-Service" pushes the idea further still, sending KV tensors across entire datacenters over standard IP networking, a regime in which intra-node bandwidth has no bearing whatsoever on the cost of the handoff.

None of this erases NVLink's value; it draws a boundary around it. Inside a single node or a rack-scale NVLink fabric, NVLink remains the decisive factor in KV transfer cost. The RDMA story only takes over once disaggregation deliberately crosses a node boundary, a choice that requires real architectural planning and brings its own latency profile along with it. The practical question for anyone designing around this isn't how much bandwidth is available in the abstract, but where the KV cache actually crosses a boundary in a given deployment, and what interconnect is waiting on the other side of that crossing. The bottleneck moves with the topology, not with the hardware generation alone.

Compression as a way to make PCIe-class bandwidth serve NVLink-class transfer budgets

KV cache tensors in BF16 and FP8 formats carry a structural quirk that general-purpose data doesn't share: their floating-point exponent fields are statistically redundant, typically compressible down to 3 or 4 bits of real information. That redundancy makes lossless compression of KV tensors practical at speeds fast enough to matter for live inference, a property that weight tensors don't exhibit nearly as strongly.

A 2026 system from Guo and Joshi called SplitZip is built around exactly this gap. It encodes common exponent values using fixed-length codes and shunts the rare ones through a separate sparse stream, all while preserving the KV tensor bit-for-bit and plugging into existing serving frameworks without any change to how the model itself runs. On real BF16 activation tensors, SplitZip compresses at high speed with correspondingly fast decompression, and in end-to-end transfer tests it delivers notably faster BF16 KV cache transfer, faster time-to-first-token, and higher request throughput. The same approach extends to FP8 KV caches, yielding modest additional compression over native E5M2. SplitZip's own evaluation is candid about its limits: the gains show up most clearly when KV cache transfer is already the bottleneck, and short-context or compute-bound workloads see less benefit because there isn't enough transfer volume for the codec's overhead to pay for itself. Compression doesn't replace a better interconnect; it stretches the usefulness of whatever interconnect is already in place, whether that's PCIe struggling under load or NVLink looking to push even more data through the same window of time.

The Bandwidth Hierarchy's Three-Tier KV Residency Policy in Production

Put together, the interconnect hierarchy points to a practical rule for where KV data should live: match each block's "temperature" to the bandwidth available at the tier that holds it. Hot blocks are data tied to active generation or accessed very recently, and they stay in local HBM. Warm blocks, completed but likely to be needed again in a multi-turn conversation, move to peer GPU HBM over NVLink where that's available, or to CPU DRAM over PCIe where it isn't. Cold blocks, historical context or prefix cache unlikely to be touched again soon, move to NVMe.

The NVLink tier is the one Harvest operationalizes directly: peer HBM handles warm KV blocks at bandwidth high enough to stay off the critical path, as long as the cluster has spare GPU memory to draw on, which production traces show is the normal state of affairs. The PCIe tier, meaning CPU DRAM, only makes sense for blocks cold enough to tolerate its latency penalty; pushing hot KV blocks onto that tier drives the workload into the memory-bound regime described earlier, where GPUs sit idle waiting on transfers that should never have been necessary.

The NVMe tier carries a constraint of its own that has nothing to do with raw bandwidth numbers. Reloading KV cache from flash storage requires medium-size contiguous transfers to hit the bandwidth an SSD is capable of; requests broken down to something as small as 4 KB can max out the device's IOPS limit while leaving most of its write bandwidth sitting unused. That mismatch between request granularity and device design has to be solved in software, through block sizing and prefetch policy, not by buying a faster drive. Research characterizing high-bandwidth flash for KV-centric serving backs this up starkly: dropping a faster storage device underneath an existing Mooncake-style KV offload stack can make end-to-end latency and SLO goodput worse, because faster reads only help when reads are actually the bottleneck, when reads substantially outnumber writes, and when the device can sustain its rated bandwidth within its thermal and endurance limits. Meeting those conditions means redesigning the runtime around the hardware, not simply swapping in better hardware underneath an unchanged runtime. Open Lake's infrastructure work treats this hierarchy as the starting assumption rather than an afterthought: storage, in this view, is an engineering surface that controls whether each tier actually delivers its promised bandwidth to the KV path, covering everything from NVMe block sizing to PCIe saturation management to how data gets placed across the NVLink tier.

A decision framework for choosing interconnect strategy based on KV workload characteristics

The question that should guide interconnect strategy is where, specifically, the KV cache crosses a tier boundary, and what that crossing costs relative to the compute time it ends up interrupting. Answering that question well requires walking through the actual topology of a deployment rather than treating "NVLink versus PCIe" as a single universal verdict.

For single-GPU inference where the entire KV cache fits comfortably in HBM, the interconnect never becomes part of the critical path at all; PCIe is more than adequate, and NVLink offers nothing extra for KV management specifically, since there's no cross-GPU transfer to accelerate. For disaggregated prefill-decode running on a single NVLink-connected node or rack, NVLink and NVLink Switch govern the cost of the KV handoff directly, and this is the setting in which disaggregated-serving architectures produce their largest throughput gains. Once disaggregation crosses node boundaries, whether through cross-node setups, Prefill-as-a-Service, or any topology where prefill and decode workers run on separate machines, the quality of the RDMA fabric becomes the governing factor, intra-node interconnect speed becomes irrelevant to the handoff, and the engineering question shifts to network bandwidth, the RDMA implementation in use, and how well transfers overlap with computation, as NIXL demonstrates. Wherever PCIe remains the binding path and can't be engineered around, lossless compression in the style of SplitZip, targeting the redundancy in BF16 and FP8 exponent fields, and lossy KV quantization approaches such as FP8 or NVFP4 are the software interventions that deliver the most value, because they cut transfer volume without requiring any change to the underlying hardware. CXL is emerging as an option that sits between PCIe and NVLink in both bandwidth and cost, and it may eventually offer a new middle path for rack-scale KV sharing, though it still needs purpose-built runtime support and isn't yet a drop-in substitute for either existing interconnect.

Open Lake's engineering practice is built around walking through exactly these decision points for each deployment, matching storage tier design, block sizing, compression strategy, and placement policy to the specific interconnect topology at hand, because the right design for a PCIe-only cluster looks nothing like the right design for a rack full of NVLink-connected GPUs. The bottleneck that governs a given system isn't fixed by the hardware generation alone; it shifts with where the KV cache actually has to cross a boundary, and the only durable design principle is to find that boundary first and build around what the interconnect there can and cannot carry.