Unified Memory and Managed Memory Pitfalls in CUDA Inference

LLM inference, during the token-by-token decode phase, is bound by memory bandwidth, with arithmetic throughput largely idle. Single-token decode runs at an arithmetic intensity of roughly 1.0 FLOPs per byte moved, but the H100's ridge point, the ratio at which compute and bandwidth are balanced, sits orders of magnitude higher. That gap means the GPU's tensor cores spend most of a decode step idle, waiting for weights and activations to arrive from HBM rather than running out of multiplications to perform.

The clearest evidence sits in the generational comparison between the A100 and the H100. HBM bandwidth, not FLOP count, sets the ceiling for this workload, so adding compute without adding proportional bandwidth buys little.

This asymmetry is a structural property of autoregressive generation, not a tuning defect that better kernels or smarter batching can dissolve. Any memory management system placed between the GPU and that data path inherits an obligation: it has to move bytes predictably and fast, because there is no surplus compute cycle available to hide a slow or uncertain transfer behind. That is where Unified Memory and its higher-level API, Managed Memory, run into trouble.

Unified Memory and Managed Memory in the CUDA Runtime

Unified Memory solves a genuine problem in GPU programming: it presents a single logical address space to both the CPU and the GPU, so a developer can write pointer-based code without manually staging copies between host and device. The cost of that convenience is deferral. Physical placement of a given page is not decided at allocation time; it's resolved lazily, page by page, the moment an access fault occurs and the runtime has to decide where the data actually needs to live.

CUDA's memory hierarchy runs across six tiers, from registers at roughly one cycle of latency, through shared memory at two to three cycles, L2 cache at thirty to forty cycles, global HBM at four hundred to six hundred cycles, out to host DRAM and NVMe further still. When a page isn't resident where it's needed, migration overhead stacks on top of that baseline.

Managed Memory is the API surface that exposes this mechanism to application code: allocations tagged cudaMallocManaged let the driver move pages between CPU and GPU memory without the application issuing explicit copies. This design was built for general-purpose HPC workloads, where a page-fault stall gets amortized across a long compute phase that the GPU is going to spend many cycles working through regardless. Decode has no such phase. Each step is already bottlenecked on bandwidth, so there's no slack to absorb a stall.

Page-fault stalls land in the worst possible place in an inference kernel

Modern inference attention kernels, including FlashAttention-3 on Hopper, achieve their efficiency by tiling queries, keys, and values through shared memory and keeping the attention computation inside SRAM. When a migration event stalls a streaming multiprocessor mid-tile, it breaks that overlap outright, and because the pipeline is sequenced, the stall doesn't stay local. It serializes the block around it.

Production inference stacks, including vLLM and SGLang, along with orchestration layers like NVIDIA Dynamo, rely on CUDA graphs to cut kernel-launch overhead by capturing a fixed execution schedule ahead of time.

These stalls are not deterministic. The exposure is worst during prefill, when the full prompt's key-value cache is being written for the first time. Migration pressure peaks at exactly that moment, because a new large request can trigger migrations across the entire managed address space just as the system is already pushing maximum bandwidth load.

Why the KV cache is the specific allocation that makes Unified Memory dangerous at production scale

The key-value cache is the one allocation in an inference system that exposes every weakness in Unified Memory's model. It grows continuously for the life of a session, it's read and written on every decode step, and it fragments badly under naive contiguous allocation. If you defer physical layout decisions to a runtime fault handler, you get a poor match for an allocation with exactly these properties.

KV cache size scales linearly with both sequence length and batch size, and at long context with high concurrency it becomes the dominant consumer of GPU HBM, often crowding out the working space a model needs for activations. PagedAttention, the mechanism underlying vLLM, treats the KV cache the way an operating system treats virtual memory, allocating non-contiguous pages on demand. That approach cuts memory waste to under 4% and gives you throughput gains of two to four times over naive allocation.

The conflict at the center of this section is structural. Paged KV blocks carry explicit placement requirements: they must stay resident in HBM through an active decode session, not get migrated to host DRAM mid-session. Managed Memory works on the opposite premise: it stays free to migrate any page it judges cold, whether or not an inference scheduler still considers that page live. An H100 SXM5 with 80 GB of HBM can sustain only one or two simultaneous users at 128K context in BF16 once weights are loaded. It destroys the session's latency.

The memory hierarchy when KV cache spills beyond HBM, and where Managed Memory's model breaks down

Once the key-value cache outgrows HBM, serving it requires orchestrating movement across several distinct tiers, each with its own bandwidth and latency contract: HBM at the top for bandwidth but smallest in capacity, host DRAM next, reachable over PCIe or NVLink-C2C with more capacity but a real bandwidth cut, local NVMe larger still and persistent but slower again, and networked storage reachable over RDMA for scale beyond a single node. Managed Memory has no model of any of it.

Host DRAM alone isn't a sufficient extension tier for long-context, multi-turn, or agentic workloads. Roughly 2 TB of DRAM retains only about five minutes of KV cache under heavy load, so NVMe or networked storage becomes a requirement, not an optimization, for production long-context serving. Research out of Cornell on a system called DAK shows why prefetch-based tiering, the practice of staging data into HBM ahead of compute, underperforms on three distinct counts: incoming prefetch traffic competes with active compute kernels for HBM bandwidth, staging buffers eat into HBM capacity that would otherwise hold live working data, and coarse-grained prefetching introduces pipeline bubbles that stall compute. Bypassing HBM staging entirely through direct access via the Tensor Memory Accelerator achieves near-optimal aggregate bandwidth and up to 3x gains on NVLink-C2C systems.

The Hermes system illustrates how punishing the alternative is: the PCIe bandwidth gap between GPU HBM and host DRAM is wide enough that naive offloading solutions spend about 99% of total runtime simply moving data back and forth. That's the regime Managed Memory enters the instant it migrates a KV page to host DRAM, because the driver has no concept of which blocks are hot, about to be touched on the next decode step, versus cold, belonging to a session that's paused. That distinction lives in the inference scheduler. The CUDA driver was never given access to it.

How Production Inference Stacks Work Around These Problems

Every production inference system handling KV cache at meaningful scale has converged on the same answer: explicit, scheduler-visible memory management. That convergence, arrived at independently by different engineering teams solving the same problem, is itself the strongest evidence that Managed Memory's implicit model doesn't work for this class of workload.

vLLM's PagedAttention puts KV block allocation, mapping, and eviction directly under the inference scheduler's control. The Tutti system, described in a 2026 arXiv preprint, persists KV tensors to local NVMe through a peer-to-peer DMA path and notifies a companion system called Mooncake to register replica metadata, making offloaded KV cache globally discoverable for later reuse through explicit, metadata-tracked offload.

NVIDIA's Inference Context Memory Storage Platform, announced at CES 2026, standardizes KV cache offload to NVMe SSDs through BlueField-4 DPUs with hardware-accelerated cache placement, while NIXL, the NVIDIA Inference Xfer Library, moves KV blocks explicitly across NVLink, InfiniBand RDMA, PCIe, and TCP tiers. LMCache benchmarks on a DGX SuperPOD show time-to-first-token at 128K context falling from over 11 seconds to 1.5 seconds, a result built on a 400 Gbps RDMA fabric, BlueField-3 DPUs, and GPUDirect Storage, none of which Managed Memory can call on because it never issues explicit RDMA operations. The prefill-versus-reload asymmetry drives the point home at scale: at 1M-token context, prefill from scratch on an H100 takes roughly 90 seconds, while loading the same KV blocks back from NVMe takes approximately 8 seconds, but only because that NVMe path depends on explicit block addressing that Managed Memory simply does not expose.

Weight loading, ingestion tax, and Unified Memory's topology constraints

The same failure mode appears outside the KV cache entirely, in how model weights get loaded. Research on file-backed weight adoption identifies what it calls the ingestion tax: a framework-level copy made into framework-owned storage even when the file-backed pages already sit in a memory domain the GPU can read directly. On integrated and coherent-memory systems, Apple Silicon and GH200 over NVLink-C2C among them, that copy accomplishes nothing but wasted bandwidth and time.

Simply removing the copy isn't enough on its own. The same research shows that an adopter using zero-copy import but routing activations through the host and dispatching outside the framework runs decode substantially slower than stock, because the synchronization overhead introduced exceeds whatever the copy would have cost. Three conditions have to hold together for the approach to pay off: adopting the mapped pages directly, keeping activations resident in the accelerator-visible domain, and establishing GPU-side ordering. Dropping any one of the three measurably degrades throughput.

Topology decides whether any of this is viable. On discrete GPU systems connected over PCIe, the identical file-backed approach runs 39 times slower than resident loading, so the deployment rule is to adopt file pages only where the GPU can already read them without crossing a slow interconnect. Managed Memory runs into the same topology wall from the other direction. The same constraint surfaces in models whose layers are split across devices with different roles, where some layers are typically replicated across GPUs while others are sharded: that imposes a precise, per-layer locality requirement on KV cache access, and Unified Memory's topology-blind migration conflicts with it directly.

The Strongest Counterargument: Future Hardware-Managed Migration

The fairest case for Unified Memory rests on the hardware roadmap. Upcoming GPU generations promise hardware-managed prefetching across NVLink-connected GPUs, and the objection runs that a sufficiently accurate migration predictor could eventually absorb the brittle software tuning, block sizes, eviction thresholds, RDMA fabric requirements, that explicit KV management currently demands from engineering teams.

That objection has real force in specific, narrower cases. Workloads with unique prefixes, general-purpose chatbots where every request is effectively distinct, gain nothing from prefix KV caching in the first place, so the fragmentation and eviction hazards that make Managed Memory dangerous elsewhere matter less here: the cache would be evicted regardless of which system manages it.

The objection loses force precisely where it matters most: long-context, multi-turn, and agentic workloads. Bandwidth across PCIe, and even across NVLink-C2C, remains a cliff relative to HBM no matter how good the predictor gets. And evicting an active KV block carries a cost that no predictor can engineer away: recovering it requires a full prefill, which at long context is prohibitively expensive regardless of how the eviction decision was made.

Fast recovery and the hidden cost of device-memory ownership under Managed Memory

When an inference engine fails, the dominant cost is the time required to reconstruct a ready serving state from nothing, not the request progress lost at the moment of failure. Managed Memory's ownership model makes that reconstruction mandatory even when it shouldn't be, because it ties every allocation to the process that created it, and when that process exits, the allocation goes with it, regardless of whether the GPU underneath is still physically intact.

A fast-recovery paper on NVIDIA Dynamo, published in September 2026, analyzed 18 weeks of production failures and found that most of them are device-preserving: the engine process fails while the GPU and its resident allocations remain untouched. Conventional recovery doesn't take advantage of that fact. Figure 1 of the paper puts the range at 106 to 156 seconds across four models they tested on 8×B200 GPUs.

Dynamo's GPU Memory Service addresses this by decoupling device-memory ownership from the engine process itself. None of that is achievable under Managed Memory's semantics. A cudaMallocManaged allocation is scoped to the process that owns it by design, and there's no mechanism for a replacement process to reattach to surviving device-resident pages after the original process exits, which is exactly the capability GMS depends on. Deployments built on Managed Memory have to overprovision spare GPU capacity to mask the recovery window, and that capacity sits idle and burns cost because the ownership model forecloses a faster path.

Correct mental models in practice: explicit placement, scheduler visibility, and storage as a first-class concern

Every failure mode examined here traces back to one design decision: Unified Memory moves placement decisions out of the hands of the systems that actually have the information needed to make them well. The inference scheduler knows which KV blocks are live. The driver doesn't. You can't just flip a configuration flag to correct for that; you need to treat memory placement and storage architecture as first-class design concerns from the outset.

You need explicit KV block management, PagedAttention or an equivalent scheme, as the baseline requirement for production serving. Without it, fragmentation alone makes inference at scale economically unworkable, regardless of whether Managed Memory is involved. The DAK result reinforces the point with hard numbers: bypassing HBM staging through direct access achieves near-optimal aggregate bandwidth, which shows that the prefetch-first model isn't simply the wrong philosophy but a measurably slower one.

Topology has to be designed for explicitly, ahead of runtime. File-backed zero-copy paths work on coherent-link and integrated-memory systems like GH200 over NVLink-C2C and Apple Silicon, but they actively hurt performance on discrete-GPU PCIe systems, so the deployment rule needs to be built into the system from the start. Storage systems serving LLM inference, whether local NVMe, networked object stores, or RDMA-connected file systems, need to be built around the access pattern KV block eviction and reload actually produces: write-once, read-many, with read bandwidth demands far exceeding write. General-purpose storage defaults were never built for that profile, and treating memory placement as an afterthought, rather than as a design requirement on equal footing with model architecture and scheduling policy, is what turns Unified Memory's convenience into a production liability.