Multi-Instance GPU Partitioning for Inference Density

GPU partitioning is back on the table in 2026 because inference workloads keep hitting the same wall: GPU memory fills up with KV cache long before GPU compute ever saturates. That mismatch leaves a measurable share of compute sitting idle on any single large model, and at today's GPU prices, idle compute is a budget problem, not a footnote. The rest of this piece works through why that gap exists, how NVIDIA and AMD carve up a physical GPU to close it, and where the strategy runs into limits that compute isolation alone cannot fix.

GPU Partitioning as an Inference Density Strategy

The decode phase of LLM inference is bound by memory bandwidth, not by compute. On every single forward pass, the GPU has to pull the full weight matrix out of VRAM, and for a large model running in FP16, that read happens token by token at a scale large enough to dominate the cost of generation. This is the detail that drives GPU selection for H-series hardware: the bottleneck is how fast data moves off HBM, not how many floating-point operations the silicon can perform per second.

KV cache adds a second source of pressure on top of that. Every active request stores key-value attention states in VRAM, and the size of that storage scales with context length, batch size, the number of layers in the model, and the precision used to store it. At long context windows and high concurrency, KV cache starts competing directly with the model's own weights for the VRAM that's left. The GPU ends up nearly full on memory while its compute units sit well below capacity.

That mismatch has a direct consequence for smaller models. A GPU running one modest model at moderate concurrency uses only a fraction of its available compute, even as its memory fills. Partitioning takes that unused compute and turns it into capacity for a second, third, or fourth workload running alongside the first, each isolated on its own slice of the chip. The efficiency case holds most clearly for smaller models and for multi-tenant API endpoints, where no single customer's traffic can fill a full GPU on its own.

Dividing a physical GPU: NVIDIA MIG and AMD GPU partitioning

Understanding the mechanics first makes it possible to read the performance claims that follow without over-generalizing them.

On the H100 and H200, MIG supports up to seven isolated instances per GPU. Each instance gets its own slice of streaming multiprocessor (SM) compute, its own L2 cache allocation, its own memory controllers, and its own share of HBM. That isolation is strict: one instance has no path to read or write another instance's memory. That makes MIG well suited to multiplexing many smaller models across isolated tenants on a single card, because each one gets a hard guarantee that neighbors cannot see or touch its data.

The two platforms differ in how flexible those partitions can be. MIG partitions follow fixed, predefined profiles, equal or unequal in size, chosen at provisioning time and not easily reshaped afterward. AMD's chiplet-based design on the MI300X is more flexible, so you can pair compute and memory domains in configurations that don't map to a single fixed profile list.

Both platforms share one hard constraint. Partitioning divides a GPU's existing bandwidth and memory. It does not multiply them. A partition can never get more HBM or more interconnect bandwidth than the physical slice it was assigned, and the aggregate resources of the chip are fixed regardless of how many ways they get split.

Research on GPU multitasking in the LLM era places MIG-style static partitioning inside a broader set of sharing strategies, alongside fine-grained spatial sharing and time-multiplexing. Each of these approaches trades off isolation strength against scheduling overhead differently, and MIG is toward the strict-isolation, lower-flexibility end of that spectrum.

Where partitioning delivers real density gains

Partitioning produces a real, measurable throughput gain for the right class of workload, and that gain has a clear boundary. On AMD's MI300X, comparing vLLM running in SPX mode, a single partition using the full GPU, against DPX mode, two partitions carved from the same total resources, DPX mode delivered higher peak throughput on a quantized Mistral-Nemo-Instruct-2407 model. When vLLM split the same GPU into two smaller, independently scheduled instances, it used the available silicon more efficiently than it did running as one large instance.

The reason this works for smaller models follows directly from the imbalance described earlier: a model small enough to leave most of a full H100 or MI300X idle during decode doesn't need the whole chip. If you give it a dedicated partition, and give a second model instance a second partition, both can run in parallel without either one starving the other for compute or memory.

That mechanism has a hard ceiling. A large model that needs the full GPU's memory and bandwidth to run at all gains nothing from being handed half a GPU. The partition simply runs out of HBM or bandwidth first, so no amount of clever scheduling can recover throughput the hardware slice cannot physically support.

Partitioning's value extends past throughput into isolation guarantees that matter for shared inference infrastructure. Each MIG instance carries its own memory protection, so one tenant's runaway request cannot read or corrupt another tenant's KV cache sitting in a neighboring partition. If you run a shared inference API serving multiple customers off the same hardware, that guarantee backs both your security requirements and quality-of-service commitments that batching alone cannot offer.

Research on static partitioning paired with fine-grained CPU offloading treats GPU underutilization as a problem that partitioning by itself doesn't fully solve, arguing that offloading strategies need to run alongside partitioning. Separately, research on ParvaGPU documents spatial GPU sharing for large-scale DNN inference in cloud computing, aimed at improving GPU utilization and cost efficiency, because dense packing of many smaller models is the normal workload pattern there, not the exception.

The strongest case against partitioning comes from smaller deployments. Below roughly eight GPUs per model, chunked prefill and solid batching inside a single, unpartitioned instance capture most of the available density gain, without the operational work of provisioning, monitoring, and rebalancing multiple isolated partitions. That objection holds for throughput alone. It doesn't address tenant isolation or QoS predictability, and partitioning's real value is that it guarantees a resource slice to a given workload or customer, something batching inside a shared instance cannot do no matter how well you tune that instance.

Partitioning solves the problem of dividing compute and memory across tenants cleanly. What happens inside each partition once it's running, and specifically how its KV cache behaves, is where the next set of constraints begins.

How die scaling complicates fine-grained scheduling inside partitioned instances

As GPU dies grow larger and more complex, and as partition counts rise to match, the scheduling assumptions baked into inference engines like vLLM start to strain, introducing latency variance that cuts against the isolation partitioning is supposed to guarantee. Research dissecting how die scaling affects fine-grained GPU scheduling finds that larger, more complex dies produce scheduling interference patterns, and you won't see these if you only look at the partition boundary itself.

In practice, that means two partitions that are fully isolated at the hardware level, with separate SMs, separate memory controllers, separate HBM allocations, can still contend for shared scheduling resources beneath that boundary: warp schedulers, shared slices of L2 cache, and queuing at the memory controller. That contention degrades latency without collapsing throughput. It appears as latency jitter in co-located partitions, intermittent and hard to attribute to a specific cause without close instrumentation.

This matters most for latency-sensitive inference, where time-to-first-token (TTFT) service-level agreements leave little room for variance. If a burst of long-context prefill requests lands on one tenant's partition, it can push latency spikes into a co-located partition, even though the two are supposedly walled off from each other by hardware-level isolation.

AMD's NPS memory partitioning, specifically NPS4 paired with CPX, is partly a response to this exact problem. By shrinking the NUMA domain size, NPS4 reduces the memory latency asymmetry between XCDs for each partition's working set, narrowing the gap that causes jitter.

The practical takeaway for anyone configuring partitions is that partition count should track the memory access pattern of the model being served, not just its parameter count. A model with high KV cache churn, constantly evicting and reloading cache entries, behaves very differently under partitioning than a model with stable, reusable prefix caches, and treating the two the same during provisioning invites the jitter described above.

The KV Cache as the Binding Constraint Inside Each Partition

Every partition inherits a reduced slice of HBM, and that reduction makes KV cache management inside the partition far more consequential than it is on a full, unpartitioned GPU. A partition can be perfectly efficient on the compute side and still stall out because its KV cache headroom has run dry. The partition hands over an isolated slice of compute and HBM. How well that slice performs depends on how the memory inside it gets used.

PagedAttention, the memory management scheme behind vLLM, addresses this directly by splitting KV cache into small, reusable pages allocated on demand rather than reserving one large contiguous block per request. That approach cuts memory waste substantially and lets more concurrent requests fit into whatever HBM a partition has, making PagedAttention a foundational primitive for serving multiple tenants out of a partitioned instance.

For multi-tenant partitions specifically, semantic-aware eviction policies add a further layer of control: tokens belonging to shared prefix caches, system prompts, common instructions, repeated boilerplate, get retained preferentially, while tokens from private context unlikely to recur get evicted first. Harvest research on peer-to-peer GPU caching for LLM inference pushes this further still, exploring opportunistic reuse of cached data across GPUs, which becomes directly relevant once multiple partitions on the same physical card, or across several cards, are serving requests that share overlapping prefixes.

The quality of a partition's cache management determines how many concurrent requests it can actually sustain at its target latency. A partition with a weak eviction policy will serve fewer requests than its raw compute allocation would otherwise allow, regardless of how well the partition boundary itself was drawn.

Precision formats and KV cache compression as tools for recovering headroom inside a partition

Partition HBM is a fixed budget set at provisioning time, and precision format is one of the few levers that changes how far that budget stretches. BF16 remains the standard baseline for serious inference deployment, natively supported on NVIDIA Tensor Cores from the Ampere architecture onward, and used widely across the Llama, Qwen, and Mistral model families.

FP8 cuts memory per parameter roughly in half compared to BF16. For a large model running on a partition with a constrained HBM slice, that difference can decide whether the model fits on the partition at all, and how much HBM is left over for KV cache once it does. FP8 also roughly doubles Tensor Core throughput relative to BF16, a gain that matters specifically for prefill-heavy workloads running inside a partition, where compute demand spikes briefly before settling into the steadier, memory-bound decode phase.

KV cache quantization pushes the same logic further into the cache itself. Research on KV cache quantization reports 3-bit quantization achieving under 0.1 perplexity degradation on standard benchmarks, a level of compression that cuts KV cache memory substantially while leaving model quality nearly untouched. Inside a partition with a fixed HBM ceiling, that saving converts directly into more concurrent requests you can serve from the same physical slice.

A separate technique addresses the model weights. The DFloat11 (DF11) approach applies lossless compression to BF16 weights by exploiting the skewed distribution of BF16 exponent values through entropy coding, shrinking the weights themselves without giving up any numerical precision. That makes it possible to run a given model on fewer physical partitions than it would otherwise require, functioning as a direct multiplier on partitioned deployment density.

The common thread across both techniques is that you can't compress BF16 and FP8 well with general-purpose methods. Their bit-value distributions are non-uniform, so getting real gains out of them requires entropy coding and quantization strategies built to understand that structure. On the KV cache side specifically, newer approaches combine eviction and compression into a single pipeline: ChunkKV preserves cache in semantically coherent chunks rather than evicting token by token, and RocketKV runs a two-stage process, first a coarse, permanent eviction pass, then a fine-grained sparse attention step that selects which remaining tokens to attend to. Both are documented in the ACL 2026 survey of KV cache techniques, and both apply directly to partitioned serving, where every megabyte of HBM recovered from the cache is a megabyte available for another concurrent request.

The storage spillover problem: when a partition's local HBM runs out

Even with FP8 weights, quantized KV cache, and a well-tuned eviction policy, a partition's HBM slice is still finite, and sustained load eventually exhausts it. At that point, the request's fate depends entirely on what lies outside the partition's HBM boundary and how fast that layer can respond.

Once KV cache data has to move off the partition's own HBM, perhaps to host memory, to another GPU's HBM over an interconnect, or to a storage tier further down the stack, the speed of that path becomes the new bottleneck, replacing whatever compute or memory efficiency the partitioning scheme achieved upstream. A partition can be perfectly sized for its model and perfectly tuned on eviction policy, but it can still produce a slow response if the system behind it cannot retrieve spilled cache data fast enough to keep the decode loop fed.

This is why partitioning cannot be evaluated purely as a GPU architecture question. The ceiling on inference density is set jointly by how well compute and HBM are divided inside the chip and by how well the infrastructure outside that boundary handles the overflow when local memory isn't enough. Partitioning narrows the problem to a smaller, well-defined slice of hardware. Whether the density gains partitioning promises actually hold up under real, sustained production load depends on what happens when that slice runs out.