Weight Loading Latency and Cold-Start Optimization

Cold start in LLM serving is not a single delay but a chain of four separate phases, and each one has its own bottleneck. Treating it as one problem leads engineers to fix the wrong stage: they speed up storage and wonder why the server still takes as long to answer its first request.

Cold start as a pipeline problem

A warm inference server answers in milliseconds per token. A cold one can take tens of seconds before it produces anything at all, and that gap, not the absolute number, is what breaks scale-to-zero designs for interactive workloads. A user who expects a chat response in under a second will not wait twenty or thirty seconds for a container to wake up, no matter how fast it runs once it's warm.

That cold start breaks into four phases, in strict sequence: pulling the container image, loading model weights from storage into GPU memory, building the CUDA context and capturing execution graphs, and warming up the KV cache before the first real request is served. Each phase is gated by a different resource. Image pull depends on image size and registry bandwidth. Weight loading depends on NVMe and PCIe throughput. Graph capture depends on CPU speed and compiler behavior; the GPU plays no role in it. KV cache warmup depends on memory allocation strategy and how expensive the first prefill turns out to be.

The first systematic characterization of vLLM startup, published in 2026, broke the process into six constituent steps and found it to be predominantly CPU-bound. That finding matters because it runs against the instinct of most infrastructure teams: faster GPUs do not shorten initialization, since the GPU sits idle for much of it while the CPU does the work of staging data and compiling graphs.

Put rough numbers on a large model with its container image already cached locally. Weight load from NVMe takes about 45 seconds. CUDA context setup and graph capture add a further substantial block of time on top of that. KV cache warmup adds still more before the engine can answer anything. If the image is not cached, several additional minutes get tacked on before any of this even starts.

These four phases interact in ways that punish partial fixes. If you accelerate weight loading alone, the CPU-bound graph capture phase, the larger cost in many deployments, stays exactly as slow as it was. If you fix graph capture alone, pulling data off disk still sits in front of everything else, because that's the first large delay. A cold-start strategy has to address each phase on its own terms, because a win in one phase is invisible to a user if the next phase in line is still the bottleneck.

Model scale, weight format, and the floor for storage I/O time

The weight transfer phase has a floor set by physics: parameter count, numeric format, and the bandwidth of the path connecting storage to VRAM. No amount of software cleverness moves more bytes than the pipe can carry.

The byte math is plain. A smaller model stored in FP16 takes up tens of gigabytes. A larger model takes up hundreds of gigabytes. All of those bytes have to cross a PCIe bus on their way from host memory to the GPU, and that bus is the immediate ceiling on every host-to-device copy regardless of what else is optimized.

Quantization cuts the physical byte count directly. FP8 cuts the footprint in half relative to FP16, and INT4 cuts it in half again. But does that reduction actually buy proportional time savings? It depends entirely on where the bottleneck sits. Over an NFS mount, even a heavily quantized model can still take many seconds to load, because the network link is the ceiling, not the size of the model. On local PCIe Gen4 NVMe, the same quantized model loads far faster, which shows that where the weights sit matters as much as how small they are.

The traditional path to the GPU runs through the NVMe driver queue, the OS kernel buffer, a block of CPU-pinned memory, the PCIe bus, and finally GPU HBM. Five hops, and the CPU sits in the middle of all of them, copying data it will never compute on, purely to stage it for the device. That staging step is the structural reason weight loading has historically been slower than the hardware's raw bandwidth would suggest.

File format changes how much of that staging is necessary. Safetensors uses a flat binary layout with an index header, which lets the loader memory-map individual shards and overlap the disk read of one shard with the CUDA transfer of the previous one. The first transformer layers can sit resident in VRAM while the last shards are still being read off disk. Legacy pickle files don't allow that overlap: they force full CPU deserialization before any transfer can begin, adding delay to this phase no matter how fast the underlying storage is.

GPUDirect Storage and RDMA as the path around the CPU bottleneck

GPUDirect Storage removes the CPU from the data path. Instead of staging weights through a CPU-pinned buffer, GDS lets NVMe drives write directly into GPU HBM over the PCIe bus, cutting a five-hop path down to two hops.

The benchmark numbers show how much that matters. On a single H100 SXM5 paired with a PCIe Gen4 NVMe drive, GDS cuts Llama 4 Scout's weight load time from several minutes down to roughly twenty-six seconds, a gain of more than seven times. On an eight-GPU H200 node, DeepSeek V3's load time drops from many minutes to a fraction of that figure, a gain of more than eight times. Both numbers describe the same underlying change: removing the CPU bounce buffer from a path that was built around it for years.

NVIDIA extends the same idea to distributed inference through NIXL, the NVIDIA Inference Xfer Library. NIXL is open source and works across vendors, so you get a single API for moving data across GPU memory, CPU memory, and storage tiers. Its backends include RDMA, GPU-initiated networking, GPUDirect Storage, NVMe, and cloud object stores, including S3 over RDMA and Azure Blob Storage. That breadth means the GDS benefit isn't confined to a single rack: it extends outward to object storage and cloud-native backends as well.

Solidigm pushes the same principle further out, pairing PCIe Gen5 SSDs with accelerator-driven NVMe-oF emulation, called SNAP. SNAP translates local NVMe commands into NVMe-oF protocol packets, which allows direct RDMA transfers between remote storage and GPU memory. The result closes most of the gap between remote and local storage: remote drives start performing close to local NVMe.

None of this is free to deploy. GDS needs specific hardware support and isn't available across every serverless environment. Where the only option is NFS-mounted storage, the CPU-staged path remains the ceiling, and the quantization and locality choices described in the previous section carry more of the weight.

CUDA graph capture, the CPU-bound wall fast storage exposes

Speeding up storage I/O doesn't make cold start disappear. It moves the bottleneck to CUDA graph capture, a phase that runs entirely on the CPU and gets no benefit from a faster GPU or more GPU memory bandwidth.

In vLLM, graph capture happens after weights are already in VRAM, and it covers the model's forward pass across a range of batch sizes. For large models, this phase by itself can consume a substantial chunk of total startup time, and if the first run also triggers torch.compile's Inductor kernel compilation, that adds further delay on top.

What makes this phase worth fixing rather than just waiting out is that it's deterministic. The same model, the same GPU, and the same configuration produce the same captured graph every time. So you can write the output to disk once and reload it on every later start, skipping the capture step. Persisting CUDA graph and Inductor caches to a mounted NVMe volume does exactly this for a standard vLLM deployment: the first run writes the cache, and every subsequent cold start on the same model and hardware reads it back. This only works where a persistent volume survives container restarts. On serverless platforms with ephemeral filesystems, that cache disappears every time the container dies, so the capture has to run fresh on every cold start.

A common intuition fails engineers here: they assume smaller files load faster, so compressing model weights should help, but general-purpose compression like gzip adds a CPU decompression step directly onto the critical path and buys almost nothing back, because model weights are close to incompressible noise. Google's measurements on GKE found zstd decompresses more than three times faster than gzip, which is the default compression format in most container runtimes, but neither algorithm shrinks weight files by much because there's little redundancy left to squeeze out. Quantization is a different kind of compression: it reduces the actual number of bytes that represent each parameter, cutting load time without adding a decompression tax. Confusing the two, treating gzip and quantization as interchangeable size-reduction tools, is a mistake that slows startup.

KV cache warmup and memory pressure as the final pipeline stage

Once weights are loaded and graphs are captured, one phase remains: initializing the KV cache and handling the first real request. The allocation choices made here do more than decide how long startup takes. They shape inference throughput for the entire life of the deployment.

KV cache memory grows linearly with context length, because every token held in context needs its own key-value pair stored. Traditional allocation reserves a contiguous block of memory sized for the maximum possible sequence length, so most of that memory sits unused in typical requests. PagedAttention, introduced by vLLM, fixes this by borrowing an idea from operating system virtual memory: it allocates KV cache in fixed-size blocks. That change drops wasted memory from a range of sixty to eighty percent down to nearly zero, and it directly increases how much context a given GPU can serve at once.

NVIDIA's answer at the hardware tier is ICMSP, the Inference Context Memory Storage Platform, announced at CES 2026. It extends GPU KV cache out into NVMe-based storage, built with NVIDIA's NVMe storage partners, and NVIDIA reports it delivers up to five times greater power efficiency than traditional storage along with up to five times higher tokens-per-second throughput.

Compression plays a role here too, and the numbers show that getting memory architecture right at this stage pays off on every request that follows, not just the first one. FP8 quantization of the KV cache cuts the per-token memory footprint in half relative to BF16, and it does so before any throughput measurement even happens. On MMLU, Llama 3 run in FP8 scores within a fraction of a percent of its FP16 baseline, which makes it effectively lossless on Hopper and Blackwell hardware. Google's TurboQuant, presented at ICLR 2026, compresses KV caches down to three bits with reported zero accuracy loss, a six-times memory reduction, and NVIDIA's KVTC, built on a transform-coding technique, achieves even higher compression than that.

For agentic and long-context workloads, the stakes of getting this allocation right extend well past the cold start itself. When the cache evicts an entry that's needed again later, the system has to recompute it, and that recomputation appears as a latency spike minutes into a session, not at startup. The allocation decisions made during warmup are infrastructure decisions, not model decisions, and they keep charging their cost for as long as the session runs.

Prefill-decode disaggregation as a structural fix for the pipeline's interference problem

Patching each phase of the pipeline in isolation leaves one structural problem untouched: prefill and decode compete for the same GPU. Prefill is compute-heavy and processes the incoming prompt in one burst; decode is memory-bandwidth-heavy and generates tokens one at a time. Running both on the same GPU pool means each interferes with the other, inflating time-to-first-token and degrading throughput exactly when a deployment tries to scale out.

Prefill-decode disaggregation solves this by assigning the two workloads to separate GPU pools. Prefill nodes handle incoming prompts and hand off the resulting KV cache to decode nodes over the network. Interference disappears, and each pool can scale independently based on its own load pattern, because the two sides of the workload don't move together.

This has moved well past research-stage experimentation. PD disaggregation is supported by vLLM, SGLang, TensorRT-LLM, LMDeploy, and NVIDIA Dynamo, and it runs in production at DeepSeek and Gemini. A single-node test on an eight-GPU MI300X system, using vLLM's MoRIIO connector, achieved substantially higher throughput within target latency bounds by separating prefill from decode on the same hardware.

NVIDIA Dynamo's KV Block Manager takes the idea a step further by offloading KV cache across GPU memory, CPU memory, SSD, and remote storage tiers. A warm node can hand its KV state to a freshly provisioned node over NIXL, letting the new node start from a warm cache. Dynamo's KV-aware routing compounds that benefit: it routes incoming requests to whichever GPU already holds the matching prefix in cache, letting that request skip prefill work it would otherwise have to repeat. If a workload has heavy prefix overlap, that routing decision cuts the effective cold-start penalty substantially.

What disaggregation really demonstrates is that the fixes from earlier sections, faster storage, cached graphs, better KV allocation, solve their own phases but don't solve the underlying resource contention between prefill and decode. That contention is a scheduling and placement problem, and it needs a different class of answer: deciding which workload runs on which hardware, not just making each workload run faster on the hardware it's already assigned to.

Proactive and predictive strategies that act before the cold start begins

The cheapest way to deal with cold-start latency is often to make sure it never has to run. Predicting which models will be needed and staging their weights ahead of time avoids the initialization sequence.

ServerlessLLM takes this approach by caching model weights across a multi-tier local storage hierarchy spanning NVMe SSDs and DRAM. When a predicted workload arrives, weight loading starts from that fast local tier instead of pulling from a remote object store, so the network-bound portion of the weight transfer phase never becomes a bottleneck. Combined with the storage, compute, and memory fixes covered in the sections above, prediction closes the loop: the phases that remain unavoidable get optimized on their own terms, and the phases that can be sidestepped through foresight are handled before a request ever arrives.