Storage Bottlenecks During Distributed Training Checkpoint Saves

Storage bottlenecks during distributed training checkpoint saves are a layered problem. To cut save time, the data path must be shorter, framework serialization must be removed, concurrent writes must be coordinated, and the work left over must be reduced or hidden. Fast NVMe matters, but it cannot compensate for CPU staging, metadata pauses, or a saturated shared fabric.

Why checkpoint I/O becomes a cascade of bottlenecks at scale

A checkpoint must preserve the full training state, including weights, optimizer states, and gradient shards, before the next failure makes recovery necessary. That makes it different from an ordinary large-file transfer. Optimizer states are often as large as or larger than model weights, so models with hundreds of billions of parameters have checkpoints that span multiple terabytes.

In a synchronous save, that burden becomes a structured pause in the GPU compute pipeline. Every second spent waiting produces no gradient progress, but saving less often exposes more completed work to loss. The checkpoint interval must balance save overhead against expected recomputation after failure. As the number of nodes grows, hardware failure becomes more likely and that interval shrinks, so the save path has to handle more frequent saves. The resulting bottlenecks are layered and must be understood one layer at a time.

The CPU bounce buffer: how the default I/O path wastes the most time

The first layer is the physical route from GPU memory to storage. The traditional checkpoint path moves data from GPU HBM across the PCIe bus into CPU pinned memory, then through an OS kernel buffer to the NVMe driver. The CPU's intermediate staging copies consume memory bandwidth as overhead for training and persistence.

For every byte written to NVMe along this path, the CPU handles it at least twice, once in pinned memory and once in the kernel buffer. This amplification becomes expensive when weights and optimizer states span a large BF16 checkpoint. At that scale, the standard CPU-staged path can take 4–5 minutes per save.

Open Lake targets a related problem in LLM inference storage through KV cache residency across GPU and NVMe tiers. Checkpoint persistence and cache residency have different jobs, but both need to move large amounts of data while keeping the CPU out of work it does not need to do. In either case, evaluating NVMe in isolation misses the cost of the route into and out of it.

Metadata serialization and GIL contention: the framework-level bottlenecks above the hardware path

Above that physical route, the checkpoint framework can impose its own sequential delay. The Ling 2.0 technical report says that distributed checkpoint metadata takes up most of the save time for Ling-1T at trillion-parameter scale. The team exploited the fact that model architecture does not change between checkpoints and introduced a metadata cache. That change cut save time from 269 seconds to a small fraction of that. The improvement came from avoiding repeated coordination work.

Concurrency introduces another cost even when the write is nominally asynchronous. PyTorch DCP uses background threads for metadata collectives and storage uploads, but those threads compete with trainer threads for the Python GIL. That contention can suppress GPU utilization during the checkpoint window.

Orbax, Google's JAX-native checkpointing library, approaches the problem by abstracting distributed storage coordination and inter-process synchronization to shorten the sequential critical path. Its reported save performance exceeds comparable PyTorch implementations by a meaningful margin. Hardware bandwidth and framework concurrency must be dealt with together, because faster storage cannot remove serialization in the framework above it.

Parallel write contention and network fabric saturation across a large cluster

A node with an efficient local path can still slow sharply when hundreds of peers save at once. Checkpointing creates an all-to-storage traffic spike rather than the steady training-data reads for which a filesystem may have been sized. When that burst saturates shared resources, individual node throughput collapses.

Lustre addresses parallel writes through striping, which distributes file data across multiple storage targets. Its stripe configuration must fit the checkpoint write pattern, because a misconfigured stripe width serializes work that could proceed in parallel. In clusters where inter-node communication and storage traffic share a fabric, checkpoint writes also compete with all-reduce gradient synchronization. Storage congestion can consequently extend training step time as well as save time.

Meta's Tectonic filesystem is designed to let thousands of GPUs save and load checkpoints in a synchronized fashion. It shows why the storage architecture must be built to coordinate bursts, rather than leaving them for a final bandwidth adjustment. The same constraint applies to production inference systems, including Open Lake's GPU and NVMe tiering context, when many concurrent actors contend for shared storage.

GPUDirect Storage: bypassing the CPU bounce buffer at the hardware level

GPUDirect Storage addresses the first bottleneck by establishing a direct DMA path between GPU memory and NVMe or networked storage. A DMA engine moves data between GPU HBM and the storage controller without the CPU pinned-memory detour. That route bypasses the kernel buffer and CPU memory copies of the traditional path. It shortens the path and stops consuming CPU memory bandwidth for staging.

Across thousands of ranks, removing those copies addresses a structural cost. This helps most when very large models need frequent saves to limit how much work is lost during recovery. GPUDirect Storage requires compatible NVMe devices or storage controllers, with storage on the same PCIe domain or reachable through RDMA. It is an end-to-end I/O design.

Eliminating the bounce buffer is necessary to remove that particular waste, but it is not sufficient to make checkpointing fast. Metadata preparation still precedes storage activity, and concurrent writers still share finite fabric capacity. The effect of removing copies through a direct path must therefore be evaluated with coordination and the remaining layers intact.

Asynchronous checkpointing and shadow clusters: overlapping the write with forward training

Asynchronous checkpointing changes when the storage work happens and can leave its volume unchanged. In the standard approach, tensors are copied device-to-host into CPU memory after a training step. The data is then written to persistent storage while the GPU starts the next step. This overlap makes overhead small when the D2H copy is fast and system RAM can buffer the full checkpoint. PyTorch Distributed, NVIDIA NeMo, and DeepSpeed support asynchronous checkpointing with that substantial memory requirement.

The residual copy still matters because D2H movement competes with kernel launches and memory bandwidth. For a large checkpoint, the transfer can extend into the next compute window. Orbax has asynchronous saving built in as a core primitive, with a blocking commit kept separate from a non-blocking write. That separation clarifies what remains on the critical path, but the practical question is still how much GPU activity the save interrupts. An asynchronous write should not be mistaken for a copy-free checkpoint.

Shadow clusters intervene at a different point by decoupling checkpoint maintenance from the training GPU. Checkmate (2025) replicates reduced gradients at programmable network switches and multicasts them to a CPU-based shadow cluster, while training GPUs receive gradients through normal AllReduce. The shadow cluster applies those gradients to maintain checkpoint state. After the gradient reduction already required for training, per-iteration checkpointing adds no GPU cost.

The Checkmate paper reports that this approach reduced wasted GPU time for a flagship-scale model from over 300,000 hours to just thousands. More frequent checkpoints also reduced recomputation after failures. ZOCheck applies a related principle to fine-tuning by maintaining state incrementally on a CPU shadow, with near-zero failure-free overhead and a small cumulative logging cost.

Lossless compression of BF16 and FP8 checkpoint data: reducing what must be written

Shrinking a checkpoint before it enters the I/O path reduces pressure on PCIe bandwidth, the storage network, and disk. Compression must be lossless and fast enough not to create a new CPU or GPU bottleneck. If compression is too slow, there may be less data to write but more time spent getting it ready.

BF16 and FP8 tensors warrant format-aware treatment because their byte patterns, entropy distribution, and block structure differ from the data targeted by general-purpose compression algorithms such as LZ4 or zstd. Per-tensor methods use fixed-size chunks and entropy coding across BF16, FP8, and FP4 data. The exponent component accounts for the bulk of the savings, with FP8 tensors compressing well below their original size in the best cases. Delta compression offers a complementary approach by encoding differences between consecutive checkpoints, with redundancy increasing as training converges.

The resources compression saves depend on where it happens. The MLCommons MLPerf Storage v3.0 Supplemental Discussion describes on-GPU lossless compression combined with RDMA as a way to reduce network traffic and CPU overhead. Compressing before transport removes bytes from downstream work. That makes payload reduction complementary to direct paths and asynchronous execution.

What MLPerf Storage v3.0 reveals about real-world checkpoint I/O at scale

MLCommons announced MLPerf Storage v3.0 results on September 1, 2026, expanding coverage across training throughput, checkpointing, and inference workloads. It is the first audited AI storage benchmark to cover that full pipeline. For checkpoint architecture decisions, the important addition is a write test representing concurrent saves at scale. This gives practitioners a workload-specific reference rather than requiring them to infer checkpoint behavior from read throughput.

The large-scale checkpoint test simulates over a thousand accelerators writing a very large model checkpoint. Its aggregate bandwidth results show what architectures built for this work can do when RDMA, on-GPU compression, and parallel write coordination work together. A result reflects the combined path.

Nebius Object Storage's submission demonstrates meaningful checkpoint write scaling across eight client nodes for Llama 3 workloads relative to a single-client baseline. That establishes object storage as a participant in high-performance checkpoint writes. The benchmark also adds an S3 object storage layer for training, checkpointing, and some VDB tests. Because production clusters use object storage for checkpoint durability, audited coverage of that path makes architectural comparisons more relevant without turning one result into a universal product ranking.

A mental model for diagnosing and prioritizing checkpoint I/O bottlenecks

Diagnosis begins by locating elapsed time rather than treating write bandwidth as the complete answer. When saves are slow and both GPU and NVMe utilization are low, the CPU bounce buffer is a likely cause, and GPUDirect Storage may help if the devices and fabric support it. A dominant pre-write pause instead points toward metadata serialization or framework contention. Metadata caching and framework-level asynchronous design address that layer, while faster NVMe alone does not.

Adequate per-node bandwidth that deteriorates during synchronized writes points toward stripe configuration, shared-network contention, or the need for a scale-out architecture such as Tectonic. If those paths are efficient but checkpoint volume still dominates, format-aware lossless compression reduces the payload. If the remaining GPU stall is unacceptable, shadow clusters offer deeper decoupling at the cost of greater infrastructure complexity.

These symptoms can coexist. Lablup's operational analysis of a 504-GPU LLM pre-training cluster illustrates how failures and recovery costs interact with the layers of checkpoint I/O. Within this broader infrastructure problem, Open Lake's work on GPU memory hierarchy, NVMe at scale, and RDMA-optimized paths addresses the hardware, runtime, coordination, and payload concerns described here. Identify where time is lost, choose the mitigation that acts at that layer, and evaluate whether the saved work disappears or merely moves downstream.