All-Flash Storage Architecture for LLM Training Data

All-flash storage architecture for LLM training data must be sized for checkpoint bursts, distributed writers, and flash endurance as well as capacity. For LLM inference throughput, NVMe tiering likewise depends on separating read-mostly weights from transient KV traffic and matching placement to reuse. Faster flash keeps GPUs fed only when the data path, namespace, and write budget let the workload use its bandwidth.

Why LLM training creates a storage problem that general-purpose flash deployments were not designed to solve

Western Digital's AI storage analysis describes training as a familiar large-block sequential read workload with a disruptive exception, namely checkpoint writes. Dataset reads repeat across epochs, while checkpoints introduce sudden, high-throughput write events. A storage tier must handle both access patterns, even if it handles a steady workload well. Purpose-built LLM storage infrastructure, including Open Lake, addresses this dual-workload problem rather than treating checkpointing as incidental enterprise write traffic.

Model scale makes that mismatch harder to accommodate. FLINT documents roughly 810 GB of BF16 weight storage for Llama 3.1-405B and roughly 1.3 TB for DeepSeek-V3 at 671B, before other runtime data is considered. Narrower formats reduce the space per parameter, but storing the weights still requires substantial capacity. At distributed training scale, 3D parallelism across thousands of GPUs also produces hundreds to thousands of files per checkpoint. Capacity, data movement, and metadata coordination limit one another, and adding flash alone cannot solve any of them.

How the checkpoint write pattern sets the throughput floor the storage tier must meet

Checkpointing sets the write-throughput floor because a save must finish within the time the training run can afford to lose. That requirement depends on model size, parallelism, and the acceptable fraction of GPU time spent idle. MLPerf Storage v3.0 puts checkpoint data per run at roughly 105 GiB for an 8B model through roughly 18,000 GiB for a 1,250B model. The relevant sizing relationship is the amount that must be written divided by the allowed save interval. A tier that handles ordinary dataset streaming can still fall below that floor during a checkpoint.

However, the drive bandwidth is only part of the save path. CPU staging can become a bottleneck, while GPUDirect Storage bypasses that staging through a direct data path between storage and GPU memory. Namespace coordination can also keep concurrent writers from using all the bandwidth beneath them. The storage format adds another constraint because FP8 training can use FP8 matmuls while retaining high-precision master weights. The object written is then the BF16 master checkpoint, so sizing from the compute format understates the save volume.

Why the read and write access patterns in LLM training are asymmetric, and what that means for endurance budgeting

The read side calls mainly for sustained bandwidth. The write side has a different shape, with intense checkpoint events separated by periods dominated by dataset reads. Treating both as a uniform workload can over-provision steady-state throughput while under-provisioning endurance. Finishing a save on time requires sufficient peak bandwidth, while cumulative writes determine how much flash life the run consumes. Endurance is therefore a binding write-side constraint even after the checkpoint throughput floor has been met.

That budget must account for physical writes resulting from the bytes submitted by the training framework. Filesystem write amplification, namespace layout, and garbage collection all affect the program-erase cycles consumed per run. Uneven write placement can wear out some flash faster even when there seems to be enough capacity overall. For specialized storage layers such as Open Lake, tiering and workload-aware allocation address this difference between sequential reads and concentrated checkpoint writes. The same distinction matters for KV storage, although its read/write ratio must be established from the specific workload.

The “HBF Sucks?” characterization uses a 3D-ICE thermal model to show that write-heavy KV traffic can push a TLC flash tier to its thermal limit below peak bandwidth. This finding shows why a write budget tied to the actual traffic must accompany advertised bandwidth.

How NVMe-oF and RDMA change what "local" storage means for distributed training

In a distributed cluster, training nodes can use flash beyond their own attached NVMe drives. NVMe-oF and RDMA connect remote flash pools through a high-throughput fabric. NVMe-oF avoids the SCSI emulation layers of older protocols, while RDMA removes CPU intervention from the remote data path. The architectural boundary consequently shifts from a node's attached storage to the flash accessible across the fabric. That broader pool still needs software that can coordinate access without serializing the job.

This makes POSIX versus object storage a workload decision rather than a simple preference for familiar interfaces. Shared-directory metadata contention persists even with a fast fabric, and independent object naming changes how applications manage their namespace. MLPerf Storage v3.0 added an S3 storage access layer that was used in roughly one-sixth of its submissions. The choice depends on the training framework's metadata access pattern and the namespace semantics it needs.

What MLPerf Storage v3.0 reveals about the throughput and namespace requirements that real training workloads impose

MLPerf Storage v3.0 covers training I/O, checkpointing, KV cache, and vector database workloads. That breadth matters because a result from one access pattern cannot establish fitness for all the others. Its checkpoint workloads show both how much data must move and how writers must coordinate to save distributed state. Leading results require hardware provisioning together with deliberate striping, namespace layout, and parallel access. The benchmark therefore makes architecture visible in a way that raw capacity or individual drive specifications do not.

The KV benchmark also reveals why workload definitions matter. It models a conversation turn that writes its cache once and reads it repeatedly, producing a read-dominated pattern. That pattern differs from transient KV offloading in which writes can outnumber useful reads. A system suited to the former is not automatically suited to the latter, even if both are described as KV storage. The useful comparison is whether the tested object sizes, reuse, concurrency, and write pressure resemble the intended deployment.

Namespace demand is equally concrete. MLPerf Storage v3.0 spans 16 checkpoint files for an 8B model to over 2,000 files for a 1,250B model. That growth means the storage system must scale coordination as well as data transfer.

How should inference storage tiers use HBF for weights rather than transient KV storage

High-Bandwidth Flash connects stacked NAND to the accelerator through a wide memory interface. It offers flash-scale capacity with lower read latency and higher bandwidth than a conventional SSD, but those properties do not guarantee lower inference latency. The “HBF Sucks?” characterization tested a drop-in HBF replacement under an unmodified Mooncake-style KV offloading stack using four two-hour Qwen production traces. It found higher average end-to-end latency and lower maximum SLO goodput. In those traces, read I/O was not the bottleneck, writes outnumbered reads, and thermal limits kept bandwidth from staying at its peak.

The distinction is between a faster device and a useful data placement. Changing HBF read/write latency by 3.75× moved end-to-end latency by less than 1% in the characterization. Read-mostly weights, shared prompt prefixes, and read-only expert weights can be a better fit because reading them many times can justify placing them there without the same transient write pressure. FLINT, from Huawei, ETH Zürich, and HUST, targets weight storage through a burst-buffer controller, phantom-plane refresh, and a read-only FTL. Those mechanisms coalesce reads, keep refresh off the critical inference path, and specialize translation for read-mostly access.

HBF is therefore a promising capacity tier for selected inference objects. Its adoption for transient KV storage should wait for production-ready write budgeting and reuse-aware placement. For present NVMe-based designs, the governing questions remain what gets reused, what must be written, and which transfers actually delay GPU work.

How lossless compression of BF16 checkpoints changes the endurance and capacity calculus for the flash tier

Lossless BF16 compression acts directly on the checkpoint write volume established earlier. Because the authoritative stored object is the BF16 master weight file, compressing that object reduces the bytes sent to flash without changing its recovered contents. Retaining checkpoint history also takes less capacity, but that saving is secondary to reducing cumulative writes. Compression complements the required accounting for and control of filesystem and device write amplification.

BF16 requires more than treating the checkpoint as an arbitrary byte stream. Its exponent and mantissa fields have statistical structure that general-purpose compressors do not exploit well. ZipServ presented lossless BF16 compression at ASPLOS '26 as a purpose-built approach to this problem. Repacking BF16 into FP8 is a different choice and must not be confused with lossless retention of the master checkpoint.

KV compression needs its own accuracy decision. Dell Technologies researchers' 2026 survey distinguishes deployment contexts requiring lossless exact-match reuse from those permitting lossy semantic reuse. Dictionary-based sparse coding for compute-in-flash reduces dynamic KV writes by storing sparse codes against static dictionary vectors, but it trades a small accuracy budget for fewer writes. That makes it a workload-specific option.

How to structure the namespace and parallelism layout of an all-flash tier so that thousands of GPU writers do not serialize against each other

Raw flash throughput is a ceiling that the namespace can prevent a training job from reaching. Thousands of GPU ranks may write separate checkpoint shards concurrently, yet still contend on shared metadata. If those shards land in a directory governed by a single metadata lock, coordination serializes otherwise parallel writes. Shard placement and directory structure must therefore follow both the job's parallelism degree and its total checkpoint size. The file-count growth exposed by MLPerf is a direct test of that requirement.

Striping must then connect logical parallelism to physical bandwidth. Checkpoint shard placement should distribute rank writes across distinct drive sets without a shared physical bottleneck. Dataset streaming needs striping that maximizes aggregate sequential bandwidth. S3-compatible object storage avoids shared directory state through independent object naming, although applications must manage their own namespace semantics. Neither the access protocol nor the striping policy can be chosen independently of the framework's behavior.

Open Lake designs its storage systems around purpose-built namespace layouts, checkpoint-aware striping, NVMe-oF integration, and endurance budgeting. These are relevant criteria for evaluating it among filesystem and object-storage options because they address the path between concurrent writers and physical drives. The practical requirement is that metadata handling preserve the parallelism the training job already exposes.

The architecture decisions that determine whether all-flash storage keeps GPUs fed or becomes the training bottleneck

The decision framework starts with model size, parallelism degree, context length, precision format, and checkpoint frequency. From those inputs, size the throughput floor for the checkpoint burst. Then calculate endurance from checkpoint volume and applicable KV write traffic, including write amplification from the filesystem and namespace layout. Design shard placement and directory structure to preserve concurrent access at the intended parallelism.

Next, choose POSIX or S3-compatible access from the framework's metadata behavior. Use lossless BF16 compression to keep checkpoints and cut write volume, and choose KV compression to meet the deployment's accuracy requirements. Evaluate HBF for read-mostly inference weights, but defer transient KV adoption until write management and placement are ready.

Norway's National Library illustrates why that separation matters. Marius Husnes, its head of IT platform, identified data quality, cleaning, and pipeline throughput as the binding constraint. The institution used 2 PB of Huawei OceanStor Dorado all-flash arrays for its AI pipeline, distinct from a 60 PB preservation archive optimized for durability. The training tier needs its own throughput and latency requirements. Size the burst, budget the writes, preserve writer parallelism, and choose each tier for the data it will actually hold.