Object Storage Throughput and Prefix Sharding for Training Pipelines

Object storage throughput in training pipelines depends on namespace design before it depends on faster hardware. Prefix sharding spreads concurrent requests across the store’s request-handling partitions, while NVMe tiering supplies hot-data bandwidth and absorbs checkpoint bursts. For pipelines that also support LLM inference, the storage architecture must address both constraints, because a fast physical tier cannot compensate for concentrated object-store requests.

Why object storage stalls training pipelines even when hardware is adequate

A training job can stall for hours while its GPU cluster sits idle and costs accumulate. Hardware dashboards show no obvious resource exhaustion. The network has spare bandwidth, and storage devices are below their throughput limits, but workers cannot get data fast enough to keep the model running. The missing signal is the queue of object-store requests competing for the same partition’s capacity.

Object storage is the structural home of training data because it is durable and cheap, but its request semantics work against highly parallel access. Per-prefix rate enforcement and per-request latency constrain how much work thousands of simultaneous readers can complete. Training is primarily throughput-bound. An inability to sustain enough independent reads makes the whole cluster wait, creating an architectural bottleneck that standard hardware monitoring can leave hidden.

How object store namespaces partition request load, and what a prefix actually controls

A prefix is the leading portion of an object key, such as dataset/train/ in dataset/train/shard_0001.parquet. It looks like a directory path but generally is not one. The string influences how an object store partitions its internal index and routes requests, making the namespace part of throughput engineering. Requests concentrated in a constrained partition compete for its capacity regardless of the network bandwidth or GPU count available.

The distinction between a prefix and a partition is important. Amazon S3 general purpose buckets have a flat structure, and S3 decides where to split partitions from the rate of requests over time. A declared prefix may share a partition with other prefixes. S3 Express One Zone directory buckets are an exception to the flat-namespace description because they support a genuine hierarchical namespace. The provider-agnostic principle is therefore to distribute request load across independent partition capacity.

For example, placing every shard under dataset/train/ concentrates access in one naming range. Putting objects under many prefixes gives the store more of the namespace across which to spread requests. The filenames are therefore more than organizational conventions. When a concentrated range remains partition-limited, adding workers increases contention and yields less than proportional gains in read throughput.

Prefix sharding — what it is and how to implement it in a training dataset

Prefix sharding redistributes objects across distinct prefixes so concurrent requests can reach independent partitions. A shard’s prefix can come from a hash of its index, a random token, or a structured multi-level key. A pattern such as aa/bb/shard.parquet shows a two-level prefix tree, while hash-derived prefixes spread shard keys across more of the naming space.

Instead of putting every shard under one training prefix, the writer assigns each shard to a prefix using the chosen distribution rule. Worker-indexed namespaces offer another arrangement, with each worker’s object range under a distinct prefix. In either case, the useful property is the spread of simultaneous requests across the available partition capacity.

File size determines how much that distribution achieves. Millions of tiny files still stress metadata and can saturate request capacity even in a sharded namespace. Aggregating small samples into fewer, larger shards reduces per-object overhead, while columnar formats such as Parquet support large, sequentially readable shards. Very large files reduce the number of prefixes needed, so prefix count and shard size belong in the same design decision.

Worker assignment completes the static layout. Workers can retain their own prefix ranges, receive reassigned ranges each epoch, or draw from the full prefix space using hash-distributed keys. For throughput planning, the criterion is whether concurrent readers distribute their load across the available partition capacity.

How prefix sharding interacts with multi-worker data loaders and parallel I/O at training time

A sharded dataset does not make a sequential loader parallel. The loader must issue independent requests concurrently through multiple reader threads or processes. Assigning readers disjoint prefix ranges lets those requests fan out across separate partitions, so aggregate bandwidth can exceed a single partition’s limit and approach the store’s total capacity. Namespace design creates the opportunity, and reader concurrency uses it.

Prefetch extends that concurrency across successive batches. The loader should fetch the next batch’s shards while the model works through the current batch. This requires enough prefix diversity that future reads do not simply queue behind current reads in the same partition. Streaming makes the relationship especially clear because chunks arrive through a continuous sequence of requests. A concentrated namespace restricts that stream, while a distributed one permits parallel service.

A local NVMe cache changes which requests reach object storage. Prefix-aware assignment makes cache population deterministic, worker-local and independent of a shared cache namespace.

Checkpointing at scale as a prefix-design problem, not just a bandwidth problem

Modern LLMs trained with 3D parallelism across thousands of GPUs generate hundreds to thousands of checkpoint files per save event. Each file needs its own object-store write request. When those writes all target a constrained range such as checkpoints/step_10000/, they create a thundering herd, and a stalled save can leave the entire cluster waiting.

Step-number and rank-indexed prefixes spread concurrent writers rather than placing every rank under one checkpoint directory. Single-file aggregation addresses a different part of the same workload by reducing request count and metadata pressure. It is preferable where collecting the tensors costs less than handling thousands of separate object-store writes.

Node-local NVMe receives the checkpoint first, and asynchronous replication then sends it to object storage while training continues. The local tier absorbs the burst, turning a spike of remote requests into a steadier stream. Prefix distribution spreads pressure across partitions, while buffering spreads it over time, so they complement each other. This pattern also improves cost efficiency by reducing the time an expensive GPU cluster spends waiting for the object-store write path.

What MLPerf Storage v3.0 reveals about object-store throughput at training scale

MLPerf Storage v3.0, released September 1, 2026, added an S3 object storage access layer that can be used in place of its existing POSIX layer. The release also introduced KV cache and vector database workload tests, while checkpointing was already part of the benchmark. The S3 layer can be used for training, checkpointing, and some vector database tests. About one-sixth of submissions used it, so object-store access was measured directly instead of being taken for granted as a background service.

The checkpointing results show throughput increasing with data-node count. That scaling is consistent with the value of independent request handling, because additional nodes provide more infrastructure over which to distribute work. It is not, by itself, proof of prefix contention in a particular pipeline. Node-level parallelism and prefix-level distribution operate at different layers, and a benchmark result does not establish a fixed correspondence between them.

The KV cache workload extends the sizing question into inference. Its read demand frames the bandwidth that storage may need when supporting model serving as well as training. But S3 support for training and checkpointing alone cannot show that every KV cache result measures object-store performance. The useful comparison is between workload demands and the tier expected to serve them, rather than between a headline benchmark result and an undifferentiated storage pool.

What storage architecture supports LLM inference throughput once prefix sharding removes contention?

Once requests are distributed effectively, the constraint shifts to aggregate device and network bandwidth. NVMe provides high sequential throughput and low latency for active training shards and checkpoint staging. RAID0 across multiple NVMe devices supplies device-level parallelism that matches the namespace’s parallel access pattern. For systems also serving LLM inference, the KV cache workload adds a further bandwidth demand that the physical tier must accommodate.

The network path can become the next bottleneck. At high link speeds, TCP interrupt handling and memory copies use host CPU resources that the pipeline needs for other work. RDMA moves transport work off the host CPU through kernel bypass, reducing that source of I/O delay. By 2026, it has become standard infrastructure for serious AI training deployments.

NVMe over Fabrics with RDMA transport approaches local-NVMe latency while allowing a distributed pool to supply aggregate bandwidth beyond one node’s local storage. GPU Direct Storage addresses another hop by allowing the storage controller or NIC to transfer data directly to or from GPU memory. Removing the CPU bounce buffer reduces the copies a checkpoint traverses.

These technologies solve different constraints in one chain. Prefix sharding distributes requests, NVMe supplies device bandwidth, RDMA reduces transport overhead, and direct GPU access removes an intermediate memory copy. For inference cost per token, tiering therefore needs evaluation against serving demand. The benchmark evidence described here does not show a measured drop in cost-per-token.

Diagnosing a namespace-limited pipeline before redesigning it

Low GPU utilization alone cannot identify the storage bottleneck. In a hardware-limited pipeline, network interfaces are saturated or storage devices are already delivering their available throughput. More prefix diversity cannot add physical bandwidth. In a namespace-limited pipeline, those resources remain underused while worker threads spend substantial time waiting on object-store API calls.

Object-store access logs or provider metrics can show the request rates and latency for each prefix. When a few prefixes get a large share of requests and latency is high in those same ranges, the requests may be crowding into the same partitions. Data-loading stalls in profiling traces strengthen that diagnosis when they do not coincide with network saturation.

A controlled comparison tests the explanation before a full redesign. Duplicate a subset of shards under a wider prefix namespace, then run a subset of workers against each layout with the same hardware provisioning. Increased throughput from the wider layout supports the conclusion that prefix contention constrained the original path. The test isolates namespace layout instead of changing storage and concurrency together.

Data-loading time, worker wait time, and per-prefix request distribution should be tracked as part of routine pipeline monitoring, so a queue that forms as concurrency grows can be seen.

A practical namespace design checklist for training pipelines at scale

The practical design review begins before dataset ingestion. Choose a prefix depth and sharding key that produce at least as many distinct prefixes as peak concurrent workers, with headroom for growth. Aggregate small files into right-sized shards, and avoid filenames that cluster lexicographically where the backend could place adjacent keys in adjacent partitions. These decisions establish the request distribution that the runtime must use.

The loader configuration should then match that distribution. Enough concurrent readers issue independent requests across prefixes, while sufficient prefetch depth lets upcoming reads overlap with the current forward pass. For repeated-epoch workloads, pin hot shards to local NVMe through prefix-aware, worker-local cache assignment. Reader count alone is insufficient if all readers still compete for the same range.

Checkpoint design needs its own review of the pressure that bursts put on storage. Use rank-indexed and step-indexed paths, stage writes to local NVMe, and replicate asynchronously to object storage. Prefer aggregation where its cost is lower than the metadata overhead of fragmented writes.

Finally, provision the physical tier for the parallelism the namespace enables. NVMe bandwidth, RAID0 or NVMe-oF pools, RDMA transport, and direct GPU transfers should address the corresponding constraints in the storage path. The acceptance test is sustained pipeline throughput. Treating the namespace as an engineering artifact makes storage part of the training design rather than a naming convention chosen after the hardware arrives.