Filesystem Selection for High-Throughput AI Storage Nodes

For high-throughput AI storage nodes, filesystem selection must match the drives, workload and transport stack. NVMe and tiered storage for LLM inference throughput work best when local NVMe serves hot data, while parallel and object storage serve workloads suited to their access models. Reducing inference latency at scale requires matching KV cache movement, metadata concurrency, and RDMA paths rather than selecting a filesystem by sequential bandwidth alone.

Why filesystem choice is a first-class infrastructure decision for AI storage nodes

An idle GPU can mean that storage is too slow even when there is enough compute. As clusters grow, I/O starvation can make storage the bottleneck before compute, and the filesystem architecture can make it worse or ease it. The filesystem governs how concurrent requests reach data, with direct consequences for throughput, metadata latency, and GPU utilization. Its position between data residency and GPU demand makes it a structural choice.

That choice also depends on the layers around it. NVMe-oF, RDMA transport, and GPUDirect Storage change the data path and therefore what the filesystem needs to provide. Open Lake’s storage infrastructure research focuses on how these protocol layers, filesystem architecture, and KV cache management interact as GPU clusters scale.

The four I/O patterns AI storage nodes must handle and how they conflict

Training ingestion, checkpointing, inference serving, and KV cache offload provide the test for each architecture. Training ingestion demands large, sustained sequential reads, while checkpointing creates burst writes that must finish within the training step budget. Inference serving mixes sequential model weight loading with small random reads for indexing and experiment tracking, plus writes for results and logs. A sequential bandwidth result shows only part of what that workload needs.

KV cache offload adds a particularly demanding pattern as context lengths grow. At BF16 precision, concurrent requests can produce tens to over a hundred gigabytes of state moving between GPU HBM, CPU DRAM, NVMe, and remote storage. Neither purely sequential nor purely random, this traffic stresses metadata and small-block paths, making it central to the storage co-design questions studied by Open Lake. The conflict is architectural, since parallel filesystems favor large transfers, local filesystems face distributed throughput ceilings, and object-native systems may require changes to POSIX-dependent applications.

XFS: when a local filesystem is the right foundation

Xinnor’s two-node xiNAS deployment achieved 117 GBps sequential read and 79.6 GBps sequential write with an XFS-based stack. Write throughput showed near-linear scaling as nodes were added, while read performance dropped only modestly after an SSD failure and remained viable during a rebuild. The same platform delivered high random read and write IOPS at low latency. These results establish what that engineered stack as a whole can deliver.

The division of labor explains the result. xiRAID supplies software RAID, NFS over RDMA supplies the transport, and XFS supplies the POSIX-compatible filesystem layer without proprietary clients. This layering principle also informs Open Lake’s research into workload-specific storage paths. XFS’s maturity and kernel integration keep operational overhead low, but the filesystem itself remains single-node unless explicitly federated. It fits single-node and small-cluster systems that mostly make large transfers, but millions of small-file operations or heavy concurrent metadata access can push it to its limits.

ext4: the conservative baseline and where it breaks down for AI

ext4 is easy to start with because it is stable and works with many monitoring and backup tools, which leaves less work for the team running it. That makes it defensible for development nodes, single-GPU workstations, and archival tiers with modest throughput needs. Its journal and metadata handling, however, were designed for single-disk or small-array workloads. At high concurrency, directory-level serialization becomes a bottleneck when processes access many files in the same directory.

The crossover depends as much on shared access as on capacity because ext4 has no native scale-out capability and needs a layer such as NFS or SMB to share access, adding latency and protocol overhead. Below that crossover, it is still easy to run. Above it, the architecture conflicts with the concurrency demands of distributed training rather than merely needing a different tuning setting.

Lustre: the established standard for large-scale training storage and where its tradeoffs bite

The Isambard-AI technical paper describes a UK deployment of Lustre on an all-flash HPE ClusterStor E1000. Its 44 flash storage servers contain over a thousand NVMe SSDs and provide over 20 PiB of usable space. Expected aggregate read throughput reaches up to 2,500 GB/s, with comparably high write throughput. This is a concrete example of Lustre’s fit for large-scale training infrastructure, although expected throughput is not a measured application result.

Lustre distributes data across Object Storage Servers and separates metadata onto Metadata Servers. That separation supports near-linear bandwidth scaling as Object Storage Servers are added for large sequential operations, especially with InfiniBand. The same design can put much of the load on metadata servers when workloads create many small files, open and close them often, or use random access. KV cache offload therefore tests a different part of the architecture from training ingestion.

Operating that architecture also requires specialized expertise. Stripe counts, stripe sizes, and OST layout need workload-specific tuning, while locking and consistency limitations under high concurrency can require non-POSIX access patterns. These are costs of Lustre’s design. It remains a strong fit where sustained training bandwidth dominates and the team already has HPC storage expertise.

DAOS: object-native architecture and what it changes for AI I/O patterns

DAOS changes the starting point by making objects, rather than a POSIX filesystem over block devices, the core data model. Designed for persistent memory and NVMe, it bypasses the block device layer and provides POSIX compatibility through libdfs and dfuse. Its fine-grained, per-object locking avoids the directory-level contention associated with many small concurrent operations. That model aligns more naturally with mixed inference I/O and cached-state access than Lustre’s sequential emphasis.

Applications that assume strict POSIX semantics may need adaptation, and DAOS has fewer experienced administrators and less tooling integration than Lustre. Contracting Optane availability also reduces the persistent memory advantage, since pure-NVMe deployments lose some of the latency edge over well-tuned Lustre. The DAOS Foundation backs the project, but teams still need to learn how to run it.

Purpose-built distributed filesystems for AI: what 3FS demonstrates about RDMA-first design

The Fire-Flyer AI-HPC paper describes DeepSeek’s 3FS as a filesystem co-designed with the cluster’s storage and network. Its RDMA-first path lets compute nodes access remote NVMe without intermediate CPU copies or kernel involvement. Storage and inter-GPU communication share a Two-Layer Fat-Tree network, where traffic is kept separate to prevent congestion. The target is the combination of large sequential reads and high-throughput writes required by distributed training.

The paper reports 8 TB/s total read throughput from 180 storage nodes, each with 16 PCIe 4.0 NVMe SSDs and two high-bandwidth InfiniBand HCAs. It also reports that the Fire-Flyer 2 cluster achieved performance approximating DGX-A100 while reducing costs by half. That cost comparison concerns the co-designed cluster. Neither result establishes that installing 3FS on a different stack reproduces the same outcome.

The requirement is therefore as important as the throughput figure. Teams need RDMA-capable networking, network co-design, and the engineering resources to operate a custom distributed filesystem. 3FS’s broader lesson is to evaluate zero-copy paths and network integration as architectural properties.

How the object storage layer is changing the boundary of the filesystem decision

MLCommons’ results page identifies MLPerf Storage v3.0, released September 1, 2026, as adding an S3 object storage access layer beside POSIX. The release also introduces a distinct KV Cache test for LLM inference cache reads and writes. This makes interface selection a benchmarkable workload question rather than an assumption that AI storage must use files. MLPerf Storage working group co-chair Curtis Anderson calls S3 “a viable – and possibly preferred – alternative to filesystem-based storage” for relevant workloads.

Key-value lookups map naturally to key-addressable objects, while horizontal object storage avoids the hierarchical metadata coordination that constrains POSIX filesystems at high concurrency. That does not establish universal superiority over filesystems. Large sequential training reads and checkpoint ordering requirements still favor parallel filesystems in relevant deployments. Exposing both POSIX and S3-compatible interfaces preserves the ability to assign each workload to the access model that suits it.

Lossless compression and BF16-native storage as a filesystem-adjacent lever

Compression changes how much bandwidth is needed before a filesystem is chosen. BF16 is widely used for LLM inference, and its exponent field accounts for half of its total bits. Trained models have a highly non-uniform distribution of exponent values, creating opportunities that general-purpose compressors underuse. Format-aware lossless compression uses that structure to shrink the data while keeping the stored numerical values the same.

ZipNN, published at IEEE CLOUD 2025, demonstrates improved compression ratios and faster compression and decompression than Zstd on BF16 models. ZipServ’s repository describes a GPU inference framework that combines hardware-aware compression with fused kernels to achieve 30% model size reduction with bit-exact numerical precision. It also reports kernel-level speedup over cuBLAS.

If compressed bytes remain compressed on the NVMe transfer path, the storage bandwidth requirement falls with the volume moved. Combined with RDMA or GPUDirect Storage, that reduction can extend across the path to GPU memory. That lower bandwidth budget may let a simpler XFS-based stack do work that would otherwise need a parallel filesystem. FP8 formats E4M3 and E5M2 provide a separate precision choice at one byte per weight, with a loss of BF16 numerical precision. Precision selection and format-aware compression should therefore be evaluated together without treating their numerical implications as equivalent.

How should practitioners match NVMe and tiered storage to LLM inference throughput?

The choice starts with the main I/O pattern, the size of the cluster, the network fabric, and what the team can run. Large sequential training workloads favor Lustre or a purpose-built parallel architecture, while mixed inference workloads favor XFS-over-RDMA or DAOS under the conditions described above. KV cache offload increasingly warrants evaluation of S3-compatible object storage or purpose-built KV layers. Single-node and small-cluster deployments can use XFS-based stacks without assuming the operational burden of a distributed filesystem.

InfiniBand supports Lustre’s strengths at scale and the RDMA-first design demonstrated by 3FS, while RoCE over Ethernet is viable for XFS-over-NFS/RDMA. TCP-only environments raise the baseline latency floor. Lustre requires HPC filesystem expertise, DAOS requires a newer operational playbook, and 3FS requires deep systems engineering.

These distinctions support a tiered architecture. Tier 1 uses local NVMe with XFS or raw block for model weights and hot checkpoints. Tier 2 uses a parallel filesystem for training datasets and distributed checkpoint draining, while Tier 3 uses S3-compatible object storage for cold data, logs, and inference KV cache at scale. Asynchronous checkpointing to local NVMe followed by draining to global storage shifts the test from sustained write bandwidth to completion within the training step budget.

For inference economics, those tiers must be evaluated through cache movement and GPU utilization, with any cost-per-token improvement requiring validation. Open Lake’s infrastructure work addresses these same decisions through KV cache offload, compression-aware paths, and RDMA-optimized movement. MLPerf Storage v3.0 provides workload-specific tests for KV cache, checkpointing, and training that can validate the resulting design. Practitioners should run or review results against their target workload profile, because synthetic sequential throughput alone cannot establish whether a storage stack will keep GPUs fed.