NFS Performance for AI Inference Model Loading

NFS performance for AI inference model loading depends on four constraints, namely shared throughput, page-fault latency, cold-cache penalties, and copy amplification. NVMe and tiered storage improve LLM inference throughput when they address the binding constraint, separating hot models and latency-sensitive KV cache from bulk weight storage. Measurements should guide the choice of tuning, RDMA, or distributed storage, because each solves a different problem.

Why model loading is the unglamorous bottleneck that now defines inference economics

Inference has overtaken training as the primary AI workload, shifting the optimization target from peak training throughput to sustained, consistent throughput under variable load. A serving cluster must absorb replica starts, restarts, and model swaps without losing too much useful GPU time. Models now occupy tens to hundreds of gigabytes, so loading is a substantial transfer. Amazon SageMaker's 2026 review identifies the consequence plainly, noting that generative AI cold starts can span multiple minutes as containers and weights transfer.

Those minutes become a recurring utilization cost across a multi-model serving cluster. A GPU waiting for weights cannot contribute inference throughput, regardless of its arithmetic capacity. Open Lake's checkpoint-loading benchmarks in storage performance competitions show that storage infrastructure and data-path choices can reduce load latency by orders of magnitude compared to network-based filesystems. That makes the path from checkpoint to GPU memory a first-order optimization target. Meeting a deployment's load-time requirements requires identifying the bottleneck in that path.

What NFS was built for and what model loading actually demands from it

NFS was designed for shared access to many small files with modest per-client bandwidth. With LLM model loading, many clients instead read an enormous file sequentially, all at maximum speed at once. Shared file access remains useful, but it does not itself provide the parallel bandwidth this workload demands. When a deployment has a single server, all clients share the bandwidth that server has. The architecture therefore couples one replica's loading performance to the activity of other clients.

A diagram of the traditional load path would place the application above the kernel buffer cache, NFS client layer, and network stack, with a network hop to the NFS server and a return path toward GPU memory. Data can pass between kernel buffers and user space several times before it gets where it needs to go. With NFS over TCP, data also goes through software on the network path between the application and storage. Many serving frameworks memory-map model weights, so the OS page cache is also part of this path. When a required page is absent from local memory, loading depends on a synchronous NFS read.

The specific bottlenecks NFS introduces at each stage of a model load

The throughput ceiling is the first distinct constraint. A single NFS server puts a cap on aggregate read speed, even if each client could use more bandwidth. When replicas start together, they share that bandwidth, so each gets less than it would if it were loading alone. A 32-GPU cluster waiting on a checkpoint illustrates the resulting idle-cluster cost. The diagnostic signal is per-client throughput that falls as concurrent loads increase.

Page-fault latency can slow each read even when there is still bandwidth to spare. Memory-mapped weights that miss the page cache generate synchronous NFS round-trips. NFS-over-TCP latency is in the hundreds-of-microseconds range, compared with the tens-of-microseconds range for local NVMe. Each fault serializes part of a load that would otherwise benefit from streaming access.

When neither the NFS client nor the server has the model in its cache, the first replica pays the full remote-read cost for every page it needs. Replicas starting before those reads warm the cache can incur similar misses. A warm-cache load leaves out the cold-cache conditions that replicas face when they start at once.

Copy amplification is the remaining constraint. On the kernel-staged path, data passes through several software layers before it reaches GPU memory, so CPU-managed memory movement is many times the checkpoint's raw size. This overhead adds to the delay from the network. Throughput limits, page faults, cold caches, and copies are separable mechanisms, so a faster result in one does not establish that the others have disappeared.

How the GPU memory hierarchy interacts with the NFS data path to compound these bottlenecks

GPU HBM bandwidth is now measured in terabytes per second, while NFS over TCP rarely sustains more than a few gigabytes per second per client. The relevant infrastructure constraint is the chain between the endpoints, including the NFS stack, OS kernel buffers, CPU pinned memory in DDR5, and the PCIe bus. Its weakest link determines load throughput. Faster HBM cannot compensate for weights that have not yet crossed that link.

The CPU-staged path reads data from the network, copies it into pinned memory, and transfers it across PCIe to the GPU. As a result, host memory bandwidth can constrain loading even when the network has capacity left. AMD's LIFE framework identifies memory bandwidth rather than compute FLOPS as the binding constraint during decode. Buying faster GPUs while leaving that path as it is can leave more expensive hardware idle during cold starts.

Closing the gap requires direct access to fast local storage tiers and prefetch strategies that account for the GPU memory hierarchy. Storage evaluations, including those considering Open Lake, therefore need to examine locality and host-memory staging alongside advertised storage bandwidth. The criterion is useful data delivered to GPU memory under the deployment's actual loading conditions.

KV cache offloading adds a second class of NFS I/O that has different requirements than model loading

Model weights are not the only storage object that matters during inference. Long-context workloads create intermediate attention state that is useful to reuse but too large to keep entirely in GPU memory. Offloading that KV cache introduces CPU DRAM, NVMe, or remote storage into serving. When the offload path uses NFS, storage performance becomes an ongoing request-time concern as well as a startup concern.

KV cache I/O is fine-grained, bursty, and latency-sensitive, unlike the large sequential reads of model loading. Consequently, settings that improve weight-loading throughput may hurt cache access latency. AMD's LIFE framework characterizes how time per output token grows as the KV cache grows over a request's lifetime. External cache transfers can add another burden when staging buffers are overprovisioned. A request with a few gigabytes of useful KV data can cause a much larger transfer because of minimum buffer allocation sizes.

These patterns compete when they share an NFS backend. A cold start during heavy serving consumes bandwidth precisely when cache latency affects active requests. The model-load benchmark alone therefore understates aggregate storage demands, which include both the time needed to bring a replica online and the interference that loading creates for replicas already serving.

The tuning interventions that address NFS bottlenecks without changing the storage architecture

Mount-level and server-side tuning are the first steps when an NFS deployment uses less capacity than it has. Large rsize and wsize values, at or near the NFS maximum, align transfer units with large-block access and reduce per-request overhead. For weight reads, rsize is the directly relevant setting. The nconnect mount option opens multiple parallel TCP connections from one client to the server, reducing per-connection serialization. IBM has stated it will add nconnect support to IBM Storage Scale for this purpose.

Readahead tuning lets the kernel fetch data before the application's page faults ask for it. This is particularly effective when weights are read sequentially from the beginning of a file, because speculative reads can turn repeated waits into streaming throughput. Server-side nfsd thread tuning addresses a separate issue, allowing the server to handle concurrent client requests more effectively. Async write settings and NFS export options change how writes are handled, but a missing model page still needs a synchronous read. These controls should be matched to the operation being measured.

Their limits remain architectural. More connections and threads still share the server's aggregate bandwidth, and larger transfers still face network round-trip latency. With all these settings, the CPU-staged copy path stays in place. Tuning is therefore a way to recover unused capacity, not a way to turn a single-server filesystem into a distributed one.

NFS over RDMA as a protocol upgrade that removes the TCP latency floor

NFS over RDMA removes the software-mediated TCP network stack from the data path. This can reduce per-operation latency by roughly two orders of magnitude while sharply reducing CPU load on client and server. The gain extends beyond faster individual operations. A storage CPU relieved of TCP processing has more capacity to handle concurrent NFS requests. That matters when CPU saturation, rather than disk bandwidth, is serializing loads.

RDMA can also address copy amplification when combined with GPUDirect Storage, or GDS. This combination eliminates the CPU bounce buffer between storage or network and GPU memory. Model load times can fall by a large factor compared with a tuned TCP NFS baseline.

With RDMA, the server still has the same bandwidth to share. The NIC's line rate and the server's internal bandwidth become the ceiling once TCP processing is no longer dominant. That capacity can be sufficient for many inference replica configurations without being sufficient for simultaneous cold starts of the largest models across a large GPU cluster. RDMA is the appropriate upgrade for transport and staging constraints, but not a complete answer to aggregate bandwidth exhaustion.

What storage architecture reduces LLM inference latency when NFS reaches its limit

Distributed storage is needed when replicas loading at once need more bandwidth than one NFS server can supply. Parallel filesystems stripe reads across storage nodes and NVMe devices, so the total bandwidth can grow as more nodes are added. In CoreWeave's distributed file storage benchmark on a 64-node H200 cluster, aggregate read throughput was above roughly 500 GiB/s, while each node stayed close to the target of roughly 8 GiB/s per node.

DeepSeek's Fire-Flyer File System (3FS) integrates NVMe SSD storage with RDMA-capable high-speed networks for large-scale AI training and inference. NVMe-oF removes the NFS protocol stack entirely and achieves storage latencies in the tens of microseconds with RDMA transports. For infrequent, large sequential model loads, its amortized latency advantage over well-tuned NFS over RDMA narrows. Removing NFS metadata overhead and copy amplification can still matter.

A hybrid architecture often fits the two inference I/O patterns more closely. NFS or NFS over RDMA can handle infrequent full-model loads, local NVMe can hold hot models and offloaded KV cache, and a distributed store can provide the cold tier. Open Lake is another option to evaluate in this landscape, with its checkpoint-loading benchmarks providing a basis for assessing the storage-to-GPU path. The cost-per-token rationale for tiering is less GPU idle time during reloads and less interference during serving. The value of tiering with NVMe depends on those gains under the actual workload.

How to decide which intervention fits a given deployment's actual bottleneck

Begin with four measurements, namely per-client throughput during simultaneous loads, NFS server CPU utilization, page-fault rate during initialization, and KV cache I/O latency during serving. Together, they distinguish concurrency collapse, TCP processing saturation, cold-cache demand paging, and contention. An isolated load test cannot make those distinctions. The benchmark must run under the conditions that cause the problem in production.

If per-client throughput is adequate in isolation but falls with replica count, start with nconnect and server thread tuning. If aggregate server bandwidth is exhausted, move to distributed storage. High page-fault rates during sequential initialization point toward readahead and hot-model caching. If TCP round-trips remain the constraint even for a single client with a warm cache, use RDMA. If aggregate throughput remains low after that upgrade because server-side disk or memory bandwidth is binding, the next step is distributed or NVMe-native storage.

If KV cache I/O contends with model loading, separate their bandwidth budgets by placing KV cache in local NVMe or CPU DRAM and weights on NFS or a distributed store. This avoids solving a serving-latency problem solely by accelerating bulk reads. SageMaker's inference recommendations tooling illustrates the same benchmark-first discipline through automated testing across instance types and configurations. Storage architecture deserves the systematic benchmarking rigor already applied to model quantization and serving framework selection.