RoCEv2 Congestion Control for AI Storage Traffic

RoCEv2 congestion control for AI storage traffic needs tuning to work well in practice. RDMA over routable Ethernet provides direct memory transfers that avoid operating system overheads, while NVMe-oF enables low-latency access to remote storage. Keeping that fabric from becoming the bottleneck requires storage-specific ECN thresholds, sufficient PFC headroom, and consistent DCQCN parameters, with rate reduction preceding pause.

Why RoCEv2 congestion control is an operational problem, not a protocol problem

Production RoCEv2 deployments depend on configuration discipline. RoCEv2 is the default inter-node fabric for GPU collectives and AI storage access in distributed AI clusters, encapsulating RDMA in UDP/IP on destination port 4791. Its appeal is RDMA performance over standard routable Ethernet. Ethernet, however, is lossy by default, and packet loss sharply degrades RDMA performance. Congestion control therefore belongs in the fabric’s core operating requirements.

InfiniBand offers an alternative to this Ethernet operating model. RoCEv2 is still the workable choice for clusters that share infrastructure, need open hardware, or must grow beyond proprietary fabrics. With that choice, operators must make switch marking, receiver feedback, sender rate control, and link-level pause work together. Operators must check that those mechanisms respond in the right order under the actual workload, since enabling RoCEv2 alone does not make the fabric lossless.

How the ECN → DCQCN → PFC control loop works end to end

The loop begins with packets leaving the sender NIC. When a switch queue passes its ECN threshold, the switch marks packets with the Congestion Experienced bit and forwards them. The receiver NIC detects the mark and returns a Congestion Notification Packet, or CNP, to the sender. Feedback is rate-limited to roughly one CNP every 50 µs per flow. Traffic continues while this exchange takes place.

The sender NIC uses DCQCN to reduce its sending rate and recover gradually as congestion clears. Its multiplicative-decrease and additive-increase cycle responds to signals shaped by the switch’s WRED marking curve. Below K-min, packets are not marked. Between K-min and K-max, the chance that a packet is marked rises linearly toward P-max, and above K-max all packets are marked. As the signal grows, rate control can respond to queue buildup without stopping traffic at every burst.

PFC intervenes when rate control cannot contain the queue. At the PFC Xoff threshold, the switch sends a PAUSE frame upstream for the affected priority, halting that class until Xon resumes transmission. Unlike end-to-end DCQCN, PFC acts hop by hop and can introduce jitter when queued traffic resumes in a burst. The healthy sequence is ECN marking, DCQCN rate reduction, and PFC only as a last resort. The space between marking and pause gives the feedback loop time to act.

What misconfigurations actually look like in production AI fabrics

Setting low ECN thresholds but an even lower PFC Xoff threshold causes a pause before marking can trigger sender backoff. Traffic accumulates at the sender NIC, then arrives in a burst when Xon releases it, triggering another pause. Microsoft Azure AI Foundry’s analysis documents this pattern as pause storms, throughput collapse, oscillations, and “ghost” latency spikes.

A high K-min produces the mirror failure. If PFC intervenes before queues reach the marking threshold, ECN remains silent and pause becomes the primary congestion mechanism. At the other extreme, K-min too close to zero causes rate cuts during normal traffic variation, suppressing throughput without real congestion. Both failures can leave operators troubleshooting apparent capacity or hardware problems when the thresholds are responsible.

With incorrect CoS mapping, unrelated flows can end up in the same paused class and become victim flows even though they did not cause the congestion. With PFC mapped to the wrong priority and ECN disabled at the top-of-rack switch, load can produce packet loss and RDMA transport collapse. Collective communication times can then jump from milliseconds to seconds.

Why AI storage traffic creates a distinct congestion signature that training collectives do not

GPU collectives and storage I/O stress buffers differently. AllReduce, AllGather, and AllToAll produce large, synchronized elephant flows, with ECMP hash collisions creating link imbalance when flows converge on the same paths. Training-data reads also need a lot of bandwidth, but large sequential reads can cope with some queuing. Checkpoint writes are periodic bursts that can overwhelm shared buffers when nodes write simultaneously. KV cache transfers add small-to-medium, latency-sensitive messages, and delays to those messages add directly to time-to-first-token.

These differences become costly when traffic shares a priority. A storage burst can pause a GPU collective mid-step, wasting compute time even though the collective did not create the congestion. High pause counts for the storage priority can also show pressure on storage I/O while the training plane looks healthy. A training-only view therefore misses part of the bottleneck between GPUs and their data.

The structural response is deliberate traffic separation. Storage and training traffic should use different QoS priorities or entirely different fabrics. Within a shared fabric, storage needs ECN and PFC thresholds matched to its message-size distribution and burst timing. Using the collective configuration overlooks how a bandwidth-bound read differs from a synchronized checkpoint or a latency-sensitive cache transfer.

NVMe-oF over RoCEv2 and the unified congestion-management problem it creates

NVMe-oF makes this shared-fabric problem concrete for disaggregated storage. It lets hosts access remote NVMe queues directly over the fabric, bypassing legacy SCSI stacks while keeping latency near local-NVMe levels. RoCEv2 is the performance-preferred RDMA transport for this pattern in AI environments. Central management and scale-out capacity do not remove its lossless-network requirements. Storage inherits the same ECN, PFC, and DCQCN machinery as compute traffic.

There is no separate storage congestion-management layer to compensate for unsuitable fabric settings. Training-data prefetch overlaps forward and backward passes, checkpoint writes arrive at step boundaries when collectives are also peaking, and KV cache evictions can generate sustained mid-sized writes during prefill compute. One shared fabric must serve traffic with different marking onset needs, burst profiles, and tolerance for pause duration. Tuning only for collectives leaves those storage requirements unresolved.

KV cache disaggregation and what it demands from the RoCEv2 fabric

KV cache disaggregation makes fabric latency directly user-facing. Long-context LLM serving is constrained by cache capacity and bandwidth, so disaggregated architectures pool CPU DRAM, SSD, and RDMA resources beyond GPU HBM. Mooncake, Moonshot AI’s serving platform for Kimi, implements this through Mooncake Store, a cluster-wide KV cache across CPU DRAM and SSDs. Its capacity expansion remains bounded by RDMA bandwidth per NIC. Moving the cache out of GPU memory does not remove the transfer constraint.

Prefill-decode separation also changes the cost of congestion. Its small-to-medium KV transfers are highly latency-sensitive, and a single PFC pause adds measurable request latency. The dominant tuning lever for these flows is ECMP hashing that places prefill-to-decode traffic on uncongested paths, with adaptive routing where supported. The KV priority should mark earlier and more aggressively than bulk training-data traffic because a pause can stall decode, while a paused checkpoint write merely slows. Quantizing KV tensors from BF16 or FP16 to FP8, INT4, or lower reduces the bytes crossing the fabric and the load on shared buffers.

Correct ECN, PFC headroom, and DCQCN parameter settings for AI storage traffic

Practitioners should first establish meaningful separation between ECN marking and PFC pause. K-min should sit well below Xoff, and K-max should leave a sufficient gap before the PFC threshold for the feedback loop to act. For latency-sensitive classes, including KV transfers and NVMe-oF command queues, K-min should be lower than for bulk training-data classes without triggering backoff during ordinary variation. These relative settings require deployment-specific byte counts.

PFC headroom must then accommodate packets still in flight after a pause is asserted. The calculation includes cable propagation delay, switch pipeline latency, and NIC processing delay, all at the port’s line rate. Higher speeds require more headroom in absolute bytes. Hardware upgrades therefore require recalculation even when cable lengths stay the same. Underprovisioning allows packets to arrive after pause assertion and overflow the buffer, defeating the lossless design.

If the sender ramps up too quickly between CNPs, it overshoots and rebuilds the queue before the next notification arrives. Recovery cannot be tuned independently of that delay. P-max should make congestion signals clear without allowing a momentary burst to cause prolonged rate suppression. NIC parameters must also be consistent across nodes, since mismatched settings can produce asymmetric rate cuts and oscillation.

Priority separation is the prerequisite for applying these settings to distinct workloads. NVMe-oF, KV cache, and checkpoint I/O should occupy a storage QoS priority separate from GPU collectives, with independent ECN and PFC thresholds. One shared RDMA priority defeats that distinction. Storage bursts can then pause collective traffic even if the shared thresholds were chosen with care.

Observability requirements for verifying the congestion loop is working as tuned

Throughput alone cannot show that congestion control is working well. PFC pause counts for each priority and port show whether pause protects buffers now and then or the fabric relies on it much of the time. Low, infrequent counts indicate ECN is acting before PFC, while sustained or rising counts point toward threshold misalignment or insufficient headroom. Absent marks under congestion suggest K-min is too high, while continuous marking under light load suggests it is too low.

CNP rates at the receiver show whether marked packets reach the notification path. Queue-occupancy histograms then reveal whether buffers remain well below Xoff or repeatedly approach it. Both are needed to interpret pause counts. Streaming telemetry over gNMI with OpenConfig models uses standards to show what is happening in each queue and priority across the fabric. Storage and compute must have separate views because a dashboard that combines them can hide victim-flow events and pressure specific to storage.

As workloads change, thresholds that work for one mix of reads, checkpoints, and cache transfers may not work for another. The practical heuristic is specific to the affected queue. If storage-priority PFC counts rise while ECN marking stays flat or absent, operators should widen the marking-to-pause gap by lowering K-min while keeping PFC Xoff unchanged.

Where the current RoCEv2 tuning model breaks down at scale and what comes next

The DCQCN/PFC model works well at production scale today, but some limits of its design remain even after tuning. PFC halts an entire priority hop by hop, allowing one congested queue to create a pause tree that reaches unrelated nodes. DriveNets’ analysis describes congestion trees that can cascade across large parts of a network and, in extreme cases, produce deadlocks requiring manual recovery. Higher port speeds further narrow the response window. More bytes arrive during the same delay, leaving less time between ECN onset and a necessary pause.

Meta’s SIGCOMM 2024 paper describes co-designing the collective library and RoCE transport for receiver-driven traffic admission, moving away from DCQCN at that layer to improve responsiveness to transient bursts. Barre targets fairness, congestion responsiveness, and scalability with minimal overhead at high line rates. Its USENIX ATC 2025 paper reports deployment for over a year in a RoCE cluster supporting up to 10,000 GPUs for large-scale LLM training.

Ultra Ethernet 1.0 introduces a different transport direction. It uses UDP/IP with packet spraying and selective repeat, distributing packets across paths independently of flow-level ECMP hashing. Its transport-specific headers on UDP port 4793 are incompatible with RoCEv2’s IB-derived headers on port 4791. It aims to reduce PFC dependence and retransmission waste while retaining congestion control. Receiver-driven variants and UEC are relevant to the largest new builds and 2026-and-beyond planning, while correctly tuned and observed ECN, DCQCN, and PFC remain the actionable model for production AI storage fabrics today.