Controlling KV Cache Memory Growth Without Sacrificing Throughput
A four-layer framework combines paging, quantization, eviction.
Priya Subramaniam
Inference Engineering
KV Cache Systems
KV Cache Offloading Stack Decisions From GPU DRAM to Object Storage
How to move LLM cache between GPU, DRAM, and storage.
Amara Osei-Bonsu
· 9 min read
KV Cache Transfer Over RDMA Between Prefill and Decode Nodes
Push mode RDMA writes and pull mode reads create a 98x throughput gap on identical hardware.
Tobias Wrenfeld
The Feed
RoCEv2 Congestion Control for AI Storage Traffic
Configuring ECN, DCQCN, and PFC settings correctly is essential for preventing fabric bottlenecks.
Nikolai Vereshchagin
How RDMA Bypasses the CPU to Sustain All-Reduce Throughput in GPU Cluster Training
Tobias Wrenfeld
RDMA Fundamentals for LLM Infrastructure Engineers
Priya Subramaniam
KV Cache Tiering Across DRAM NVMe and Remote Storage for Inference Latency
Amara Osei-Bonsu
KV Cache Tiering and Eviction Policy Design for 128k Context Windows
Tobias Wrenfeld
KV Cache Lifecycle Orchestration Across Request Queues and Preemption Boundaries
Managing KV cache as a shared resource across request lifecycles improves LLM serving efficiency.
Amara Osei-Bonsu
Object Storage Throughput and Prefix Sharding for Training Pipelines
Priya Subramaniam
SSD Endurance and Wear Management in KV Cache Offload Deployments
Nikolai Vereshchagin
GDS and NVMe-oF for Direct Storage-to-GPU Data Paths
Priya Subramaniam