KV Cache Offloading Stack Decisions From GPU DRAM to Object Storage
How to move LLM cache between GPU, DRAM, and storage.
Amara Osei-Bonsu
Section
KV Cache Systems
10 stories in KV Cache Systems.
Controlling KV Cache Memory Growth Without Sacrificing Throughput
A four-layer framework combines paging, quantization, eviction.
Priya Subramaniam
KV Cache Lifecycle Orchestration Across Request Queues and Preemption Boundaries
Managing KV cache as a shared resource across request lifecycles improves LLM serving efficiency.
Amara Osei-Bonsu
KV Cache Sizing for Concurrent Request Budgets
How to predict GPU memory limits before your inference server crashes from hidden cache overhead.
Amara Osei-Bonsu
Distributed KV Cache Across Inference Nodes
Splitting inference across nodes requires moving KV cache efficiently between machines.
Priya Subramaniam
KV Cache Offload to NVMe Under Memory Pressure
Offloading KV cache to NVMe requires bypassing the CPU bottleneck entirely.
Priya Subramaniam
Prefix Caching and Prompt Deduplication in vLLM
How vLLM reuses cached token blocks to slash GPU compute waste across requests.
Priya Subramaniam
Paged Attention and KV Cache Fragmentation
How a virtual memory trick cuts KV cache waste from 80% to 4%.
Amara Osei-Bonsu
KV Cache Memory Layout on Multi-GPU Hosts
Memory layout decisions determine whether multi-GPU serving saturates compute or bandwidth.
Priya Subramaniam
LRU vs GDSF vs Recency-Weighted Eviction for Long-Context KV Cache
LRU fails on long contexts, but GDSF and attention-aware policies offer concrete improvements.
Amara Osei-Bonsu