LRU vs GDSF vs Recency-Weighted Eviction for Long-Context KV Cache
LRU fails on long contexts, but GDSF and attention-aware policies offer concrete improvements.
Amara Osei-Bonsu
Inference Engineering — Page 4
KV Cache Systems
KV Cache Memory Layout on Multi-GPU Hosts
Memory layout decisions determine whether multi-GPU serving saturates compute or bandwidth.
Priya Subramaniam
· 12 min read
Prefill and Decode Disaggregation in LLM Serving
Splitting prefill and decode workloads cuts latency and improves GPU utilization.
Nikolai Vereshchagin