Continuous Batching Mechanics and Head-of-Line Blocking
Continuous batching solves one head-of-line problem but leaves two deeper mechanisms intact.
Priya Subramaniam
Senior Staff Writer
Priya Subramaniam
Priya Subramaniam has spent over a decade covering systems software and hardware co-design, previously embedded with semiconductor research teams in Austin and Bengaluru before focusing exclusively on AI infrastructure. Her reporting zeroes in on how memory hierarchies and storage pipelines shape real-world inference latency at scale.
5 stories
Distributed KV Cache Across Inference Nodes
Splitting inference across nodes requires moving KV cache efficiently between machines.
Priya Subramaniam
KV Cache Offload to NVMe Under Memory Pressure
Offloading KV cache to NVMe requires bypassing the CPU bottleneck entirely.
Priya Subramaniam
Prefix Caching and Prompt Deduplication in vLLM
How vLLM reuses cached token blocks to slash redundant GPU computation.
Priya Subramaniam
KV Cache Memory Layout on Multi-GPU Hosts
Attention architecture determines KV cache size; wrong block sizing wastes bandwidth on every token.
Priya Subramaniam