Scheduler Design for Long-Context and Short-Context Mixed Traffic
KV cache residency, not queue position, determines how to schedule mixed-length requests fairly.
Tobias Wrenfeld
Inference Engineering — Page 3
Inference Serving
Chunked Prefill and Its Effect on Time to First Token
Chunked prefill eliminates scheduling bottlenecks that hurt response time.
Tobias Wrenfeld
· 11 min read
Continuous Batching Mechanics and Head-of-Line Blocking
Continuous batching solves one head-of-line problem but leaves two deeper mechanisms intact.
Priya Subramaniam
The Feed
GPUDirect Storage Integration in Inference Pipelines
GDS excels at training but underperforms for inference's small random reads.
Tobias Wrenfeld
Inference on Open-Source Models vs. Proprietary APIs at Scale
Amara Osei-Bonsu
Tensor Parallelism and KV Cache Sharding Patterns
Tobias Wrenfeld
Speculative Decoding Storage and Memory Overhead
Tobias Wrenfeld
KV Cache Sizing for Concurrent Request Budgets
Amara Osei-Bonsu
Distributed KV Cache Across Inference Nodes
Splitting inference across nodes requires moving KV cache efficiently between machines.
Priya Subramaniam
KV Cache Offload to NVMe Under Memory Pressure
Priya Subramaniam
Prefix Caching and Prompt Deduplication in vLLM
Priya Subramaniam
Paged Attention and KV Cache Fragmentation
Amara Osei-Bonsu