Scheduler Design for Long-Context and Short-Context Mixed Traffic
KV cache residency, not queue position, determines how to schedule mixed-length requests fairly.
Elias Sabatini
Senior Writer
Elias Sabatini
Elias Sabatini is a senior writer at Inference Engineering covering inference serving. Based in Melbourne, Elias has written for Inference Engineering since 2022.
1 story · Melbourne