KV Cache Offload to NVMe Under Memory Pressure
Offloading KV cache to NVMe requires bypassing the CPU bottleneck entirely.
Freya Halvorsen
Editor at Large
Freya Halvorsen
Freya Halvorsen is a editor at large at Inference Engineering covering kv cache systems. Based in Amsterdam, Freya has written for Inference Engineering since 2016.
1 story · Amsterdam