Inference on Open-Source Models vs. Proprietary APIs at Scale
Open-source models beat APIs on cost only above tens of millions monthly tokens.
Amara Osei-Bonsu
Contributing Editor
Amara Osei-Bonsu
Amara Osei-Bonsu began her career benchmarking distributed database systems at a financial data firm in London before the generative AI wave pulled her toward inference performance analysis. She authors the site's recurring benchmark methodology guides and holds the team accountable to reproducible test conditions.
4 stories
KV Cache Sizing for Concurrent Request Budgets
How to predict GPU memory limits before your inference server crashes from hidden cache overhead.
Amara Osei-Bonsu
Paged Attention and KV Cache Fragmentation
How virtual memory fixes GPU memory waste in large language model serving.
Amara Osei-Bonsu
LRU vs GDSF vs Recency-Weighted Eviction for Long-Context KV Cache
Three eviction policies battle for control of shrinking KV cache memory.
Amara Osei-Bonsu