Speculative Decoding Storage and Memory Overhead
Trading latency gains for hidden memory costs that scale with context length.
Lena Petrova
Section
1 story in Inference Serving.
Trading latency gains for hidden memory costs that scale with context length.