Unified Memory and Managed Memory Pitfalls in CUDA Inference
Unified Memory's lazy page migration breaks decode kernels starved for bandwidth.
Amara Osei-Bonsu
Section
GPU Memory
7 stories in GPU Memory.
NVLink vs. PCIe Bandwidth Impact on KV Cache Transfers
NVLink's massive bandwidth advantage over PCIe makes peer GPU memory viable as a KV cache tier.
Priya Subramaniam
Data Residency Strategies for Multi-Tenant Inference
Prefix caching in multi-tenant LLM inference leaks prompts through timing side channels.
Nikolai Vereshchagin
Multi-Instance GPU Partitioning for Inference Density
KV cache memory pressure, not compute limits, makes GPU partitioning worthwhile for inference.
Priya Subramaniam
Weight Loading Latency and Cold-Start Optimization
Separate bottlenecks in each cold-start phase demand targeted fixes, not faster hardware.
Nikolai Vereshchagin
HBM Bandwidth Saturation in Autoregressive Decoding
Why autoregressive decoding starves GPU compute cores on bandwidth-limited hardware.
Priya Subramaniam
GPU Memory Hierarchy for LLM Practitioners
Memory bandwidth, not compute power, is what actually limits how fast LLMs generate tokens.
Amara Osei-Bonsu