Inference Cache Optimization
Methods for optimizing cache transfer and memory in LLM inference clusters to reduce latency and improve throughput.
5 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
How this thread developed
Nov 2026 · Georgia Institute of Technology
Avoiding routing collisions cuts cache-transfer delays in AI inference clusters
Introduces a distributed path reservation mechanism to avoid routing collisions for KV cache transfers in inference clusters.
1 further paper
Nov 2026 · Colby College
Mid-depth layers judge KV-cache evictions better than the owning layer
Reveals that mid-depth layers are better than the owning layer for scoring KV-cache evictions, improving cache management decisions for inference throughput.
1 further paper
Nov 2026 · Thompson Rivers University
Chunked prefill, not cache architecture, drives LLM serving engine speed differences
Compares vLLM and SGLang, showing chunked prefill drives speed differences, which informs cache optimization.
3 of 5 papers shown