prefix caching
- Papers
- 4
- Released code
- 0
- First seen
- Oct 2026
- Latest
- Nov 2026
4 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 1Simple LRU rivals sophisticated eviction for LLM prefix caching
Prefix cache management should prioritize recency and selectively add quick demotion, compute-aware eviction, and capacity-dependent granularity.
Oct 2026 - 2Chunked prefill, not cache architecture, drives LLM serving engine speed differences
Most of a large benchmarked gap between two LLM serving engines comes from one scheduler default, chunked prefill, not from cache design.
Nov 2026 - 3AI agent servers can stay overloaded after traffic bursts end
A transient burst can permanently destroy prefix-cache hit rates and throughput in LLM agent serving.
Nov 2026
How fast it is moving
Often studied with
agent memoryA repository where an agent stores past interactions and outcomes to inform future decisions.1 togethermodel routingDirecting each query or subtask to the most suitable model to optimize cost, accuracy, or quality.1 togethermulti-turn tool useThe capability of an agent to invoke external tools sequentially over multiple conversational turns to complete complex tasks.1 togetherThe papers
Most central to this idea first, not most recent.
- Independentcs.DC
AI agent servers can stay overloaded after traffic bursts end
A transient burst can permanently destroy prefix-cache hit rates and throughput in LLM agent serving.
Nov 2026
- Academiccs.DC
Chunked prefill, not cache architecture, drives LLM serving engine speed differences
Most of a large benchmarked gap between two LLM serving engines comes from one scheduler default, chunked prefill, not from cache design.
Thompson Rivers University · Nov 2026
- Top Universitycs.DC
Simple LRU rivals sophisticated eviction for LLM prefix caching
Prefix cache management should prioritize recency and selectively add quick demotion, compute-aware eviction, and capacity-dependent granularity.
Harvard University · Oct 2026
- Big Techcs.LG
Canopy learns when tree-based shortcuts fail in LLM inference
CANOPY adapts multi-fidelity tree bandits by learning where smoothness is valid, improving LLM inference tasks.
Amazon, Columbia University · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.