They studied production traces comprising over 20 billion processed tokens from two companies.
Simple LRU rivals sophisticated eviction for LLM prefix caching
Analysis of over 20 billion tokens from production traces shows recency is unusually predictive due to steady session pacing.
Top University
Yiyu Liu · Minlan Yu · Juncheng Yang
Harvard University
Research Digest··2 min read
The authors analyzed production traces from two companies and evaluated 14 eviction algorithms for LLM prefix caching.
Why this paper
From Harvard University · Part of Context Engineering for Agents, now 41 papers
In one line
Prefix cache management should prioritize recency and selectively add quick demotion, compute-aware eviction, and capacity-dependent granularity.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§