The paper studies prefix caching in LLM applications using production traces from two companies. It evaluates 14 eviction algorithms and finds that sophisticated policies provide little benefit over LRU because prefix reuse is dominated by the regular pacing of active sessions, making recency unusually predictive. The paper introduces a compute-savings ratio and offline oracles to quantify cache effects, and recommends retaining recency as a foundation while adding selective techniques: quick demotion for one-hit prefixes, compute-aware partial eviction, and capacity-dependent eviction granularity. These findings are interesting because they challenge the need for complex eviction policies and offer practical guidance for optimizing LLM serving.
Simple LRU rivals sophisticated eviction for LLM prefix caching
Analysis of over 20 billion tokens from production traces shows recency is unusually predictive due to steady session pacing.
Top University
Yiyu Liu · Minlan Yu · Juncheng Yang
Harvard University
Research Digest··2 min read
In one line
Prefix cache management should prioritize recency and selectively add quick demotion, compute-aware eviction, and capacity-dependent granularity.
Read it as
The authors analyzed production traces from two companies and evaluated 14 eviction algorithms for LLM prefix caching.
They studied production traces comprising over 20 billion processed tokens from two companies.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.