Large language models (LLMs) keep a key-value cache, a short-term memory of earlier text, to answer questions about long documents; eviction shrinks it by scoring each stored token and keeping only the highest scores. Today each layer of the model scores its own cache. Tests across many model sizes showed scoring from different depths. Shallow scorers and the final layer did worse than middle layers, because a layer only judges which tokens matter after it has used the question to locate the answer. Choosing a mid-depth scorer improved retrieval accuracy by up to 43 percentage points in the best tested setting. The practical lesson is that cache eviction should not default to the layer being pruned.
Mid-depth layers judge KV-cache evictions better than the owning layer
Across 18 models from 0.5B to 109B, scoring eviction by a mid-depth layer improves long-context retrieval by up to 56 percentage points over shallow or final layers
Academic
Ekaterina Seregina
Colby College
Research Digest··2 min read
In one line
Mid-depth layers score KV-cache eviction better than each layer scoring its own cache, improving long-context accuracy at small and mid-sized models.
Before you read this
Two ideas this paper builds onkv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.58 papers →role decouplingSeparating distinct functions such as planning, reasoning, and execution into separate components improves efficiency, quality, and reliability.43 papers →
Read it as
The authors show that the layer used to score tokens for KV-cache eviction critically determines accuracy.
The authors investigated a previously unnoticed hyperparameter in KV-cache eviction: which layer's attention scores are used to rank tokens for eviction.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.