Large language models (LLMs) generate text with a key-value (KV) cache: a growing store of earlier context. Keeping every entry makes them slower and memory-hungry, so engineers evict entries, but an entry useful for one query may still hold facts needed later. Naive scoring confuses current relevance with safety to delete.
The authors train a gate, a small learned scorer, to rate each cache entry, and place it inside the softmax, the function that turns scores into attention shares, so removing an entry transfers its share to the rest rather than wasting it. They found that training on the LLM's usual next-word objective fails, because the model prefers keeping everything, so they deny the gate access to full attention, make kept entries compete for a fixed share, and teach it to copy the full model's predictions. They also insert centroid tokens, averaged summaries of evicted entries, back into the same softmax.
At heavy compression, the gated model scored 90.14 on one long-context benchmark and 46.03 on another, close to full-cache accuracy, meaning large memory savings with little quality loss.