The authors develop KV-Surgeon, which uses calibration queries to greedily remove cache entries while updating the remaining keys and values to minimize output error.
KV-Surgeon optimizes LLM cache compaction by jointly selecting and compensating for deletions
The method uses a quadratic surrogate to account for interactions between removed cache entries, which improves accuracy at high compaction ratios.
Academic
Moritz Wagner · Christophe Roux · Max Zimmer · Sebastian Pokutta
Zuse Institute Berlin · Technische Universität Berlin
Research Digest··2 min read
The authors introduce KV-Surgeon, a compensation-aware method for offline KV-cache compaction that adjusts remaining entries after each deletion.
§