The paper investigates whether compressed KV cache representations (CKVs) degrade an LLM's ability to handle off-context queries—such as general knowledge, instruction following, and avoiding context contamination. The authors find that the popular Cartridges compression method performs well on document-related (on-context) queries but harms off-context capabilities. They propose Cartridges++, two simple modifications: data-mixing (adding a small fraction of off-context question-answer pairs during training) and routing (deciding at inference time whether to use the compressed memory). Evaluations show Cartridges++ preserves off-context abilities with little to no cost to on-context fidelity. This is interesting because it highlights that evaluating compression only on document accuracy can hide broader model degradation, and that benign fixes can restore those capabilities.
Compressed LLM memories can preserve skills beyond their source documents
The authors show that learned KV-cache compression can impair unrelated queries, then mitigate the problem through data mixing or relevance-based routing.
Big Tech
Sonia Laguna · Joao Monteiro · Marco Cuturi · Pierre Ablin · Eleonora Gualdoni
Apple
Research Digest··2 min read
In one line
Cartridges++ retains off-context abilities in compressed KV caches with negligible cost.
Before you read this
Three ideas this paper builds onkv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.82 papers →model routingDirecting each query or subtask to the most suitable model to optimize cost, accuracy, or quality.89 papers →context compactionCompressing or summarizing an LLM's past context into a smaller representation to extend memory and improve reasoning within limited token budgets.81 papers →
Read it as
Laguna and colleagues test whether compressed key-value caches remain safe to reuse when a user asks questions unrelated to the cached document.
The authors study compressed key-value caches, compact representations of a document's transformer states that can be computed once and reused for many queries.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.