Large language models must store a growing memory of past tokens to generate text, but this memory can outsize the model itself, limiting how much context can fit on a device. Current methods to shrink this memory fail at high compression ratios needed for long contexts. The paper introduces Transformer-FSQ, which replaces standard memory components with compact integer codes, drastically reducing memory use while maintaining high retrieval accuracy and decoding speed.
Language models preserve more retrieval accuracy with 97-fold smaller caches
Across three Qwen3 model sizes, the authors’ compression method outperformed the evaluated alternatives on long-context retrieval at compression ratios from 12-fold to 97.5-fold.
Top University
Andras Szecsenyi · Thomas L. Lee · Vaisakh Shaj · Elliot J. Crowley · Amos Storkey
The University of Edinburgh
Research Digest··2 min read
In one line
Extreme KV-cache compression is practical using finite scalar quantization, achieving accurate retrieval and fast decoding.
Before you read this
Two ideas this paper builds onkv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.73 papers →quantization-aware trainingTraining a neural network with simulated quantization in the forward pass so its weights and activations adapt to low-precision formats, improving deployment accuracy.23 papers →
Read it as
Szecsenyi and colleagues show that language models can sharply reduce the memory used to retain context while preserving more retrieval accuracy than competing compression methods.
A language model’s key-value cache stores information about previously processed tokens, allowing it to generate text without recomputing that information.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.