Language models can cut cache memory nearly 27-fold, retaining most performance

Delgado and colleagues report that predicting cached values from keys, combined with other compression methods, retains 94.5% of baseline performance on long contexts.

Chinese Tech

Teresa Delgado · Hugo Hazard · Martin Benfeghoul · Jun Wang · Zafeirios Fountas

Huawei, Noah’s Ark Lab · University College London

Research Digest·6 November 2026·2 min read

Illustration: Language models can cut cache memory nearly 27-fold, retaining most performance

In one line

LowRAM compresses the KV cache by predicting values from keys and using low-rank keys, achieving 26.9x compression with 94.5% baseline performance.

Before you read this

Two ideas this paper builds on

kv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.67 papers →test-time adaptationAdapting a deployed model during inference using new data or interactions, without a separate training phase.79 papers →

Read it as

Large language models generate text one word at a time. To speed up generation, they store keys and values from previous words in a cache. But this cache grows with text length and can limit how many requests fit in memory.

The researchers noticed that values are often predictable from keys. LowRAM fits a small predictor for each attention head (a component that focuses on different input positions) at test time to reconstruct values from keys, storing corrections only for the hardest tokens. Keys are compressed using a low-rank technique (representing them with fewer numbers) shared across layers.

LowRAM stays close to baseline quality at four times value compression. Combined with other methods, it achieves 26.9 times total compression while keeping 94.5% of baseline performance on long texts, making long-context inference more efficient.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

kv cache compression value cache amortisation cross-layer low-rank key compression test-time adaptation meta-learning selective reconstruction