Larger language models can lose more retrieval ability after compression

The authors find that larger models compensate less for comparable damage to retrieval components, allowing more of that damage to reach the output.

Big Tech

Gautam Gupta · Dikshant Kukreja · Unique Kumar · Avni Mittal · Deepanshu Dabas · Kshitij Sah · +4 more

Indraprastha Institute of Information Technology, Delhi · TurboML · Microsoft · Nanyang Technological University · Indian Institute of Technology, Kanpur

Research Digest·6 November 2026·2 min read

Illustration: Larger language models can lose more retrieval ability after compression

In one line

Larger language models are more vulnerable to compression because their internal compensation for damaged circuits weakens as scale increases.

Before you read this

Three ideas this paper builds on

large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.279 papers →mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.118 papers →post-training quantizationQuantizing pre-trained neural network weights and activations to47 papers →

Read it as

are compressed to reduce cost, but standard testing only checks output scores. This hides internal damage because other parts may compensate.

The authors study in-context retrieval, a task that uses a small set of attention heads to pick a name from context. By damaging those heads via compression and measuring the network's response, they could see how much compensation occurs.

Smaller models restore most lost function, making compression seem harmless. Larger models compensate less, so the same compression causes more harm. This means compression that works on small models may break larger ones, and overall benchmarks can be misleading.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

large language model mechanistic interpretability in-context learning post-training quantization scaling law pruning