Language models can lose 90% of weights with cheaper pruning

The authors report better next-token predictions than a competing pruning method, while pruning a seven-billion-parameter model with 49% fewer GPU-hours at 90% weight removal.

Big Tech

Jean Lucien Randrianantenaina · Hung Nguyen · Nasib Ullah · Jinbin Zhang · Nilesh Gupta · Inderjit S. Dhillon · +1 more

University of Bath · Aalto University · University of Texas at Austin · Google

Research Digest·2 November 2026·2 min read

Illustration: Language models can lose 90% of weights with cheaper pruning

In one line

ETAPS prunes LLMs gradually and adapts sparsely, achieving lower perplexity than prior state-of-the-art at extreme sparsity with far less memory and compute.

Read it as

Large language models are expensive because they read billions of numbers from memory to generate each word. Weight pruning removes many numbers to reduce cost, but extreme pruning severely degrades accuracy.

ETAPS prunes gradually: it removes numbers in stages. After each stage, it fine-tunes only a small subset of numbers (sparse delta). This avoids the large memory needed by previous methods.

ETAPS reduces error by up to 42% compared to the prior best method at the same extreme pruning levels, while using far less memory and computing time. This makes high-quality, compact models more accessible.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

weight pruning gradual pruning sparse delta sparse adaptation gradient-weighted saliency post-pruning recovery