Weight pruning can invert a transformer circuit while preserving perplexity

In GPT-2, magnitude neuron pruning at 30% sparsity flips the sign of Name Mover contributions, causing the model to systematically predict the wrong name despite intact circuit components and low perplexity.

Academic

Nabeelah Faumi · Deshan Sumanathilaka

Informatics Institute of Technology

Research Digest·28 October 2026·2 min read

Illustration: Weight pruning can invert a transformer circuit while preserving perplexity

In one line

Weight pruning can zero or invert transformer circuits, causing systematic errors that accuracy and perplexity miss.

Before you read this

An idea this paper builds on

mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.94 papers →

Read it as

When we compress language models by pruning away parts, we usually check only if output performance stays the same. But a model can give correct answers for wrong reasons, hiding internal damage.

This study pruned GPT-2 by removing either attention heads or MLP neurons (two types of internal components), choosing which to remove by magnitude (smallest weights) or randomly. They used a known circuit, the chain of components for identifying an indirect object, to check internal integrity.

They found two failure modes. Removing attention heads by magnitude destroyed the circuit, leading to random guessing. Removing MLP neurons by magnitude inverted the circuit: the model confidently chose the wrong name (17.7% accuracy) while perplexity rated it less damaged than random pruning (74.6). Standard metrics miss this mechanistic damage.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

mechanistic interpretability pruning circuit analysis large language model activation patching failure mode analysis