Massive activations make AI model interpretability tools unreliable

In several large language models, a single dominant activation direction hides reconstruction errors and makes top features inconsistent across training runs.

Top University

Nathan Delisle · Chenhao Tan

University of Chicago

Research Digest·12 November 2026·2 min read

Illustration: Massive activations make AI model interpretability tools unreliable

In one line

Dominant activation directions make sparse-autoencoder feature rankings unstable and reconstruction scores misleading; removing those directions before training improves both.

Before you read this

Three ideas this paper builds on

mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.131 papers →sparse autoencoderA neural network that learns a compact, interpretable representation of data by activating only a small subset of its neurons at once.9 papers →activation steeringActivation steering modifies a model's internal vector representations during inference to shift behavior, such as editing outputs, removing capabilities, or controlling trust.74 papers →

Read it as

A language model’s internal activity can be dominated by one direction, called the first principal component (PC1). That can make a reconstruction score look excellent even when much of the other activity is missed, and make the most important features change between training runs.

(SAEs) rebuild a model’s internal activity using a small set of learned features. The researchers trained them after removing PC1 from the input, then added that direction back to the reconstruction.

One SAE explained 99.96% of total variation while missing 33% of the variation beyond PC1. Removing PC1 before training made top-ranked features more consistent and improved reconstruction and model behavior, so evaluations better reflected what the SAE captured beyond the dominant direction.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

mechanistic interpretability sparse autoencoder activation steering