Language models can outdo text searches at revealing internal signals

The authors train a model to turn internal activation directions into text examples, outperforming corpus search across several interpretation tasks.

Big Tech

Celeste De Schamphelaere · Juan J. Vazquez · Ari Spiesberger · Tomáš Gavenčiak · Gavin Leech · Euan Ong · +1 more

MATS · Arb Research · Charles University · Anthropic

Research Digest·6 November 2026·2 min read

Thread:Activation Steering Vectors

Illustration: Language models can outdo text searches at revealing internal signals

In one line

Maximally-Activating Example Metamodels (MAEMs) generate text that strongly activates any given direction in a language model's residual stream.

Before you read this

Three ideas this paper builds on

mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.118 papers →reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.152 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.305 papers →

Read it as

Language models (LMs) process words as mathematical vectors inside them. To understand what a specific vector means, researchers look for text that makes that vector value high. But finding such text in existing documents is limited by what those documents contain.

The authors developed MAEMs, which are LMs that take a vector as input and generate new text designed to maximize that vector's value. They used reinforcement learning, a trial-and-error training method, to directly reward the MAEM for producing text that activates the target vector in the original model.

The MAEM outperformed searching through 10 million tokens of text on most languages and domains. It also activated 89% of held-out features from sparse autoencoders, which are tools that break down model internals. This provides a scalable, verifiable way to interpret model vectors without relying on human-written explanations.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

mechanistic interpretability activation inversion reinforcement learning with verifiable rewards sparse autoencoders verification activation steering