The authors define a "neuron feature" as a transcoder feature whose encoder and decoder weights strongly align with one MLP neuron and whose activation pattern tracks that neuron.
Many learned transcoder features closely reproduce individual transformer neurons
Across 13 open-source transcoders, features aligned with single MLP neurons supplied much of the stable structure, especially in later layers.
Big Tech
Stepan Shabalin · Mateusz Piotrowski
Anthropic Fellows · Georgia Institute of Technology · Anthropic
Research Digest··3 min read
Thread:Linear Probe Validity
Shabalin and Piotrowski test whether transcoders, sparse learned approximations of transformer MLP layers, discover useful structure beyond the original neurons.
§