Sparse autoencoders (SAEs) are used to interpret and steer language models, but their token-level reconstruction objectives often capture lexical and formatting details rather than true semantics. The authors propose chunk-level SAEs that operate on mean-pooled activations over contiguous token spans, with three variants: Mean-Chunk (reconstructs the observed chunk), Cross-Chunk (predicts a neighboring chunk's mean), and Joint-Chunk (combines both targets). These methods improve the discovery of high-level semantic features and show gains in downstream tasks like document retrieval, reasoning detection, classification transfer, and steering, with complementary strengths across variants.
Chunk-level sparse autoencoders discover more reliable semantic features than token-level counterparts
By encoding mean-pooled activations over chunks and using reconstruction and neighbor prediction, the method improves high-level feature discovery, reasoning detection, and steering.
Chinese Tech
Xu Wang · Yifan Yang · TingHao YU · Difan Zou
The University of Hong Kong · Tencent
Research Digest··2 min read
In one line
Chunk-level sparse autoencoders trained on mean-pooled activations learn more reliable semantic features than token-level SAEs.
Before you read this
Two ideas this paper builds onmechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.143 papers →activation steeringActivation steering modifies a model's internal vector representations during inference to shift behavior, such as editing outputs, removing capabilities, or controlling trust.83 papers →
Read it as
The authors introduce chunk-level sparse autoencoders (SAEs) that encode mean-pooled activations over contiguous token chunks instead of individual tokens.
Wang et al. propose three variants of chunk-level SAEs: Mean-Chunk reconstructs the chunk's mean activation, Cross-Chunk predicts the mean activation of an independently processed neighboring chunk, and Joint-Chunk combines both objectives.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.