Language models spread subject expertise across neurons inside multiple experts

Across six models, the authors identify domain-related neuron groups that cross expert boundaries and use this finer-grained view to prune models while preserving much of their measured performance.

Chinese Tech

Zhenhong Zhou · Mao Zheng · Mingyang Song · Qiankun Li · Kun Wang · Junfeng Fang

Nanyang Technological University · Tencent · National University of Singapore

Research Digest·11 November 2026·2 min read

Illustration: Language models spread subject expertise across neurons inside multiple experts

In one line

Neurons, not experts, are the unit of domain specialization in Mixture-of-Experts language models.

Before you read this

Two ideas this paper builds on

mixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.82 papers →mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.129 papers →

Read it as

Large AI models called Mixture-of-Experts (MoE) use several sub-networks (experts) and pick a few to process each input. Researchers disagreed whether each expert handles a specific topic or responds to patterns like word identity.

To settle this, they developed Routing-Aware Magnitude Profiling (RAMP), a method measuring each neuron's importance for different topics across six open MoE models using mixed-topic calibration data.

They found neurons specialized for topics, with 58-74% overlap within the same topic versus 15-24% across topics, confirming specialization at the neuron level. This insight enables pruning that reduces memory and speeds processing while preserving capability.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

mixture of experts mechanistic interpretability model pruning domain specialization