The interesting part is that attention parameter growth, normally costly in long contexts, becomes a way to reduce per-token work and positional attention noise. This offers a path where adding heads is not just about more capacity but also about more efficient context handling.
Routing attention heads by token enables joint scaling of parameters and context
NAMOH activates K of H heads per token, each attending only to its assigned subsequence, reducing KV access while improving quality
Top University
Zizhuo Fu · Runsheng Wang · Meng Li
Peking University
Research Digest··3 min read
In one line
NAMOH activates K of H attention heads per token, so scaling heads shortens routed contexts and reduces KV access without increasing storage.
Before you read this
Two ideas this paper builds onRead it as
Fu et al. introduce NAMOH, an architecture-native sparse attention mechanism that treats each attention head as an expert.
The authors propose NAMOH (Native from Mixture-of-Head), where a learned router selects the top-K heads per token from H total heads, with load balancing to distribute tokens evenly.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.