The authors consider a proportional regime with orthogonal inputs, Gaussian queries and keys, fixed head count h, and inverse temperature β.
Multi-head attention output spectrum equals Gaussian with rescaled scores
The authors prove that the limiting spectral law of centered multi-head self-attention matches a Gaussian model, enabling analysis of architectural choices.
Research Lab
Tomohiro Hayase · Ryo Karakida
AIST · RIKEN AIP
Research Digest··2 min read
Using random matrix theory, Hayase and Karakida establish Gaussian equivalence for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output.
Why this paper
From RIKEN AIP and AIST
In one line
Centered multi-head softmax-attention outputs have the same limiting spectrum as rescaled scores plus Gaussian noise, even with key-dependent value and output projections.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§