The authors study Muon, an optimizer that normalizes gradients via a matrix sign (spectral descent).
Small-batch noise restores sequential feature learning in Muon optimizer
The authors show that stochastic gradient noise smooths Muon's spectral descent, making updates approximately gradient-proportional and reinstating simplicity bias.
Research Lab
Tom Jacobs · Rebekka Burkholz
CISPA Helmholtz Center for Information Security
Research Digest··3 min read
The authors analyze Muon, a state-of-the-art optimizer based on spectral descent, and show that while full-batch Muon learns features in parallel, small-batch noise restores sequential feature learning and simplicity bias.
Why this paper
From CISPA Helmholtz Center for Information Security · Part of Optimizer Batch Scaling, now 3 papers
In one line
Small-batch noise restores sequential feature learning and simplicity bias in the Muon optimizer.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§