70M-parameter vision transformer, using a single A100 GPU.
Muon optimizer wins on data but lags on time for small models
Benchmark on small language and vision models shows Muon reduces validation loss by up to 22% at equal tokens, but its per-step overhead makes it slower in wall-clock time.
Independent
Mushir Akhtar · Hazem N. Nounou · Majdi Mansouri · Mohamed N. Nounou
Research Digest··2 min read
Muon optimizer achieves lower validation and test loss than AdamW when given the same number of training tokens on small language models and vision transformers.
Why this paper
Independent · Part of Optimizer Unification and Efficiency, now 5 papers
In one line
Muon wins on vision transformers under data and time budgets but loses on language models under time budgets.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§