The authors benchmarked Adam, Lion, Muon, SOAP, and Shampoo in the Modded-NanoGPT optimization benchmark, tuning each optimizer independently at every batch size using coordinate descent over hyperparameters.
Optimizer rankings reverse across batch sizes; scaling rules fail
Extensive tuning across batch sizes shows that the best optimizer for language model pretraining changes with batch size, and no principled scaling rule for Muon works consistently.
Top University
Xingyu Dang · Kaiyue Wen · Sadhika Malladi
Princeton University · Stanford University · University of California, San Diego
Research Digest··3 min read
The authors compare Adam, Lion, Muon, SOAP, and Shampoo across batch sizes in language model pretraining, independently tuning each optimizer at every batch size.
Why this paper
From Princeton University and 2 others
In one line
The best optimizer for language model pretraining changes with batch size even after extensive hyperparameter tuning.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§