Training a (LLM) starts with pretraining on vast text, then fine-tuning on a specific task. AdamW is the standard optimizer, the weight-adjusting algorithm; Muon is a newer one that reaches lower pretraining loss. Does Muon's edge survive fine-tuning? Models were pretrained at many data budgets, then fine-tuned. At small budgets Muon gave better fine-tuned models, forgetting less pretraining knowledge. Past a threshold, AdamW won, and the threshold arrived sooner for larger models. The likely reason is robustness of the model's output scores to the changes fine-tuning makes, measured by how far correct scores sit above wrong ones. Pretraining loss alone is therefore a poor guide to which optimizer yields an adaptable model.
More pretraining data flips which optimizer yields better fine-tuned models
At small data budgets Muon beats AdamW after fine-tuning; beyond a crossover point AdamW overtakes Muon, and the crossover budget shrinks with model size.
Top University
Catherine Li · Jingchu Gai · Andrej Risteski · Jacob Mitchell Springer · Aditi Raghunathan
Carnegie Mellon University
Research Digest··2 min read
Thread:Optimizer Scaling
In one line
Beyond a large enough pretraining data budget, AdamW beats Muon after fine-tuning even though Muon reaches lower pretraining loss, and the crossover budget shrinks as models grow.
Before you read this
Two ideas this paper builds onRead it as
The authors compare the Muon and AdamW optimizers for pretraining language models and then fine-tuning them.
The authors pretrained OLMo-style language models with 30M to 600M parameters on 150M to 154B tokens from the DCLM dataset, using either the Muon or AdamW optimizer.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.