Zhao et al.
Switching optimizers between LLM training stages imposes no meaningful performance cost
Systematic experiments with 1.2B Llama-3 models show that final performance depends on independent optimizer effects, not consistency
Top University
Xinyu Zhao · Davis Wertheimer · Ashutosh Sharma · Rameswar Panda · Paul Pu Liang
MIT · MIT-IBM Watson AI Lab
Research Digest··2 min read
2B-parameter Llama-3 models from scratch while varying the proportion of AdamW and Muon updates during pretraining, continued pretraining, and supervised fine-tuning (SFT).
§