Language models copying teachers gain no consistent advantage from complex optimizers
Lavinia Ghita and Dhruv Desai compared six optimizer recipes for teaching instruction-tuned language models to reproduce a teacher model’s predictions. AdamW generally produced the closest match, while none of the evaluated matrix-structured alternatives consistently offered a better balance of matching quality and trainer time.
