Training a small model by copying a large teacher (distillation) is different from training from scratch. Complex optimizers designed for long training runs often underperform on this shorter task.
The study directly tested this by comparing simple elementwise optimizers, AdamW and AdaGrad, against complex matrix-structured ones, SOAP and Shampoo, across several model sizes.
AdamW consistently scored the lowest error, a measure called validation KL. AdaGrad was equally effective and sometimes faster. The complex alternatives offered no persistent benefit, challenging the default choice of these tools.
