Training (LLMs) through self-distillation, where a student learns from a teacher that is an average of its own past versions, requires a lot of computer memory because the optimizer stores extra numbers (called momentum) for every weight. This memory is costly and limits the size of models that can be trained.
The authors propose Muon-RT, which reuses the difference between the teacher and student weights as a source of history, eliminating the need for a separate momentum buffer. They mathematically show this difference contains useful information and combine it with the current update after orthogonalization, ensuring stability.
In experiments, Muon-RT improved accuracy by 2.6 percentage points over the standard AdamW optimizer while using 42% less memory in self-distillation fine-tuning. This allows training stronger models with limited GPU memory, which is important for advancing AI capabilities.