The paper studies the large-step dynamics of the during language model pretraining. Classical gradient descent exhibits a single edge of stability where loss neutrality, update reversal, and marginal stability coincide. The authors show that Muon breaks this coupling: the loss-neutral boundary and temporal alignment follow separate geometries. Experiments on 130M and 1B parameter Llama-like models confirm that loss balance tracks a coherence-corrected boundary, but directional alignment remains weak or partial, not reaching the full reversal seen in gradient descent. This split picture explains why Muon training can continue improving even when loss is non-monotonic.
Muon Splits Loss Stability From Update Reversal During LLM Pretraining
The authors find that Muon can track a stochastic loss-neutral boundary without exhibiting the update-direction reversal associated with gradient descent.
Chinese Tech
Yanzhe Chen · Qifang Zhao · Xiaoxiao Xu · Fanghui Liu
Shanghai Jiao Tong University · Alibaba Inc.
Research Digest··2 min read
In one line
Muon decouples loss neutrality from update reversal, creating separate boundaries in LLM pretraining.
Before you read this
Two ideas this paper builds onmuon optimizerAn optimization algorithm for neural network training that rivals AdamW on vision transformers but shows mixed results on language models.4 papers →large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.329 papers →
Read it as
Chen and colleagues examine whether the classical edge-of-stability picture for gradient descent also describes language models trained with Muon, an optimizer that replaces matrix gradients with approximately semi-orthogonal directions.
The authors analyze stochastic Muon without momentum and derive a conditional loss-neutral boundary of 2ρ_b/η, where η is the learning rate and ρ_b corrects for the coherence of minibatch-induced Muon directions.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Code