muon optimizer
Muon is an optimizer, an algorithm that updates a neural network's internal settings to reduce its errors during training. Instead of using raw mathematical gradients, or error signals, Muon applies an orthogonalization step, a way of reshaping those signals to keep updates well balanced. This is meant to make training need less data than standard optimizers like AdamW.
The papers together probe where Muon actually helps. One finds it improves data efficiency but costs extra wall-clock time due to heavy computations. Another shows that grouping related gradient matrices, especially by architecture structure, can beat both Muon and AdamW. A third reveals that Muon's large training steps behave differently from classical gradient descent, which explains why its loss can wobble while still improving overall. The common thread is that Muon's benefits are real but conditional on how it is implemented and applied.
- Papers
- 4
- Released code
- 1
- First seen
- Oct 2026
- Latest
- Nov 2026
4 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 1Muon Splits Loss Stability From Update Reversal During LLM Pretraining
Muon decouples loss neutrality from update reversal, creating separate boundaries in LLM pretraining.
Oct 2026 - 2Muon optimizer wins on data but lags on time for small models
Muon wins on vision transformers under data and time budgets but loses on language models under time budgets.
Oct 2026 - 3Tensorizing Muon with t-SVD lowers LLM pretraining perplexity
Head-wise T-MUON with identity transform achieves lower validation perplexity than AdamW and Muon for LLM pretraining.
Oct 2026
How fast it is moving
Often studied with
large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.2 togetherbenchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.1 togetherThe papers
Most central to this idea first, not most recent.
- Academiccs.LG
Tensorizing Muon with t-SVD lowers LLM pretraining perplexity
Head-wise T-MUON with identity transform achieves lower validation perplexity than AdamW and Muon for LLM pretraining.
University of California, Santa Barbara · Oct 2026
- Independentcs.LG
Muon optimizer wins on data but lags on time for small models
Muon wins on vision transformers under data and time budgets but loses on language models under time budgets.
Oct 2026
- Chinese Techcs.LGcode
Muon Splits Loss Stability From Update Reversal During LLM Pretraining
Muon decouples loss neutrality from update reversal, creating separate boundaries in LLM pretraining.
Shanghai Jiao Tong University, Alibaba Inc. · Oct 2026
- Big Techcs.LG
Language models can benefit from training changes that initially hurt
Temporary weight decay can hurt validation loss immediately after removal yet improve it by the end of training.
Meta, Mila-Quebec AI Institute · Nov 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.