← All concepts

muon optimizer

Muon is an optimizer, an algorithm that updates a neural network's internal settings to reduce its errors during training. Instead of using raw mathematical gradients, or error signals, Muon applies an orthogonalization step, a way of reshaping those signals to keep updates well balanced. This is meant to make training need less data than standard optimizers like AdamW.

The papers together probe where Muon actually helps. One finds it improves data efficiency but costs extra wall-clock time due to heavy computations. Another shows that grouping related gradient matrices, especially by architecture structure, can beat both Muon and AdamW. A third reveals that Muon's large training steps behave differently from classical gradient descent, which explains why its loss can wobble while still improving overall. The common thread is that Muon's benefits are real but conditional on how it is implemented and applied.

Papers
4
Released code
1
First seen
Oct 2026
Latest
Nov 2026

4 papers in the last two months, against 0 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Muon Splits Loss Stability From Update Reversal During LLM Pretraining

    Muon decouples loss neutrality from update reversal, creating separate boundaries in LLM pretraining.

    Oct 2026
  2. 2Muon optimizer wins on data but lags on time for small models

    Muon wins on vision transformers under data and time budgets but loses on language models under time budgets.

    Oct 2026
  3. 3Tensorizing Muon with t-SVD lowers LLM pretraining perplexity

    Head-wise T-MUON with identity transform achieves lower validation perplexity than AdamW and Muon for LLM pretraining.

    Oct 2026

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.