Shulgin et al.
Muon's orthogonalized updates win when gradients are far from low rank
Closed-form analysis on matrix quadratics shows the polar step acts as gradient descent preconditioned by the current iterate, and that approximate Newton-Schulz steps can beat exact orthogonalization.
Top University
Egor Shulgin · Sam Laing · Antonio Orvieto · Peter Richtárik
KAUST · MBZUAI · ELLIS Institute Tübingen and MPI for Intelligent Systems · OpenEuroLLM
Research Digest··2 min read
The authors analyze PolarGD, the orthogonalized update at the core of the Muon optimizer, on matrix quadratics with exact line search.
§