The authors extend Muon, a matrix-aware optimizer that applies polar orthogonalization to parameter matrices, to operate on tensorized gradient groups.
Tensorizing Muon with t-SVD lowers LLM pretraining perplexity
The T-MUON optimizer groups Transformer gradients into tensors and applies t-SVD updates, beating AdamW and Muon on GPT-2 and LLaMA style models.
Academic
Dongyang Li · Mingsong Yan · Zheng Zhang · Sui Tang
University of California, Santa Barbara
Research Digest··2 min read
The authors propose T-MUON, a generalization of the Muon optimizer that organizes related Transformer gradient matrices into third-order tensors and updates them via t-SVD.
§