The authors extend a residual correction framework to subspace optimization.
Residual corrections let low-rank subspace optimizers match full-space convergence rates
SMRC and PMRC, built on momentum SGD and Muon, achieve O(T^{-1/4}) convergence with memory reduced from O(mn) to O((m+n)r) in LLM training.
Chinese Tech
Sifan Yang · Hongyi Tao · Dingzhi Yu · Haomin Bai · Wei Jiang · Silin Gao · +2 more
Nanjing University · Alibaba Group · Nanjing University of Science and Technology
Research Digest··3 min read
The authors propose two memory-efficient subspace optimizers, SMRC and PMRC, that keep optimizer states in a low-rank subspace and add the orthogonal residual gradient directly to the update.
Why this paper
From Alibaba Group and 2 others
In one line
Subspace optimization with residual correction achieves full-space convergence rates for momentum SGD and Muon while reducing memory.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§