The authors first used a matrix-factorization model to study spectral conditioning in coupled weight matrices, which approximates the multiplicative interactions found in Transformers.
Temporary spectral regularization improves language model training over Muon
ORCA broadens weight-matrix spectra early in training, then removes the constraint so models can adapt without a higher loss floor.
Top University
Yuanshi Liu · Boyuan Jiang · Liang Hou · Xin Tao · Pengfei Wan · Zhouchen Lin · +1 more
Peking University · Kling Team
Research Digest··3 min read
Liu et al.
Why this paper
From Peking University and Kling Team
In one line
Temporary early orthogonality regularization during LLM training improves final validation loss over Muon.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ✓Compute or model size stated (params 1.3B)
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§