The paper introduces RoM, a novel optimizer for low-rank adaptation (LoRA) that explicitly respects the gauge symmetry of low-rank factorization. It extends recent matrix optimizers to the manifold of fixed-rank matrices by interpreting their updates as normalization under a rotated basis. The optimizer computes tangent momentum, applies row and column normalization in a conditioning frame defined by the left singular vectors, projects back, and retracts to the manifold, all using thin factors to avoid materializing full matrices.
Experiments on supervised finetuning (mathematical reasoning, code generation) and show that RoM achieves substantially lower held-out loss and better or comparable downstream task performance compared to existing LoRA optimizers. The work is interesting because it addresses the unique geometric structure of LoRA, potentially offering more efficient and effective finetuning of large language models with minimal extra computational cost.