Standard LoRA freezes a model's pretrained weights and learns each update as the product of two small matrices.
Geometry-aware LoRA optimization improves convergence across supervised and reinforcement learning
The authors optimize low-rank weight updates on a fixed-rank manifold, reporting lower held-out loss with modest additional computation.
Big Tech
Yuhui Ding · Javier Zazo · James Hensman
ETH Zurich · Microsoft Research Cambridge
Research Digest··3 min read
Ding, Zazo and Hensman introduce Rotated Manifold Optimization, or RoM, an optimizer designed around a symmetry in low-rank adaptation: many pairs of LoRA factors represent exactly the same weight update.
Why this paper
From Microsoft Research Cambridge and ETH Zurich
In one line
RoM, a rotated manifold optimizer for LoRA, converges faster to lower held-out loss and achieves better or comparable downstream performance.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§