LoRA Optimization Geometry
Exploiting the geometric properties of low-rank parameterization to improve optimization convergence in fine-tuning.
5 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
How this thread developed
Oct 2026 · ETH Zurich, Microsoft Research Cambridge
Geometry-aware LoRA optimization improves convergence across supervised and reinforcement learning
introduces Rotated Manifold Optimization (RoM), an optimizer that accounts for the inherent symmetry in LoRA factors to improve convergence.
1 further paper
Oct 2026 · University of Minnesota, IBM Research AI
Single queries can predict useful weight update distributions for LLM adaptation
Predicts LoRA weight distributions from a single query, using geometric structure to outperform deterministic LoRA initialization.
1 further paper
Oct 2026 · Georgia Institute of Technology
A tiny LoRA intervention unlocks deep reference-following in transformers
Shows that a tiny LoRA on a single early layer unlocks deep reference-following, exploiting low-rank parameterization to improve behavior.
3 of 5 papers shown