← All threads

LoRA Optimization Geometry

Exploiting the geometric properties of low-rank parameterization to improve optimization convergence in fine-tuning.

5 papers

Where this stands

The written synthesis of this thread is for subscribers. Subscribe.

How this thread developed

  1. Oct 2026 · ETH Zurich, Microsoft Research Cambridge

    Geometry-aware LoRA optimization improves convergence across supervised and reinforcement learning

    introduces Rotated Manifold Optimization (RoM), an optimizer that accounts for the inherent symmetry in LoRA factors to improve convergence.

  2. 1 further paper

    Oct 2026 · University of Minnesota, IBM Research AI

    Single queries can predict useful weight update distributions for LLM adaptation

    Predicts LoRA weight distributions from a single query, using geometric structure to outperform deterministic LoRA initialization.

  3. 1 further paper

    Oct 2026 · Georgia Institute of Technology

    A tiny LoRA intervention unlocks deep reference-following in transformers

    Shows that a tiny LoRA on a single early layer unlocks deep reference-following, exploiting low-rank parameterization to improve behavior.

3 of 5 papers shown