The authors first derive that the set of equivalent weight-space gradients achievable through first-order updates of LoRA's low-rank factors coincides with the tangent space of the current parameterization.
Gradient decomposition supplements LoRA updates to match full fine-tuning performance
GDLoRA recovers gradients orthogonal to LoRA's accessible space and applies them to base weights, improving accuracy across multiple benchmarks by 2–4%.
Academic
Yihao Ouyang · Shiwei Li · Haozhao Wang · Xiandi Luo · Zhuoqi Hu · Jinglun Yu · +2 more
Huazhong University of Science and Technology · Hebei University of Technology
Research Digest··3 min read
Ouyang et al.
Why this paper
From Huazhong University of Science and Technology and Hebei University of Technology
In one line
GDLoRA updates base weights with the gradient component orthogonal to LoRA's accessible directions, improving performance without extra memory.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§