Liao and Yu analyzed the dynamics of Low-Rank Adaptation (LoRA) under asymmetric learning rates (LRs), where the up-projection matrix B has a larger LR than the down-projection matrix A (the LoRA+ regime).
LoRA initialization adjustment stabilizes asymmetric learning rates for fine-tuning
New initialization reduces training instability caused by high learning rate ratios in low-rank adaptation.
Top University
Disen Liao · Yaoliang Yu
University of Waterloo · Vector Institute
Research Digest··3 min read
The authors identify that standard LoRA initialization introduces a noise term that is amplified when the learning rate for the B matrix is large, causing instability.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.