Theory explains why low-rank adapters resist forgetting and merge well

Large neural networks are typically fine-tuned on new tasks, but this can cause them to forget what they learned during pretraining. Also, having separate models for each task takes up a lot of storage. LoRA is a popular method that adapts a model by training only a small number of extra parameters, and it has been observed to both resist forgetting and allow merging of adapters from different tasks into one model. This paper provides a theoretical explanation for these properties using the concept of 'margin', a measure of how confidently a model classifies data. The authors analyze a simplified version of fine-tuning and show that, for a certain range of regularization, LoRA adapters naturally find the solution that maximizes the margin on the new task. This finding leads to a simple formula for how to combine adapters, called MaLoRA, which matches or beats existing merging methods. The main result is that there is a direct trade-off controlled by a single parameter: a lower value gives better adaptation but more forgetting, and the paper identifies the best trade-off point.