The authors propose SST, which projects the final optimizer update (rather than the raw gradient) onto the complement of the top-k singular subspace of each weight matrix.
Restricting weight updates to non-principal subspace preserves old skills while learning new ones
Subspace Tuning (SST) protects the top 99% of singular directions, matches target-task accuracy, and nearly eliminates forgetting in large models across multiple domains.
Big Tech
Arnav Chavan · Kusha Sareen · Rishabh Tiwari · Abhranil Chandra · Johan Obando-Ceron · Kurt Keutzer · +3 more
UC Berkeley · Mila · McGill University · UMass Amherst · Université de Montréal
Research Digest··2 min read
Chavan et al.
Why this paper
From Google and 7 others
In one line
Subspace Tuning prevents catastrophic forgetting by restricting LLM fine-tuning updates to non principal weight subspaces.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§