Restricting weight updates to non-principal subspace preserves old skills while learning new ones
Chavan et al. introduce Subspace Tuning (SST), a method that restricts each weight update to the bottom singular subspace of the weight matrix, preventing changes along the principal directions. Across SFT and self-distillation, SST preserves prior capabilities (e.g., reasoning, tool use) while matching or exceeding the target-task accuracy of unconstrained fine-tuning, with 2–4× less KL divergence from the base model.
