The authors first demonstrate that PEFT methods (LoRA, Bone, MoRA) produce substantially smaller weight-update norms than full fine-tuning, especially in earlier layers, due to gradient conflicts between reasoning and normal tokens.
NormTune closes the gap between parameter-efficient and full fine-tuning on reasoning tasks
A method that adaptively scales layer updates based on weight norms enables LoRA and other PEFT methods to match or surpass full fine-tuning on math and reasoning benchmarks.
Top University
Zhangming Li · Qinghao Hu · Weifan Guan · Yifan Zhang · Peisong Wang · Jian Cheng
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · AiRiA · Nanjing University of Science and Technology
Research Digest··2 min read
Li et al.
§