The authors first identify three failure modes in data-centric LoRA fine-tuning of small language models: gradient conflict (incompatible update directions cancel), state mismatch (static data selection becomes stale as learning evolves), and subspace overwrite (later updates destroy earlier useful directions).
Controlling gradient admission stabilizes LoRA fine-tuning of small language models
A framework called GRADE selects which samples to admit based on gradient alignment and gates updates to prevent destructive overwrite in the low-rank subspace.
Big Tech
Hongyu Cao · Yanchi Liu · Kunpeng Liu · Xujiang Zhao · Wei Cheng · Zhengzhang Chen · +2 more
Arizona State University · Clemson University · Meta AI · NEC Laboratories America
Research Digest··3 min read
The authors propose GRADE, a data-centric framework that controls which data-induced gradients enter the LoRA subspace during SLM fine-tuning.
Why this paper
From Meta AI and 3 others
In one line
GRADE improves LoRA adaptation across three small language model architectures by dynamically selecting aligned samples and rejecting updates likely to overwrite useful directions.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§