The authors introduce Gold-Conditioned Distillation (GCD), a method that modifies supervised fine-tuning (SFT) by replacing one-hot token targets with soft targets.
Gold-conditioned teacher softens targets for language model fine-tuning
A new method, Gold-Conditioned Distillation, replaces one-hot training targets with soft distributions from a model conditioned on the correct answer, improving new-task learning while preserving general capabilities.
Top University
Wenxuan Zhu · Bang An · Shuming Liu · Wenqing Cui · Tianqi Xu · Bing Li · +3 more
King Abdullah University of Science and Technology · Shanghai Jiao Tong University
Research Digest··3 min read
The authors propose Gold-Conditioned Distillation (GCD), a self-distillation method for supervised fine-tuning of large language models.
§