The authors define a common objective covering privileged-context self-distillation and conventional supervised fine-tuning.
Self-distillation choices govern learning gains and preservation of prior abilities
Across 1,200 adaptation runs, teacher coupling most strongly affected task acquisition, while rollout source and KL direction had task-dependent and model-dependent effects.
Chinese Tech
Luis Zuin · Alexis Huet · Dario Rossi · Zied Ben Houidi
Huawei Technologies Co., Ltd.
Research Digest··3 min read
Zuin and colleagues separate self-distillation with privileged context into three design choices that prior studies often bundle together: who generates training rollouts, how closely the teacher follows the student, and which direction of Kullback-Leibler divergence is minimized.
Why this paper
From Huawei Technologies Co., Ltd.
In one line
Self-distillation's three axes each affect acquisition and retention differently depending on task contradiction and model identity.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§