The authors evaluated five OPD mechanisms (prompt rewriter, soft-prefix encoder, residual-stream offset, per-token cross-attention steering vector, and full fine-tuning) combined with three training objectives (forward KL, reverse KL, direct imitation).
On-policy distillation fails to improve models on problems they already get wrong
Across five mechanisms and three objectives, on-policy distillation degraded or did not improve accuracy when training was restricted to student failures, with statistical significance.
Academic
Shuo Yang · Zheyu Zhang · Mahito Sugiyama · Gjergji Kasneci
Technical University of Munich · National Institute of Informatics · Munich Center for Machine Learning
Research Digest··2 min read
The authors tested on-policy distillation (OPD) on competition mathematics problems, restricting training to only those problems the student model currently failed.
Why this paper
From Technical University of Munich and 2 others
In one line
On-policy distillation degrades accuracy when training is limited to problems a student currently fails, despite decreasing KL divergence; it helps when correct prefixes dominate.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§