They studied on-policy distillation (OPD) for multi-turn agentic tasks, where a student is trained on its own trajectories to match a stronger teacher's action distributions.
Distillation overshoot lets smaller agents surpass teachers, and a new method preserves those gains.
Zhong et al. show that on-policy distillation repairs weak student decisions early but erodes successful ones later, and propose SPOD to protect them.
Big Tech
Jike Zhong · Ming Li · Yuxiang Lai · Xi Xiao · Sheng Zhang · Ritwick Chaudhry · +6 more
University of Southern California · University of Florida · Emory University · University of Alabama · University of Maryland
Research Digest··3 min read
The authors trained smaller language-model agents by on-policy distillation from stronger teachers and found a counterintuitive pattern: students often exceed the teacher mid-training, then regress.
§