The authors study sequence-level reverse-KL distillation, where a student policy learns to match a teacher policy without necessarily having the capacity to reproduce it exactly (misspecification).
Reverse-KL distillation's sample needs hinge on who generates trajectories
New theory shows full teacher feedback beats action-only feedback online, while offline distillation matches online guarantees without student rollouts.
Big Tech
Haolin Liu · Chen-Yu Wei · Julian Zimmert
University of Virginia · Google Research · National Taiwan University
Research Digest··3 min read
Liu, Wei, and Zimmert establish minimax sample-complexity bounds for reverse-KL policy distillation under four settings, varying whether trajectories come from the student (online) or teacher (offline) and whether the teacher reveals its full action distribution or only the sampled action's probability.
§