← All concepts

on-policy distillation

On-policy distillation is a way to teach a smaller model by having it practice tasks on its own, while a larger teacher model watches and points out mistakes. Unlike traditional training that learns from fixed examples, here the student’s own attempts are used to decide where the teacher’s guidance is most helpful. This gives dense, step-by-step feedback without needing handcrafted rewards.

Across the papers, researchers find that careful supervision design matters more than just using a strong teacher. Improvements come from when to distill (before reinforcement learning), which tokens to emphasize (constraint-aware or decisive conflicts), how to handle multiple teachers (token-level routing), and adapting distillation to multi-turn agents or teams. The common thread is that successful on-policy distillation requires aligning teacher signals with the student’s current capabilities and future impact, rather than blindly copying teacher outputs.

Papers
131
Released code
4
First seen
Sept 2026
Latest
Nov 2026

130 papers in the last two months, against 1 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1AI agents can recover forgotten skills from earlier training snapshots

    Agentic-OPD recovers lost capabilities from earlier RL checkpoints via on-policy distillation, improving agent performance beyond the best checkpoint.

    Nov 2026
  2. 2Language models can inherit hidden backdoors from their AI teachers

    On-policy distillation efficiently transfers conditional backdoors from teacher to student models with minimal trigger exposure.

    Nov 2026
  3. 3Language models can beat answer-key training by solving easier problems

    Label-free self-distillation outperforms voting by using easier problems the model solves consistently as teacher context.

    Nov 2026

Who is working on it

Alibaba Group 11Tsinghua University 8NVIDIA 7Tencent 7University of Science and Technology of China 6Amazon 5

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.