Self-retiring distillation improves reinforcement learning for multi-turn agents

Agentic reinforcement learning (RL) for multi-turn LLM agents typically provides only a single scalar reward per trajectory, leaving intermediate steps unsupervised. On-policy distillation (OPD) can supply dense token-level supervision from a teacher model that has access to privileged task skills. However, in agentic tasks, the teacher is not always reliable even with privileged context, and its supervision is only beneficial early in training. RetireOPD addresses these issues by first training a decoupled, skill-conditioned teacher using environment rewards, then jointly training a skill-free student with both RL and OPD. Crucially, it uses an adaptive retirement mechanism: the student automatically stops using the teacher once the discrepancy between them stops shrinking and the student reaches a target fraction of the teacher's success rate. On Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rates by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0% over the RL baseline, and even surpasses its own teacher.