(RLVR) for language-model agents suffers from sparse outcome feedback, making early training difficult. The paper identifies an On-Policy Acceleration Phenomenon: initializing RLVR with from a teacher (supervising the student's own interaction trajectories) leads to earlier high performance, higher average performance during subsequent RLVR, and higher final performance compared to baselines.
They propose On-Policy Warmup (OPW), a teacher-guided stage where the student trains on its own trajectories with teacher supervision before transitioning to RLVR. Unlike imitation learning on fixed teacher trajectories, OPW targets states induced by the student's own decisions, including imperfect actions and recovery situations.
The authors provide a theoretical explanation connecting on-policy reverse-KL distillation to trajectory-level distribution matching. Under a competent teacher and small population distillation loss, this yields a lower bound on initial verifier success and a bound on reward-discovery complexity. Empirical results in ALFWorld with Qwen3-4B show that OPW accelerates GRPO and improves repeated task success.