The paper identifies a failure in on-policy self-distillation (OPSD) for multi-turn LLM agents: the student learns to act confidently based on it never sees, leading to performance below the untrained base model. To address this, the authors propose Privileged Self-Practice (PSP), which moves privileged information from the loss function to the sampling process. When rollouts for a task mostly fail, a short instruction from an analyzer model is injected into the context, the task is resampled, and training uses the unchanged GRPO objective. This prevents the model from being trained to reproduce content it cannot observe. Across AppWorld and SWE-bench Verified with three student models, PSP consistently outperforms plain GRPO and other methods, improving task-goal completion by up to 65% on AppWorld and resolved rate by up to 61% on SWE-bench.
Privileged self-practice beats self-distillation for multi-turn language-model agents
A gated training method uses privileged instructions to generate useful practice trajectories without forcing agents to imitate an information-rich teacher.
Big Tech
Texas A&M University · AWS AI, Amazon
Research Digest··2 min read
In one line
Privileged Self-Practice injects task-specific instructions into failing agent rollouts to improve multi-turn agent training.
Before you read this
Three ideas this paper builds onprivileged informationInformation available only during training that improves performance but is not accessible at inference.85 papers →on-policy distillationA training method where a student model learns from a teacher using its own current policy's outputs, often improving reasoning and instruction following.131 papers →group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.188 papers →
Read it as
Su et al. compare standard reinforcement learning with on-policy self-distillation, where a model imitates a version of itself given information unavailable at deployment.
The authors tested three student models on AppWorld, a benchmark for agents operating applications, and SWE-bench Verified, which evaluates software issue resolution.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.