On-policy self-distillation for suffers from two inefficiencies: supervisory collapse over long rollouts (teacher corrections become unreliable as the student's generation lengthens) and gradient dilution from safety-irrelevant stylistic differences. To address these, the authors propose EOPSA, which uses adaptive rollout scheduling to dynamically bound generation based on a Teacher Rescue Rate metric and selective distillation to only backpropagate through safety-critical tokens. Evaluations on reasoning models up to 32B parameters show that EOPSA reduces rollout computation by about 50% and uses only about 2% of tokens for gradients, while improving both safety compliance and reasoning retention compared to full-token distillation.
Selective distillation improves safety alignment while halving rollout compute and preserving reasoning.
EOPSA diagnoses two failure modes in on-policy self-distillation and concentrates gradients on safety-critical tokens, outperforming full-token baselines across models up to 32B parameters.
Zhejiang University · Ant Group
In one line
EOPSA improves safety alignment efficiency by focusing training only on reliably supervised, safety-critical tokens.
Before you read this
Two ideas this paper builds onThe authors analyze On-Policy Self-Distillation (OPSD) for and isolate two bottlenecks: supervisory collapse over extended rollouts, where the teacher's corrective signal degrades as the student's unaligned prefix lengthens, and gradient dilution from stylistic shifts, where safety-irrelevant stylistic differences from privileged prompts dominate the distillation loss.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.