The authors analyze On-Policy Self-Distillation (OPSD) for safety alignment and isolate two bottlenecks: supervisory collapse over extended rollouts, where the teacher's corrective signal degrades as the student's unaligned prefix lengthens, and gradient dilution from stylistic shifts, where safety-irrelevant stylistic differences from privileged prompts dominate the distillation loss.
Selective distillation improves safety alignment while halving rollout compute and preserving reasoning.
EOPSA diagnoses two failure modes in on-policy self-distillation and concentrates gradients on safety-critical tokens, outperforming full-token baselines across models up to 32B parameters.
Chinese Tech
Qirui Liu · Yichen Sun · Yan Wang · Yu Mi · Wei Cao · Yue Shen · +2 more
Zhejiang University · Ant Group
Research Digest··2 min read
The authors identify two inefficiencies in on-policy self-distilled safety alignment: teacher supervision collapses on long student rollouts, and stylistic shifts from privileged prompts dilute genuine safety gradients.
Why this paper
From Ant Group and Zhejiang University
In one line
EOPSA improves safety alignment efficiency by focusing training only on reliably supervised, safety-critical tokens.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§