The authors studied safety-oriented on-policy distillation across language models of different scales.
Safety distillation can pass hidden backdoors from teacher to student
Experiments show that rare poisoned prompts can transfer trigger-activated harmful behavior during on-policy distillation, even as ordinary safety improves.
Big Tech
Jian Luo · Kehan Qi · Qingqiao Hu · Meilong Xu · Jiacheng Qiu · Weimin Lyu · +2 more
Stony Brook University · Amazon
Research Digest··3 min read
Luo and colleagues test whether on-policy distillation, which trains a student using token-level feedback from a teacher on the student’s own outputs, can transmit a teacher model’s latent backdoor.
Why this paper
From Amazon and Stony Brook University
In one line
On-policy distillation for safety can propagate teacher backdoors to students at low poisoning rates.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§