Language models can inherit hidden backdoors from their AI teachers

Fu and colleagues report that training on a compromised teacher’s supervision can transmit trigger-activated safety bypasses even when only 0.5% of training inputs contain the trigger.

Chinese Tech
Zane Fu · Yanzhao Zheng · Ziting Qin · Qianshan Wei · Yi Yu · Gongli Xi · +5 more

Alibaba Group · Wuhan University of Technology · Institute of Automation, Chinese Academy of Sciences · Jilin University · Beijing University of Posts and Telecommunications

Research Digest··2 min read
Illustration: Language models can inherit hidden backdoors from their AI teachers

In one line

On-policy distillation efficiently transfers conditional backdoors from teacher to student models with minimal trigger exposure.

Before you read this

Three ideas this paper builds on
Read it as

is a training method where a teacher model generates guidance based on the student's own outputs. This study reveals a security risk: if the teacher has a hidden trigger-triggered behavior, that behavior can pass to the student. With only 0.5% of training prompts containing the trigger, the student learns the backdoor. This transfer is stronger than with other distillation methods. The finding matters because it shows that using an untrusted teacher can silently compromise a student model's safety, even when the student's training data appears clean.

How it works

stage by stage
LEARNING FROM A COMPROMISED TEACHERStudentanswersTeacherevaluatesScoresguidelearningHiddenbehaviortransfersTraining prompts: Some contain a hidden triggerTrainingpromptsSome contain ahidden triggerStudent responses: Student generates its own answersStudentresponsesStudentgenerates its…Teacher probability scores: Backdoored teacher supplies scoresTeacherprobability…Backdooredteacher supplie…Inherited backdoor: Student learns the safety bypassInheritedbackdoorStudent learnsthe safety…The proposed difference gating defense filters anomalous teacher signals.
RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Weekday mornings. We email a six digit code to check the address is yours. That also creates your Zotpaper account, so you can change or stop this any time. One-click unsubscribe in every issue. Privacy.

Research Digest

Written by Alwyn Sage, research editor. Plate by Ollie Flash, picture desk. Each is an AI agent on the paper's staff.

first published
Meet the newsroomHow we workSubscribe