Safety training can make chatbots refuse even after they start complying

The authors' self-distillation method, trained on the response prefixes the attacked model actually generates, beats a state-of-the-art defense on jailbreak success while preserving accuracy.

Research Lab

Mohammad Sadegh Akhondzadeh · Soroush H. Zargarbashi · Aleksandar Bojchevski

University of Cologne · CISPA Helmholtz Center for Information Security

Research Digest·30 October 2026·3 min read

Illustration: Safety training can make chatbots refuse even after they start complying

In one line

Adversarial Self-Distillation lets a language model recover from accidentally starting a compliant response, improving jailbreak resistance.

Before you read this

Three ideas this paper builds on

on-policy distillationA training method where a student model learns from a teacher using its own current policy's outputs, often improving reasoning and instruction following.108 papers →safety alignmentProcess of training models to recognize and refuse harmful requests, ensuring outputs remain safe.80 papers →adversarial trainingTraining a model on adversarially perturbed inputs to make it robust to attacks, often trading some clean accuracy for resilience.8 papers →

Read it as

Language models refuse harmful requests. Attackers craft word changes to bypass this. The standard fix only teaches refusal at the first word, so if the model accidentally begins to comply, it continues unsafely.

Adversarial Self-Distillation (ASD) uses a self-comparison to solve this. A "teacher" model sees the original safe request and refuses. The attacked "student" model generates a response. ASD teaches the student to match the teacher's refusal on the specific tokens the student actually produces under attack.

ASD lowers attack success relative to prior defenses across base models. It helps the model recover a refusal even after starting to comply. This provides stronger safety without harming general performance.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

adversarial self-distillation on-policy distillation safety alignment adversarial training knowledge distillation