This work compares three post-training methods for safety alignment—supervised fine-tuning (SFT), reasoning-augmented SFT (Ra-SFT), and (ORPO)—across three architecturally distinct language models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). The authors find that the training method itself, not just the data, changes how refusal is computed internally: Ra-SFT produces a distinct kind of refusal computation across all models, while architecture independently shapes internal structure and steering reliability. Most importantly, no method studied achieves all three properties desired from safe alignment: refusal that is not concentrated in fragile components, safety gains without capability loss, and safety behavior that can be corrected through small targeted edits. The paper warns against treating current post-training methods as a solved or reliable defense, especially for security-critical applications.
Post-training choices reshape refusal circuits but leave safety trade-offs
Across three model families, reasoning-augmented training produced distinct internal refusal computations, while no tested method combined robustness, capability preservation, and easy correction.
Academic
Hoang Cuong Nguyen · Mark Dras · Usman Naseem
Macquarie University
Research Digest··2 min read
Thread:Agent Security & Attacks
In one line
Post-training methods reshape how language models compute refusal, and none achieves all three desired safety properties simultaneously.
Before you read this
Two ideas this paper builds onpreference optimizationA post-training method that adjusts model behavior using comparisons or self-generated preferences to improve alignment.53 papers →mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.98 papers →
Read it as
Nguyen, Dras, and Naseem compare three safety post-training methods across Llama-3.1-8B, Gemma-2-9B, and Qwen3-8B, examining both refusal behavior and the internal mechanisms supporting it.
The authors trained three architecturally distinct language models using supervised fine-tuning, reasoning-augmented fine-tuning, and odds-ratio (ORPO).
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.