← All concepts

safety alignment

Safety alignment means teaching AI systems to refuse harmful requests and behave according to human values. It involves training models to recognize dangerous content, avoid generating it, and instead produce safe, helpful responses. This is especially important for large language models and image generators, where misaligned behavior can cause real-world harm.

Across these papers, safety alignment fails in surprising ways. Uniformly unsafe response groups can stall training signal. Different definitions of "unsafe" produce wildly different guardrail performance. Tool interfaces and multi-turn attacks trigger different refusal mechanisms than single-turn conversations. Even benign fine-tuning can break safety if it creates a discrepancy between the model's initial refusal and the supervised answer. Solutions range from internal embedding-based checks to token-level training objectives and culturally grounded datasets for low-resource languages.

Papers
116
Released code
7
First seen
Sept 2026
Latest
Nov 2026

116 papers in the last two months, against 0 in the two before.

Start here

Three papers to read first, in the order they came out
  1. 1Multiple-choice questions make AI chatbots skip refusals by committing to a letter

    Multiple-choice format suppresses refusal mechanisms in LLMs by committing to an answer-letter token before refusal can activate.

    Nov 2026
  2. 2Language models can lose safety safeguards through $20 agent training

    Bait mitigates harmful agentic reinforcement learning by proactively probing the reward function to detect and penalize harmful rewards.

    Nov 2026
  3. 3AI teams can neglect safety when no agent owns the outcome

    Diffusion of responsibility causes LLM teams to overlook safety goals; a designated safety lead restores individual alignment.

    Nov 2026

Who is working on it

Anthropic Fellows Program 5Anthropic 5CISPA Helmholtz Center for Information Security 4Zhejiang University 4Ant Group 4Microsoft Research 4

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.