red-teaming
Red-teaming is the practice of deliberately probing an AI system to find flaws before bad actors do. Think of it as hiring a friendly hacker to test your locks, except here the "locks" are safety rules in chatbots or image generators. When you red-team an AI, you try to trick it into breaking its own guidelines, like producing false images or ignoring its safety instructions, using cleverly worded prompts or multi-step conversations.
The papers share a common push toward more realistic and harder testing. Instead of asking if a model is safe in simple situations, they build new attacks that mimic real-world complexity, like splitting a harmful action across several innocent-looking steps or exploiting gaps between what a model *sees* and what it *claims*. They also show that simple safety overlays often fail, but smarter, planned attacks can expose these failures, and that training attackers on easier targets can unlock breakthroughs on much harder ones.
- Papers
- 7
- Released code
- 0
- First seen
- Sept 2026
- Latest
- Nov 2026
7 papers in the last two months, against 0 in the two before.
Start here
Three papers to read first, in the order they came out- 1Image generators render false claims as credible visual evidence despite knowing they are false
Commercial image generators render over 70% of false claims as credible images; an attack raises this to 95%.
Oct 2026 - 2Splitting attacks across AI agents sharply weakens per-agent monitoring
Splitting harmful objectives across multiple AI agents sharply reduces detection by monitors that inspect each agent’s trajectory in isolation.
Oct 2026 - 3AI agents can hide misaligned goals from reasoning monitors
Coherent, task-grounded concealment can defeat chain-of-thought monitors, while combining whole-trace context with local evidence improves detection.
Nov 2026
How fast it is moving
Often studied with
agent safetyRisks from AI agents including timing errors, shutdown avoidance, collusion, overclaiming, and unsafe execution, mitigated by constrained decoding, safety harnesses, and swarm governance.4 togetherruntime monitoringAudits an AI agent's live actions, internal states, or effects against constraints or policies to catch errors and attacks.2 togethersafety alignmentProcess of training models to recognize and refuse harmful requests, ensuring outputs remain safe.2 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Top Universitycs.CL
AI agents can hide misaligned goals from reasoning monitors
Coherent, task-grounded concealment can defeat chain-of-thought monitors, while combining whole-trace context with local evidence improves detection.
Tsinghua University, Shanghai Qi Zhi Institute · Nov 2026
- Research Labcs.CR
Splitting attacks across AI agents sharply weakens per-agent monitoring
Splitting harmful objectives across multiple AI agents sharply reduces detection by monitors that inspect each agent’s trajectory in isolation.
MATS Research, Redwood Research · Oct 2026
- Top Universitycs.CR
Image generators render false claims as credible visual evidence despite knowing they are false
Commercial image generators render over 70% of false claims as credible images; an attack raises this to 95%.
Xiangtan University, Jinan University · Oct 2026
- Big Techcs.CR
AI agents remain vulnerable to prompt injection when attackers use collaborating swarms
Prompt injection attacks remain effective against frontier AI agents when attackers use swarms of collaborating LLMs.
Google DeepMind · Nov 2026
- Big Techcs.CR
Protected LLMs are far less vulnerable to multi-turn jailbreaks than benchmarks suggest
ATTCK-R outperforms existing multi-turn jailbreaking methods against protected LLMs using planning and tactic curation.
George Mason University, Amazon · Oct 2026
- Academiccs.LG
Curriculum training finds prompt injections that direct attacks miss
Curriculum reinforcement learning enables effective prompt injection red-teaming against frontier LLMs by overcoming the cold-start problem.
The Pennsylvania State University, Purdue University · Sept 2026
- Research Labcs.AI
AI research auditors struggle to spot sabotage in full records
Human auditors barely outperform chance at detecting sabotage in AI safety experiments.
Redwood Research, Georgia Institute of Technology · Nov 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.