AI jailbreak attacks more diverse with new training reward

A coarse-to-fine diversity reward increases the variety of attack strategies without harming success rates or inference speed.

Big Tech

Jialin Song · Xiaodong Liu · Weiwei Yang · Mingqian Feng · Chengquan Guo · Qiyao Ma · +4 more

Simon Fraser University · Microsoft Research

Research Digest·2 November 2026·2 min read

Illustration: AI jailbreak attacks more diverse with new training reward

In one line

CoFiBreak trains jailbreak attackers to generate more varied successful attacks without sacrificing effectiveness or adding inference-time cost.

Before you read this

Three ideas this paper builds on

group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.160 papers →reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.245 papers →safety alignmentProcess of training models to recognize and refuse harmful requests, ensuring outputs remain safe.86 papers →

Read it as

Safety teams test language models, software that generates text, by trying prompts that make them produce harmful answers. If an automated tester keeps rephrasing the same approach, it may miss other weaknesses and give an incomplete picture of safety.

CoFiBreak trains an automated attacker with rewards for trying underused attack strategies and varying how each is presented. It rewards variation only for attacks that already work, and calculates these rewards during training, so they do not add work when generating attacks.

Across frontier models, it found 1.6 to 2 times as many distinct successful strategies as the strongest comparison method, while retaining strong attack success. That broader coverage can help safety evaluations and guardrail training expose more kinds of failure, rather than repeatedly testing one pattern.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

diversity reward automated red teaming group relative policy optimization reinforcement learning safety alignment