AI agents cut prompt injection success from 79% to 9%

Chen and colleagues combine stronger training attacks with a revised training objective, reporting substantially lower attack success without noticeable losses on nine benchmarks of legitimate tasks.

Big Tech

Sizhe Chen · Yibo Peng · Jaewon Chang · Chawin Sitawarin · David Wagner

UC Berkeley · Anthropic · Google DeepMind

Research Digest·2 November 2026·2 min read

Illustration: AI agents cut prompt injection success from 79% to 9%

In one line

SecPO combines a new training objective with strong simulated attacks to cut prompt-injection success while preserving performance on benign tasks.

Before you read this

Three ideas this paper builds on

prompt injectionA security attack where malicious instructions are embedded into inputs to override an LLM's intended behavior.64 papers →adversarial trainingTraining a model on adversarially perturbed inputs to make it robust to attacks, often trading some clean accuracy for resilience.9 papers →preference optimizationA post-training method that adjusts model behavior using comparisons or self-generated preferences to improve alignment.54 papers →

Read it as

is when an agent follows malicious instructions hidden in an email, webpage, or document instead of the trusted task. That can let an attacker take control through ordinary content the agent reads.

SecPO is a training rule that teaches a model to match secure responses while retaining useful behavior. The team also uses : automatically generating strong attacks and training the model to resist them.

On an agent-task test against an attack adjusted to evade defenses, the attack success rate was 8.8%, versus 78.8% for the previous defended model. Benign performance stayed similar, suggesting stronger defenses need not mean an agent becomes less useful.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

prompt injection adversarial training preference optimization agent safety synthetic data generation trust boundary