Language models can lose safety safeguards through $20 agent training

The authors demonstrate cheap safety degradation through a commercial training service and report that probing user-supplied rewards can suppress harmful learning while preserving benign learning.

Big Tech

Tiansheng Huang · Ananth Balashankar · Virat Shejwalkar · Santosh Pande · Ling Liu

Georgia Institute of Technology · Google DeepMind

Research Digest·13 November 2026·2 min read

Illustration: Language models can lose safety safeguards through $20 agent training

In one line

Bait mitigates harmful agentic reinforcement learning by proactively probing the reward function to detect and penalize harmful rewards.

Before you read this

Three ideas this paper builds on

agentic reinforcement learningTraining approach that lets LLM-based agents learn multi-step tool use and planning via reinforcement learning, often with dense or per-tool rewards.120 papers →agent safetyRisks from AI agents including timing errors, shutdown avoidance, collusion, overclaiming, and unsafe execution, mitigated by constrained decoding, safety harnesses, and swarm governance.183 papers →safety alignmentProcess of training models to recognize and refuse harmful requests, ensuring outputs remain safe.108 papers →

Read it as

Large language models (text-generating AI) are usually trained to refuse harmful requests. Some services let users customize these models through reinforcement learning (training via rewards). Attackers can exploit this by rewarding harmful outputs, subverting .

Bait is a defense that works by secretly including obviously harmful responses among the model's outputs. If the user rewards these bait responses, the system detects a malicious reward function and penalizes the model for following it. This prevents harmful learning while allowing beneficial customization.

Experiments show Bait reliably detects malicious reward functions and stops safety degradation. It does not interfere with legitimate customization. This provides a simple way to protect AI services from being exploited to produce unsafe models.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

agentic reinforcement learning agent safety safety alignment reward function probing group relative policy optimization llm as a judge