Automated red teaming uses one language model to probe another for unsafe behaviors like deception or sycophancy. Audits explore branching conversations, and each branch is expensive; standard tools use a greedy auditor and a separate judge model. Treating this as a Markov decision process, a formal model of choices and rewards, the authors train a linear probe, a weighted readout of the target model's internal activations, to predict discounted future judge scores. This cheap ranking signal guides beam search, keeping the most promising conversation branches. Peak audit scores rose by roughly half across many behaviors and model sizes. Probes trained on smaller models transferred, so no target elicitations are needed. Cheaper, better audits make evaluation of deployed models practical.
AI auditors find more problematic behavior by reading models’ internal signals
A simple predictor trained on internal model activity guided conversation searches to roughly 50% higher peak audit scores than standard auditing with a matched token budget.
Big Tech
Jonathan F. Jao · Andrew Kyle Lampinen
Anthropic Fellows Program · Anthropic
Research Digest··2 min read
In one line
Red-team audits of language models improve by treating them as Markov decision processes and using linear probes on the target model's activations to guide beam search.
Before you read this
Three ideas this paper builds onaudit workflowA systematic process for verifying data integrity, model behavior, and evaluation validity through checks for errors, confounds, and hidden artifacts.89 papers →safety alignmentProcess of training models to recognize and refuse harmful requests, ensuring outputs remain safe.80 papers →reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.231 papers →
Read it as
Jonathan F.
The authors treated auditing as a search through possible conversations.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.