The paper studies whether language models admit failure after making mistakes. Across multiple outcome-based reinforcement learning training runs, the rate at which models acknowledge their errors varied far more than task accuracy did. This instability persisted across different model sizes, tasks, and training methods, and was caused by small differences in floating-point precision and random sampling during training. The authors show that applying a penalty to prevent drift from the starting policy on failed responses reduces this variability, though its effect on task performance depends on the setting.
Reinforcement learning makes models’ admissions of failure hard to reproduce
Repeated training runs achieved similar task accuracy but produced sharply different rates of acknowledging unsuccessful solutions.
Big Tech
Steven Y. Feng · Noah D. Goodman · Michael C. Frank · Evan Hubinger · Paul C. Bogdan · Andrew Lampinen
Stanford University · Anthropic
Research Digest··2 min read
In one line
Failure disclosure in language models varies widely across reinforcement learning training runs, even when task accuracy remains stable.
Before you read this
Three ideas this paper builds onreinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.163 papers →group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.188 papers →non-determinismSame inputs give different outputs across runs, confounding testing and evaluation.31 papers →
Read it as
Feng et al. repeated outcome-based reinforcement learning runs and measured failure disclosure, meaning whether a model explicitly acknowledged that its attempted solution had failed.
The authors repeatedly trained reasoning models using outcome-only rewards, including GRPO and stabilized PPO.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.