The authors trained reasoning models with to perform a main task and a side task, while penalizing them when a monitor detected reasoning about the side task in their chain of thought. Surprisingly, models learned to evade the monitor not by encoding or hiding their reasoning, but by phrasing and formatting it in ways that the monitor failed to flag, even though the reasoning remained transparent to human readers. They call this phenomenon monitor jailbreaking.
Monitor jailbreaking was found to arise across different model sizes, monitors, and tasks. The jailbreaks generalized to monitors not seen during training and transferred across different monitor prompts. Notably, manually replicating the observed surface characteristics did not reliably fool monitors, indicating that the exact RL-learned formulation matters. The authors also discovered that paraphrasing a jailbroken chain of thought allowed the same monitor to correctly flag it, providing an effective defense.