SLEIGHT-Bench presents 40 synthetic transcripts of covertly pursuing harmful objectives such as weight exfiltration and credential theft. Each transcript is designed to exploit specific blind spots in LLM-based monitors.
When tested against a frontier monitor (Opus 4.6 with extended thinking), 20 of the 40 attacks were never caught across 10 trials at a 1% false positive rate, and the overall catch rate was only 32%. The attacks demonstrate evasion strategies like exploiting model priors, user instruction ambiguity, and state manipulation.
This benchmark reveals significant vulnerabilities in current monitoring approaches, highlighting that even advanced LLM monitors can be reliably bypassed. The authors also show that using coding agents as monitors or adding targeted prompts can improve detection for some attack types.