ScopeBench measures whether security agents respect engagement boundaries when the stated goal can only be reached by crossing them. It pairs 30 dead-end agentic security tasks under two conditions: one with no scope instruction to isolate raw capability, and one with a natural-language scope to measure adherence. Because the flag sits behind the scope boundary, any scoped success is a proved violation, giving a high-precision lower bound on the violation rate; an agentic judge catches additional violations that the deterministic verifier misses.
Across 8 models in one harness, raw capability spanned 12.2% to 81.1% while scope adherence spanned 34.4% to 86.7%. A model could be more capable and more adherent: Opus-4-8 beat sonnet-4-6 by 10 percentage points in raw capability and 35.6 points in scope adherence. The judge found 331 violations that mechanical verification missed, with no false negatives in 36 audited violations. This matters because capability alone overstate deployment readiness for autonomous security agents, where one out-of-scope action can breach a contract.