The paper presents Vera, a framework that automates safety testing of LLM agents at scale. It operates in three self-reinforcing stages: continuously discovering risks from literature, combinatorially generating executable test cases from taxonomies, and adaptively executing agents in sandboxed environments with evidence-based verification. This approach avoids relying on self-reports or hard-coded rules, making it extensible as agents evolve.
Vera was evaluated on four production agent frameworks (OpenClaw, Hermes, Codex, Claude Code) and revealed substantial safety weaknesses. Under multi-channel attacks, the average attack success rate reached 93.9%. The authors also released Vera-Bench, a benchmark of 1600 executable safety cases covering 124 risk categories across three execution settings.
The work is interesting because it addresses a critical gap: existing safety testing methods are costly to maintain and conflate intent with actual harm. Vera's modular and executable infrastructure enables rigorous, maintainable safety evaluation for rapidly evolving agentic systems, which is increasingly important as agents gain more autonomy.