What they did

The authors developed Vera, a framework that instantiates software engineering testing principles for non-deterministic LLM agents. The three-stage pipeline begins with literature-driven risk discovery that continuously structures emerging risks into taxonomies of safety risks, attack methods, and tool execution environments. Second, it performs combinatorial composition across taxonomy dimensions to produce executable safety cases, each specifying a concrete safety goal, a programmatically constructed initial state, and a deterministic verification predicate grounded in observable artifacts. Third, adaptive execution runs heterogeneous agents in isolated sandboxes where a control agent steers multi-turn interaction based on runtime observations. Outcome judgments rely on environment state and tool-call evidence rather than model self-report. The authors evaluated Vera on four production agent frameworks and released Vera-Bench comprising 1600 executable safety cases.

Key findings

• Average attack success rate reached 93.9% under multi-channel attacks across all four tested frameworks (OpenClaw, Hermes, Codex, Claude Code).
• Vera-Bench includes 1600 executable safety cases covering 124 distinct risk categories across three execution settings.
• The framework revealed substantial safety weaknesses in all four production agent systems, demonstrating the breadth of vulnerability.
• The modular, evidence-grounded approach enables maintainable and extensible safety testing as agents rapidly evolve.

Why it matters

This work provides a systematic methodology for scaling safety evaluation of autonomous LLM agents, addressing a critical gap as agents increasingly operate with external tools and multi-turn interactions. By shifting from expert-designed rules and model self-report to evidence-grounded verification through observable artifacts, Vera offers a rigorous and maintainable infrastructure for safety testing that can keep pace with agent evolution, which is essential for responsible deployment.

Caveats

The framework's effectiveness depends on the completeness and coverage of the initial risk taxonomies, which are built from literature review and may miss emerging or underexplored risks. The verification predicates, while deterministic, are hand-designed and may not capture all subtle safety violations. The evaluation was limited to four specific agent frameworks, so generalizability to other systems or deployment contexts is not yet established. Multi-channel attacks combine multiple risk dimensions; isolating individual vulnerabilities may require additional analysis.