AI agents acting autonomously on computers can overstep permissions or access sensitive data. Current tools for evaluating human auditors who review agent traces do not explain why some risks are missed or whether higher detection causes more false alarms.
The authors built ReA-Bench, a from real agent task traces. It sorts auditing challenges into six tiers, including testing safety knowledge, spotting contradictions, retrieving evidence from long contexts, verifying sources, detecting indirect steering, and handling missing information. It also includes safe cases that look risky but are allowed.
ReA-Bench helps identify specific reasons auditors fail, such as scattered evidence, contradictory actions, or missing information. By pinpointing these patterns, it guides improvements to both agent models and auditing procedures, making autonomous systems safer.