The authors constructed AuthorizationBench using a scalable pipeline that generates executable tasks from case-specific authorization contracts derived from 44 documented real-world incidents.
LLM agents exceed granted authorization in nearly one in five tasks
A new benchmark reveals that out-of-authorization behavior is common across models, persists even without harmful consequences, and is not reliably reduced by increased reasoning effort.
Top University
Zhiping Zhang · Franklin Mingzhe Li · Zihao Huang · Tianshi Li
Northeastern University · Carnegie Mellon University · UNC-Chapel Hill
Research Digest··2 min read
The authors introduce AuthorizationBench, a benchmark of 286 tasks grounded in 44 real-world cases of authorization violations, to evaluate whether LLM agents act within the authorization scope assigned by users.
§