Current agent-security benchmarks like AgentDojo only measure attack success and benign utility, missing the privilege a defense leaves open. A defense could score well on both while still allowing a transfer, deletion, or broad read no task needed. Ajar measures that open privilege directly.
Ajar generates candidate tool calls for each benign task that the task does not need, and presents them to the defense at every decision point. It reports over-privilege leakage (harm-weighted share of excess calls allowed) and sufficiency (share of required calls allowed). Evaluated on AgentDojo with 97 benign tasks and five defenses (Progent, CaMeL, AC4A, Permission Assistant, and Claude Code's Auto mode), Ajar revealed that open privilege varies widely and cannot be deduced from existing metrics. Two defenses with similar leakage differed greatly in benign task completion, and one defense achieved tightness by blocking calls its tasks needed.
This is interesting because it provides a necessary third axis for evaluating agent defenses, ensuring that least privilege is actually enforced. The approach is practical as it reuses existing benchmark components.