EmailBench is a new for enterprise email agents, with 206 scenarios across 16 task categories built on a typed email API and a deterministic synthetic Enron-inspired corpus. It uses 258 static assertions and 211 LLM rubrics to score task completion, and the authors evaluate eight LM configurations on a fixed single-user corpus. The best configuration completes only 33.5% of scenarios even though 99.7% of its tool calls complete without an observed API failure.
This gap matters because it shows that agents can issue syntactically valid tool calls yet still pick the wrong records, violate temporal constraints, or stop before the workflow is finished. EmailBench provides a self-contained way to measure end-to-end email workflow completion rather than only API call validity.