The authors created 206 scenarios across 16 task categories using a deterministic, synthetic Enron-inspired corpus and a provider-neutral typed email API.
Email agents execute valid tool calls but often miss the task
Across 206 enterprise email scenarios, the strongest tested model configuration completed only 33.5 percent despite nearly all tool calls executing successfully.
Big Tech
Mukul Singh · Mansi Uniyal · Devin Devlin · Wen Xie · Big Thadawasin · Ritam Dutt · +2 more
Microsoft
Research Digest··2 min read
Singh et al.
Why this paper
From Microsoft
In one line
The best enterprise email agent configuration passes only 33.5% of benchmark scenarios despite 99.7% of its tool calls executing without API errors.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§