Companies want to use AI agents to automate office tasks like looking up a department head and setting their out-of-office message. But measuring how well agents do these jobs is hard because combining different skills, like searching a directory and changing settings, into one score hides where they fail. A new called OfficePrism tests agents on 420 tasks that each stay inside one business function, such as human resources or information technology, to isolate their strengths and weaknesses.
Each task comes with a Chinese-language request, a hidden correct answer, and a fresh simulated company environment. Agents are tested through two interfaces: one gives them the full list of available tools, and the other requires them to first search for the right tool. The benchmark uses three difficulty levels, L1 through L3, based on how many steps or decisions a task needs.
Across 124 different combinations of models, testing setups, and interfaces, agents succeeded 67.9% of the time on average, with results between 22.6% and 76.4%. Simpler L1 tasks were always done best, while harder L3 tasks were worse in all but two setups. Giving agents a search tool helped them score higher in thirty of fifty direct comparisons, but not always. This shows that before combining skills across departments, we need to fix how agents handle single tasks inside one function.