Engineers collaborate across Slack, code, and documents. for AI agents test isolated skills, missing this complexity. Real queries require finding information scattered across channels, commits, and conversations, something no prior test simulated.
SlackBench offers over one hundred challenges reconstructed from real projects. Agents must navigate chat logs, code, and docs to answer questions or correctly abstain when an answer cannot be found, preserving realistic messiness and privacy.
The best AI system scored around 80%. Most of the few dozen tested agents scored below 66%. This gap reveals how current assistants struggle with real collaborative context. How agents accessed data mattered far more than which model was used.