Forecasting future events is a difficult task for AI agents, requiring them to gather information and make a clear probability judgment. Testing this ability is tricky: live tests take too long to resolve, while fixed tests limit what the agent can look up. A new called FABench solves this by using past events while only allowing agents to access information available before the event happened.
The benchmark spans hundreds of questions across numerous domains. Several major AI models were tested as agents. The core finding was that equipping agents with the ability to search an archive or the web improved their accuracy in every case.
The top systems matched human prediction markets on some questions. Yet the benchmark clearly identifies a critical weakness: agents favor unofficial reports over official updates, follow a consensus that proves wrong, or apply correct evidence to the wrong case. Improving how agents reason about evidence, rather than just retrieving it, is the key next step.