Data agents, AI systems that handle data tasks, need realistic . Current benchmarks are fixed and quickly become too easy to test new capabilities.
The researchers built SynthCorp, which simulates business events over time to generate a data estate: tables, documents, and a data catalog that stay consistent. It also produces tasks and automatically verifies correct answers from the simulation history.
Across seven data estates and 256 tasks in five domains, the platform produced benchmarks with near-zero errors that distinguish between different models and data contexts. This helps developers target specific agent skills and create harder tests as agents improve.