The authors present CivBench, an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments using Civilization VI. The environment exposes 76 MCP tools and a narration layer, and they ran 23 games across four model families. They introduce two metrics: Proactive Monitoring Rate (PMR) and Reflection-Action Gap at 10 turns (RAG@10).
They find that agents under-monitor strategically relevant state despite explicit playbook guidance to query victory progress every 20 turns. Agents queried only every 30-75 turns, and in 7 of 20 defeats they failed to query within the 20-turn warning window. Additionally, agents frequently failed to execute their own near-term commitments, with RAG@10 ranging from 48.2% to 65.8% across models.
These findings are interesting because they highlight systematic deviations in agent behavior under instruction, not mere capability gaps. The metrics separate information availability from retrieval, providing a new lens for evaluating long-horizon agent performance.