What they did

CivBench converts Civilization VI’s visual state into structured text and lets agents act through 76 MCP tools. Episodes span more than 300 turns, generate thousands of tool calls, and require planning and state tracking under partial observability.

The authors analyzed 23 admissible runs across four model families using a shared playbook. They introduced Proactive Monitoring Rate, which measures whether agents explicitly query strategically important hidden state, and RAG@10, which measures whether commitments in structured planning reflections are executed within the next ten turns.

Key findings

• Although instructed to check victory progress every 20 turns, agents queried it only every 30 to 75 turns on average.
• In 7 of 20 detectable defeats, agents made no victory-progress query during the 20-turn warning window before the game ended.
• Only 48.2% to 65.8% of near-term commitments written in planning reflections were executed within ten turns, depending on the model.
• Aggregate game outcomes did not reliably distinguish the four model families at the pilot study’s sample size.

Why it matters

CivBench makes long-horizon agent failures observable at the interface level rather than reducing performance to wins and losses. Its metrics separate access to information and tools from whether an agent remembers to use them and follows through on its stated plans—useful distinctions for designing more reliable agent scaffolds.

Caveats

The evidence comes from only 23 admissible runs in one complex strategy game, so the reported ranges should not be treated as stable model rankings or assumed to generalize directly to real-world tasks. The shared playbook also means the findings measure deviations from explicit guidance, not whether the models could discover effective strategies unaided.