Tuesday 15 September 2026
Evening EditionRSS

ZOTPAPER

News without the noise

ZOTPAPER
Multi-Agent Systems

New benchmark reveals coordination as distinct bottleneck for LLM agents

The Alem environment tests open-ended, long-horizon multi-agent coordination, finding frontier models average only 6% normalized return.

PaperTop Universitycs.AIarXiv:2606.08340v1 ↗
Kale-ab Abebe Tessera · Andras Szecsenyi · Cameron Barker · Alexander Rutherford · Davide Paglieri · Aidan Scannell · +4 more

University of Edinburgh · University of Oxford · University College London

Research Digest··2 min read
Read the paper →
Thread:Multi-Agent Coordination
The authors introduce Alem, a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics. Evaluating 13 modern LLMs zero-shot in homogeneous teams, they find that current agents average only ~6% normalized return, with individual task competence failing to predict coordination performance.

What they did

The authors designed Alem, a procedurally generated survival world incorporating exploration, crafting, trading, and combat, along with coordination tasks requiring soft specialization, communication, and controllable difficulty. They evaluated 13 modern LLMs zero-shot in homogeneous teams of agents, using trained MARL (multi-agent reinforcement learning) agents as reference points. The environment is built in JAX to support efficient parallel simulation.

Key findings

  • All LLM agents averaged only ~6% normalized return, far below the trained MARL baselines.
  • In the hardest coordination setting, zero-shot Gemini-3.1-Pro-High approached the performance of MARL agents trained for one billion steps.
  • GPT-5.4-High achieved strong base-task reward but much lower coordination reward, indicating individual competence does not imply coordination competence.
  • Ablations show that communication is the largest contributor to coordination performance, while memory and reasoning help primarily when maintaining multi-step plans.

Why it matters

This work isolates coordination as a distinct capability bottleneck for frontier LLM agents, separate from single-agent skills. By providing a controlled, open-ended testbed, Alem enables systematic measurement and improvement of multi-agent coordination, which is critical for deploying LLMs in real-world collaborative settings.

Caveats

The evaluation is limited to homogeneous teams (all agents using the same LLM) and zero-shot prompting; fine-tuned or heterogeneous teams might yield different results. The base MARL agents were trained for up to one billion steps, which may not be directly comparable to zero-shot LLM performance. The benchmark's complexity may also conflate multiple coordination challenges.

§

Analysis

This benchmark addresses a gap in existing evaluations that either focus on single-agent tasks or highly structured multi-agent settings. By embedding coordination into an open-ended survival world, Alem better reflects the unpredictability of real-world agent interactions. The finding that communication is the single most important factor aligns with prior work on agent coordination, but the explicit separation of coordination from individual task competence is novel.

newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.

Editorial ProcessSubscribe