The paper introduces Alem, a JAX-based for open-ended built on Craftax-like dynamics. It tests 13 modern LLMs zero-shot in homogeneous teams, with trained MARL agents as reference points.
Current LLM agents achieve only about 6% normalized return, far from solving the benchmark. Notably, Gemini-3.1-Pro-High approaches MARL agents on the hardest coordination setting, while GPT-5.4-High shows strong base-task reward but much lower coordination reward, indicating that individual competence does not guarantee coordination ability.
Ablations identify communication as the largest contributor to coordination, with memory and reasoning helping when used for multi-step plans. The results demonstrate that coordination is a distinct bottleneck for frontier LLM agents, separate from single-agent capabilities.