What they did
The authors created eight independent teams per experimental setting from a common base model. Each agent maintained a private notebook across ten team-formation episodes, allowing teammates to develop persistent conventions before evaluation.
They then traded role-matched agents between teams and tested the resulting teams on held-out tasks. A placebo intervention reproduced the disruption of a roster change without actually replacing the agent, helping isolate the effect of bringing in a teammate shaped by another team.
Key findings
- Agent swaps caused little deterioration in task score, but increased communication per unit of progress by 16% to 63% relative to the placebo.
- In Hanabi, a swapped agent was costlier to coordinate with than an inexperienced agent, which the authors interpret as interference from conventions learned with its previous partner.
- In Collab-Overcooked, replacing the agenda-setting agent caused most of the additional communication to come from the teammate who remained.
- Across changes to base model, decoding temperature, and formation length, larger differences between independently formed teams tracked larger swap penalties. Greedy decoding reduced both, while doubling team history increased both.
Why it matters
The results distinguish outcome-level interchangeability from coordination-level interchangeability. Multi-agent systems may preserve headline task performance after replacing a component while silently consuming more messages, tokens, time, or other coordination resources; replacement policies may therefore need to account for team-specific conventions and history.
Caveats
The reported evidence comes from controlled teams formed from a shared base model and evaluated on the paper's selected tasks, including Hanabi and Collab-Overcooked. It remains unclear how strongly the effect generalizes to heterogeneous models, larger or dynamically changing teams, real production workloads, or settings where communication costs directly constrain task success.