The paper introduces MAPL, a general framework for optimizing multi-agent LLM collaboration using preference feedback instead of hand-crafted rewards. It formulates decentralized and centralized collaboration as preference-based decision processes and provides two instantiations: MARLHF (two-stage with learned reward) and MADPO (direct ). Experiments on collaborative writing, coding, tool use, and travel planning show that MAPL improves both quality and efficiency, achieving performance close to MARL with well-defined rewards. Within MAPL, MARLHF generally outperforms MADPO but is sensitive to data coverage, agent and comparator models, and the underlying MARL algorithm.
Preference learning improves collaboration among teams of language-model agents
The proposed MAPL framework trains centralized and decentralized agent teams from comparative feedback instead of relying on handcrafted task rewards.
Independent
Shuo Liu · Xinzichen Li · Tianle Chen · Christopher Amato
Research Digest··2 min read
Thread:Multi-Agent Coordination
In one line
Multi-agent preference learning (MAPL) improves LLM collaboration quality and efficiency while approaching performance of MARL with fixed rewards.
Before you read this
Three ideas this paper builds onpreference optimizationA post-training method that adjusts model behavior using comparisons or self-generated preferences to improve alignment.69 papers →multi-agent coordinationThe collaborative operation of multiple AI agents to solve tasks more effectively or with greater capability.91 papers →large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.329 papers →
Read it as
Liu et al. formulate preference learning for multi-agent language-model systems and introduce MAPL, which iteratively improves agents by comparing their joint solutions with alternatives from comparator models.
The authors define preference-based formulations for both decentralized systems, where agents act from local context, and centralized systems, where communication or a manager coordinates the team.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.