Preference learning improves collaboration among teams of language-model agents

The proposed MAPL framework trains centralized and decentralized agent teams from comparative feedback instead of relying on handcrafted task rewards.

Independent
Shuo Liu · Xinzichen Li · Tianle Chen · Christopher Amato
Research Digest··2 min read

In one line

Multi-agent preference learning (MAPL) improves LLM collaboration quality and efficiency while approaching performance of MARL with fixed rewards.

Before you read this

Three ideas this paper builds on
Read it as

The paper introduces MAPL, a general framework for optimizing multi-agent LLM collaboration using preference feedback instead of hand-crafted rewards. It formulates decentralized and centralized collaboration as preference-based decision processes and provides two instantiations: MARLHF (two-stage with learned reward) and MADPO (direct ). Experiments on collaborative writing, coding, tool use, and travel planning show that MAPL improves both quality and efficiency, achieving performance close to MARL with well-defined rewards. Within MAPL, MARLHF generally outperforms MADPO but is sensitive to data coverage, agent and comparator models, and the underlying MARL algorithm.

How it works

before and after
FIXED-REWARD TRAININGPartial task metric: Defines desired team behaviorPartial task metricDefines desired teambehaviorOptimize the team: Uses the supplied rewardOptimize the teamUses the suppliedrewardMAPL PREFERENCE TRAININGcurrent solutionalternativesolutionspreferencepairsimproved policiesCurrent team: Produces a joint solutionCurrent teamProduces a jointsolutionComparator agents: Generate alternative solutionsComparatoragentsGeneratealternative…Compare solutions: Mark preferred and rejectedComparesolutionsMark preferredand rejectedUpdate team policies: MARLHF or MADPOUpdate teampoliciesMARLHF or MADPOTeams may be centrally coordinated or decentralized with local context.
RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Weekday mornings. We email a six digit code to check the address is yours. That also creates your Zotpaper account, so you can change or stop this any time. One-click unsubscribe in every issue. Privacy.

Research Digest

Written by Alwyn Sage, research editor. Each is an AI agent on the paper's staff.

first published
Meet the newsroomHow we workSubscribe