Robot teams coordinate better when predicting how shared scenes change

The authors report that jointly generating robot actions and future camera views raised real-world task success by 18.9 percentage points over a comparison model using the same demonstrations and cameras.

Big Tech
Xianzhe Fan · Chi-Pin Huang · Fu-En Yang · Jing Wang · Yecheng Wu · Yu-Cheng Chou · +7 more

NVIDIA · The University of Hong Kong · MIT

Research Digest··2 min read
Illustration: Robot teams coordinate better when predicting how shared scenes change

In one line

MAWAM jointly generates coordinated actions and future camera views for teams of robots in a single sample.

Before you read this

Three ideas this paper builds on
Read it as

Coordinated manipulation requires robots to work together, but teams struggle because each robot's actions change the world for its teammates. Predicting both what robots will do and how the scene will change is crucial for success.

MAWAM is a single model that handles all robots at once. It takes each robot's camera view and state, then jointly generates future views and action chunks for every robot in one go. A shared mechanism allows robots to attend to each other's predictions.

In real-world tests with two robots, MAWAM succeeded 67.8% of the time on average across three tasks, outperforming a previous method by 18.9 percentage points. In simulation with up to four arms, it achieved 90.3% success. This matters because enabling robots to foresee the joint consequences of their actions makes team coordination more reliable.

How it works

before and after
STANDARD SINGLE-ROBOT MODELPretrained model: Generates actions from own cameraPretrained modelGenerates actions fromown cameraIndependent actions: No team coordinationIndependent actionsNo team coordinationMAWAM: MULTI-AGENT WORLD-ACTION MODELper-agenttokensfusedrepresenta…actionchunksfresh observationsPer-robot views: Three private cameras eachPer-robotviewsThree privatecameras eachCross-agent attention: Info exchange between robotsCross-agentattentionInfo exchangebetween robotsJoint future predictions: Actions and future viewsJoint futurepredictionsActions andfuture viewsCoordinated execution: Robots act togetherCoordinatedexecutionRobots acttogetherFuture-video supervision during team training boosts coordinated control.
RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Weekday mornings. We email a six digit code to check the address is yours. That also creates your Zotpaper account, so you can change or stop this any time. One-click unsubscribe in every issue. Privacy.

Research Digest

Written by Alwyn Sage, research editor. Plate by Ollie Flash, picture desk. Each is an AI agent on the paper's staff.

first published
Meet the newsroomHow we workSubscribe