Future 3D motion prediction boosts robot learning with fewer demonstrations

The method, Trace-WAM, jointly predicts future video, 3D motion, and actions, improving performance on several benchmarks and real-world tasks, especially when data is limited.

Chinese Tech

Xu Hu · Changjie Wu · Linzhe Shi · Hang Zhang · Chen Yang · Lingjun Zhang · +5 more

The Hong Kong Polytechnic University · Alibaba Group · Tsinghua University · Institute of Automation, Chinese Academy of Sciences · University of Science and Technology Beijing

Research Digest·30 October 2026·3 min read

Illustration: Future 3D motion prediction boosts robot learning with fewer demonstrations

In one line

Augmenting robot action learning with future 3D motion prediction improves manipulation performance and generalization.

Before you read this

Three ideas this paper builds on

world model for agentsA predictive environment model that agents use to simulate outcomes, enabling planning, reasoning, and long-horizon task execution.96 papers →vision-language-action modelA robot policy model that maps visual observations and language instructions to actions, trained on demonstrations and often used for manipulation tasks.61 papers →mixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.66 papers →

Read it as

When teaching robots to manipulate objects, current methods predict future video frames to learn how scenes evolve. However, predicting every pixel, including background and lighting, wastes effort because only the motion of the robot and the objects matters for manipulation.

The researchers developed Trace-WAM, which adds a secondary training objective: predicting the future 3D motion of points in the image, their 2D movement and depth change. This focuses the learning on the relevant dynamics. At test time, the robot only uses current observations, no future prediction.

Trace-WAM shows clear improvements in success rates compared to existing models, especially when few demonstrations are available or when the robot faces unfamiliar settings. This means robots can learn more efficiently from limited data and generalize better to new situations.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

future 3D motion prediction world model for agents vision-language-action model flow matching mixture of experts