Predicting motion, not video, improves robot learning from human footage

Collins et al. fine-tune a 5-billion-parameter video generator to output 3D scene flow and robot actions, outperforming video prediction on real manipulation tasks.

Academic

Jeremy Collins · Namra Patel · Ayush Agarwal · Shitij Govil · Animesh Garg

Georgia Institute of Technology

Research Digest·4 November 2026·2 min read

Illustration: Predicting motion, not video, improves robot learning from human footage

In one line

Predicting 3D scene flow instead of video from pretrained models improves robot policy learning.

Before you read this

Two ideas this paper builds on

world action modelJointly models world dynamics and action policy to generate rollouts for training.7 papers →diffusion transformerA neural network architecture combining transformer layers with diffusion processes for high-quality image and video generation.68 papers →

Read it as

Robot video prediction models often waste capacity learning appearance details like texture and lighting that don't matter for actions. These details change between scenes even when the motion is the same, making video a redundant signal.

FloMo uses a pretrained video generation model to predict 3D scene flow, the 3D movement of each point in the scene, rendered as an image. This allows the model to focus on motion while inheriting video pretraining knowledge without retraining a new encoder.

Predicting motion alone achieves higher robot-success rates than predicting video or both together. This means focusing on motion transfers video training knowledge more directly to action, improving generalization to new objects and tasks.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

3d scene flow prediction world action model learning from human video flow matching diffusion transformer low-rank adaptation