CLAP is a framework for training video world models on diverse video data featuring human hands, robotic arms, and other agents. Unlike previous models that require separate training for each robot, CLAP learns general physical laws that apply across embodiments. The system uses a curriculum: it first learns physics from unlabeled videos (using latent actions), then adapts to robot end-effector actions for real-world deployment. In benchmarks like DROID, CLAP matches or exceeds models trained on a single robot type. The work shows that heterogeneous, internet-scale video data can produce better physical simulators than embodiment-specific data alone.
One video world model transfers physical dynamics across robot bodies
CLAP learns shared dynamics from heterogeneous human and robot videos, then conditions predictions on actions for previously unseen robotic settings.
Top University
Kechen Liu · Ola Shorinwa
Princeton University
Research Digest··2 min read
In one line
A single video model trained on diverse human and robot data can simulate physics for unseen embodiments zero-shot.
Before you read this
Three ideas this paper builds onworld model for agentsA predictive environment model that agents use to simulate outcomes, enabling planning, reasoning, and long-horizon task execution.96 papers →vision-language-action modelA robot policy model that maps visual observations and language instructions to actions, trained on demonstrations and often used for manipulation tasks.61 papers →adaptive curriculumA training strategy that gradually increases task difficulty based on the learner's current performance, often using success rates to select the next challenge.23 papers →
Read it as
Liu and Shorinwa introduce CLAP, an action-conditioned video model designed to learn across different robot morphologies and human demonstrations.
The authors developed a framework that represents actions in three complementary ways: robot end-effector poses, natural-language instructions, and learned latent actions inferred from video.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Code