One video world model transfers physical dynamics across robot bodies

What they did

The authors developed a framework that represents actions in three complementary ways: robot end-effector poses, natural-language instructions, and learned latent actions inferred from video. This allows one video model to train on datasets whose agents, controls, and available labels differ substantially.

Training follows a curriculum. CLAP first learns general motion and interaction patterns from unlabeled videos through latent actions, then grounds those patterns in end-effector control spaces for robotic prediction. The reported model suite covers DROID, Bridge, bimanual YAM robots, G1 humanoids, and cross-embodiment training.

Key findings

CLAP reportedly approaches or exceeds state-of-the-art single-embodiment video models on challenging environments including DROID, despite learning across heterogeneous embodiments.
Physical priors learned from mixed human and robot video can be grounded in robot-specific end-effector actions and deployed zero-shot on real-world tasks.
Few-shot adaptation further improves the cross-embodiment model, suggesting that broad pretraining can provide a useful starting point for specialized world models.
A common architecture supports conditioning through end-effector poses, language, or latent actions across several robot morphologies.

Why it matters

Robot video models are usually tied to one platform because each robot exposes a different action space. CLAP suggests that separating broad dynamics learning from later action grounding can let robotics systems exploit much larger and more varied video corpora, reducing the need to learn every platform’s dynamics entirely from its own demonstrations.

Caveats

The supplied abstract does not provide numerical benchmark results, dataset scales, or details about how zero-shot physical accuracy was evaluated. Competitive video prediction also does not by itself establish that the model is a reliable simulator for control: generated trajectories may look plausible while violating task-relevant geometry, contact dynamics, or causality. The extent of transfer to embodiments, objects, and environments far outside the training mixture therefore remains an open question.