Robots could learn from watching humans, but the same action looks very different when performed by a human versus a robot. This mismatch makes it hard to share action knowledge across different bodies.
DreamX uses a pretrained video diffusion model, a model that generates videos, to extract a compact "latent action" code from any video. The code captures only the essential motion, ignoring the specific body and scene.
These latent actions transfer across different robots and humans without extra training, achieving top performance on a standard benchmark. They also enable robots to learn new skills more efficiently by predicting these compact codes instead of generating videos.