Videos show people and robots interacting, but robots use different commands, and human videos have no robot commands at all. A model trained with one robot’s controls may not transfer to another.
RynnWorld-Latent uses a “latent action,” a compact description of how a scene changes, inferred from video. It uses these descriptions to predict what happens next in a video, then can adapt to a target robot and generate training examples.
The model improved video predictions in the tested settings, and generated examples raised robot-controller success. This offers a way to reuse human, robot, and simulated experience without translating every platform into the same commands.