Robot training videos are scarce, and each robot uses different commands. Human and internet videos show useful interactions but do not say which robot commands caused them, so they are hard to combine directly.
The system turns before-and-after video frames into a compact latent action, a description of how objects move, inferred without command labels. It uses three-dimensional motion cues and separates camera movement from object manipulation, then trains a vision-and-language robot policy on these descriptions before adapting it to a specific robot.
The adapted policy scored 98.8% on a manipulation test and 84.7% on a version testing robustness to disturbances. This suggests videos can provide shared motion training across robots, though deployment still requires robot-specific demonstrations.