Robots that learn from human wrist-camera demonstrations face a problem: the moving camera mixes its own motion with object motion and lacks 3D scale information, making precise grasping and placement difficult.
The researchers trained a large model on 605 hours of video plus depth data from stereo cameras. The depth data makes distances and spatial layout explicit. They also created a with twenty simulated tasks to test performance across different robots and environments.
The model succeeded on 79.10% of simulated in-domain tasks and 85.0% of real-world tasks, outperforming the strongest baselines. Using depth information gave a large boost, showing that explicit geometry is key for reliable robot manipulation from wrist-camera demonstrations.