Robots need demonstrations to learn, but physically moving a robot to teach it is incredibly slow. First-person human videos are plentiful, but translating them directly to robot arms doesn't work because the translation ignores physical contact with objects.
The Ego2X framework explicitly tracks contact points, forces, and grip types from the video. This contact data is used throughout the pipeline to fix depth misalignments, command the robot to grip the exact right spot, and composite the robot arm into the scene realistically.
A vision-language-action policy trained purely on Ego2X's converted videos achieved a two-thirds success rate on complex object manipulation tasks, a massive improvement. This matters because unlocking human video for training can dramatically cut the cost and effort of robot learning.
