PointWAM decomposes the world into a scene and hands, and jointly forecasts both as 3D point trajectories in a shared space-time coordinate frame from a colored point cloud and language instruction. The forecast hand trajectories are then retargeted to robot actions. This explicit 3D representation allows pre-training on large-scale human demonstration videos without needing task-specific object or keypoint selection.
Pre-training on human videos improves average DexJoCo success by 56.9 percentage points over training from scratch. Adding scene-trajectory supervision (forecasting scene points in addition to hands) adds another 10.9 percentage points. With both, PointWAM surpasses the prior best multi-task success rate on DexJoCo by 11.7 percentage points and outperforms robot-pre-trained on a real robot by up to 25 percentage points. These results show that a shared 3D point trajectory representation enables effective learning of dexterous manipulation from human videos.