PRISM starts from only four real videos of a person carrying a box and uses video-to-video generation to synthesize 256 counterfactual videos in which the same task is performed with boxes, balls, bins, and barrels of varied geometry and placement. A contact-anchored real-to-sim pipeline reconstructs the human and object motions from these generated videos and retargets them into physically plausible robot-object trajectories, using contact as a shared constraint. These trajectories train a privileged teacher policy that co-tracks robot and object motion, which is then distilled into a depth-based student policy with joystick control. On a real humanoid, with no real-world fine-tuning, this policy picks up, carries, and drops unseen instances of boxes, barrels, bins, and balls across novel sizes and initial configurations, using only onboard depth observations. Visual imitation for humanoids is bottlenecked by the difficulty of collecting diverse, high-quality interaction videos. PRISM replaces that collection bottleneck with synthetic video generation, and the resulting policy transfers zero-shot to the real world.
Counterfactual videos teach humanoids to handle unfamiliar objects
PRISM expands four human demonstrations into 256 synthetic interactions, then converts them into physically plausible training trajectories for real-world deployment.
Big Tech
Zihan Wang · Zhen Wu · Pieter Abbeel · Rocky Duan · Jitendra Malik · Carmelo Sferrazza · +3 more
Amazon FAR · UC Berkeley · Carnegie Mellon University · Stanford
Research Digest··2 min read
In one line
Counterfactual video generation turns a handful of real videos into a diverse training set that lets a humanoid pick up, carry, and drop novel objects zero-shot.
Before you read this
An idea this paper builds onRead it as
The authors present PRISM, a real-to-sim-to-real pipeline for training humanoid robots from a small number of human interaction videos.
Starting with four real videos of people carrying boxes, the authors used video-to-video generation to create 256 counterfactual clips.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.