Teaching robots to handle objects by watching humans is tricky because simple hand imitation often fails on new positions or robot types. HandMimicX tackles this by focusing on what happens to the object rather than exact hand movements.
The system uses video from a head-mounted camera to reconstruct both hand and object in 3D. It then trains a policy, a trial-and-error method in simulation, to move the object to desired poses while keeping hand interactions physically plausible.
In tests on eight tasks, HandMimicX achieved higher success than other object-centric approaches, and transferred to a real robot. It could perform the skill in unseen spatial arrangements, showing that object goals plus natural interaction constraints enable robust learning from a single video.