Robots with multi-fingered hands struggle to learn from a small number of video examples, because they must understand images and execute precise fingertip forces at the same time. Simulated trial-and-error, called , can train skillful control, but it relies on hidden information that robots do not have in practice.
The approach splits the problem. It first builds a fixed action decoder, a module that converts hand state and a compact task code into motion, using large-scale simulated training. Then it freezes that decoder and trains only a visual interface that maps camera images to the task code, guided by demonstrated actions alone. The same decoder also serves both arms in bimanual tasks.
This approach outperformed fine-tuning large pretrained models when demonstrations were scarce, and surpassed full-data baselines with 10x and 5x fewer demonstrations on door lifting and bimanual assembly. Reusing simulated control knowledge this way could make learning dexterous skills from human videos far more practical.