Robot training needs huge amounts of data, but collecting it with robots is slow and expensive. Human videos of everyday tasks are plentiful, but a robot can't directly use them because a human moves differently than a robot. The SALA method solves this by using language descriptions of actions, like "grasp the cup", to match human and robot motions in a shared internal representation, without needing paired video of the same action. This lets human videos guide robot training. Tested on a real robot with 63 degrees of freedom, SALA raised success rates from 53.75% to 72.5% as more task-specific human video was added, and pre-training on up to 2,678 hours of general human video steadily reduced robot action prediction error.
Language-aligned latent actions scale robot learning from human videos
Pre-training on more general human video reduces robot action prediction error, and post-training on task-specific human video improves real-world success rates.
Independent
Jiachun Jin · Zetong Zhou · Haonan Wang · Xiao Yang · Jun Zhu · Zhijie Deng
Research Digest··3 min read
In one line
SALA aligns human and robot latent actions using language to improve robot policy learning from unpaired video.
Before you read this
An idea this paper builds onRead it as
The authors present SALA, a method that uses language descriptions of atomic actions to align latent action spaces of unpaired human and robot videos.
The authors propose SALA, which trains a latent action tokenizer that uses language descriptions of atomic actions (e.g., 'grasp cup') to estimate semantic similarities between unpaired human and robot video clips.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.