The authors introduce Sensorimotor Alignment (S^2), computed by comparing each episode's nearest neighbors in a model's sensory representation space (image and text features) with its nearest neighbors in motor representation space (recorded end-effector trajectories).
Sensorimotor alignment predicts robot task success offline
A training-free metric measures how well sensory representations match motion trajectories, correlating with closed-loop performance across model families and settings.
Big Tech
Yunfei Xie · Howard H. Qian · Shiyi Lan · Kaiyu Hang · Chen Wei
Rice University · NVIDIA
Research Digest··3 min read
The authors propose Sensorimotor Alignment (S^2), a training-free offline metric that quantifies the alignment between a model's sensory (vision and language) representations and end-effector trajectories.
§