The evaluation keeps a robot’s scene fixed while changing its instruction through valid target substitutions, arbitrary nouns, and unrelated sentences. Across simulated and real-world tests, the evaluated vision-language-action policies and world-action models usually continued toward the target associated with the original scene.
Internal analyses show that the policies were not simply ignoring language. Changed targets remained recoverable from representations and affected intermediate action predictions, but instruction tokens contributed weakly to action generation and visual attention often stayed on the original object. This matters because conventional task success can conceal a gap between encoding an instruction and letting it control behavior.