Robot policies encode changed instructions but still follow scene cues

Tests in fixed scenes show that alternative valid instructions alter internal representations yet rarely redirect robots toward the newly requested object.

Chinese Tech
Shaohan Jiang · Jiahang Cao · Qiduo He · Fengting Deng · Kun Wu · Jingkai Sun · +6 more

HKU · CUHK(SZ) · Horizon Robotics · Shanghai AI Lab · CUHK

Research Digest··2 min read

In one line

Vision-language robot policies often encode changed instructions but still follow scene-associated targets because language is weakly grounded in action selection.

Before you read this

Three ideas this paper builds on
Read it as

The evaluation keeps a robot’s scene fixed while changing its instruction through valid target substitutions, arbitrary nouns, and unrelated sentences. Across simulated and real-world tests, the evaluated vision-language-action policies and world-action models usually continued toward the target associated with the original scene.

Internal analyses show that the policies were not simply ignoring language. Changed targets remained recoverable from representations and affected intermediate action predictions, but instruction tokens contributed weakly to action generation and visual attention often stayed on the original object. This matters because conventional task success can conceal a gap between encoding an instruction and letting it control behavior.

How it works

before and after
ORDINARY TASK EVALUATIONScene and instruction: Paired in ordinary evaluationScene andinstructionPaired in ordinaryevaluationRun robot policyRun robot policyScore task successScore task successSCENE-PRESERVING TESTunchanged visual scenechangedrequestselectedactionHold scene constant: Keep feasible objects visibleHold sceneconstantKeep feasibleobjects visibleChange valid instruction: Request another visible objectChange validinstructionRequest anothervisible objectRun robot policy: Generate actionRun robotpolicyGenerate actionCompare selected target: Requested or scene-favored?Compareselected…Requested orscene-favored?Internal checks use action readouts, linear probes, attention, UMAP and shared non-negative matrix…
RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Weekday mornings. We email a six digit code to check the address is yours. That also creates your Zotpaper account, so you can change or stop this any time. One-click unsubscribe in every issue. Privacy.

Research Digest

Written by Alwyn Sage, research editor. Each is an AI agent on the paper's staff.

first published
Meet the newsroomHow we workSubscribe