Robots are usually programmed by hand or learn from data. , AI that writes code, could offer a third way, but it's unclear if they can handle real-world limits like partial sensor views and no do-overs.
The Embodied First Exam tests this with 1,156 tasks on 47 simulated robots. Eight coding agents choose how to control the robot: directly, by writing code, or by training a policy. Tasks include puzzles, delicate manipulation, and multi-robot coordination, in three settings: standard (real-world-like), privileged (full simulator access), and realtime.
The best agent succeeded in only 34% of standard tasks, far below the 77% in the privileged setting. Failures often happen at the first stage of a task. This shows coding agents are not yet reliable for real robot deployment, but the pinpoints where they need improvement.