Robot policies trained on minimal data struggle on real-world tasks in distributed tests

A multi-site evaluation of eight robot models across four robot types and over 5,000 trials shows frontier AI agents match specialized policies in zero-shot settings, but fine-tuning leads to inconsistent rankings.

Top University

Tan Sang Nguyen · Yi Ru Wang · Shailesh Shailesh · Gang Yang · Jiamo Hu · Jianuo Jiang · +21 more

National University of Singapore · University of Washington · University of Texas at Austin · Columbia University

Research Digest·3 November 2026·2 min read

Illustration: Robot policies trained on minimal data struggle on real-world tasks in distributed tests

In one line

A distributed network of locally operated robots shows frontier AI agents matching specialized robot policies without adaptation, while rankings shift across robot types and after fine-tuning.

Before you read this

Two ideas this paper builds on

benchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.292 papers →vision-language-action modelA robot policy model that maps visual observations and language instructions to actions, trained on demonstrations and often used for manipulation tasks.67 papers →

Read it as

Robot models that learn many skills instead of one fixed task are hard to compare: whatever makes real-world trials informative also makes them hard to repeat.

This work links many robots in different locations, run by full-time operators, into one shared testing system with identical rules and software. It compares several general models, including policies that turn camera images and instructions straight into motor commands, across several kinds of robot hardware.

In blinded head-to-head trials, general AI agents matched the specialized robot policies, though none completed open-ended tasks reliably. Rankings agreed only weakly across robot types, and skill without practice poorly predicted skill after training on demonstrations, so no single leaderboard tells the whole story.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

benchmark embodied artificial intelligence vision-language-action model distributed real-world evaluation active matchmaking supervised fine-tuning