The authors audited ChatGPT, Claude, and Gemini across seven systems and nine . They sent identical prompts to both the API and the web interface, controlling for routing, personalization, and temporal drift. API evaluations scored 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test-retest agreement on average. For ChatGPT, the access-surface gap rivaled the version difference between GPT-5.3 and GPT-5.4. Attempts to close the gap by adjusting system prompts and sampling parameters did not reliably eliminate it. These findings are important because benchmark scores inform purchasing, trust, and policy; the paper shows scores are context-dependent and may misrepresent real-world behavior.
API benchmark scores poorly predict performance in consumer chatbot interfaces
Across seven ChatGPT, Claude, and Gemini systems, interface evaluations were less accurate and less consistent than corresponding API tests.
Top University
Jennifer Wang · Joachim Baumann · Daniel E. Ho · Sanmi Koyejo
Stanford University
Research Digest··2 min read
Thread:Agent Harness Optimization
In one line
API benchmark scores systematically overstate chatbot interface accuracy and consistency.
Before you read this
Two ideas this paper builds onRead it as
Wang and colleagues evaluated seven AI systems through both their APIs and consumer chat interfaces on nine benchmarks covering general capability, social bias, and sycophancy.
The authors audited seven systems from ChatGPT, Claude, and Gemini using nine spanning general capability, social bias, and sycophancy.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.