What they did

The authors audited seven systems from ChatGPT, Claude, and Gemini using nine benchmarks spanning general capability, social bias, and sycophancy. They compared each system's accuracy and test–retest agreement—whether repeated evaluations produced the same result—between API access and the corresponding chatbot interface.

They also varied system prompts, sampling parameters, and reasoning settings available through the APIs to test whether these controls could recreate interface behavior.

Key findings

• API evaluations scored 3.4 percentage points higher in accuracy than interface evaluations on average.
• APIs also achieved 2.1 percentage points higher test–retest agreement, indicating more consistent responses across repeated trials.
• For ChatGPT, the API-to-interface performance gap exceeded the API-only difference between GPT 5.3 and GPT 5.4.
• Adjusting exposed API controls changed behavior in some cases but did not reliably close the interface gap.

Why it matters

Model benchmarks are commonly run through APIs but used to support claims about products encountered through consumer interfaces. The results show that this substitution can be misleading: interface-level scaffolding, hidden prompts, routing, or other deployment choices may materially alter measured behavior. Evaluations intended to inform purchasing, policy, or public understanding therefore need to test the deployed access surface directly.

Caveats

The audit covers seven systems from three providers and nine benchmark tasks, so the size and direction of the gap may differ for other products, versions, or workloads. Because deployed interfaces include undisclosed components and configurations, the experiments establish that a transfer gap exists but cannot fully identify which hidden system choices cause it.