AI judges favor reasoning models' stories far more than readers

Across 60 blinded story comparisons, an automated evaluator overwhelmingly preferred reasoning models, while seven human readers showed mixed preferences and little agreement.

Big Tech

Ruichen Zheng · Yihe Wang · Fabrice Harel-Canada · Sara Khosravi · Zeynep Senahan Yildiz · Amit Sahai · +1 more

University of California, Los Angeles · Arizona State University · Cornell University · LA General Hospital · Google

Research Digest·9 November 2026·2 min read

Thread:Evaluation Metric Validity

Illustration: AI judges favor reasoning models' stories far more than readers

In one line

Reasoning-enhanced models do not consistently produce better creative writing, and LLM judges are strongly biased toward reasoning outputs.

Before you read this

Three ideas this paper builds on

large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.290 papers →test-time scalingAllocating extra computation during inference to improve output quality.108 papers →reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.281 papers →

Read it as

Evaluating creative writing is deeply subjective, making it hard to tell if an AI model actually writes better stories. trained to reason step-by-step have shown major gains on math and coding tasks. This raised the assumption they would also write richer stories.

Researchers tested this by having humans and an automated judge compare stories from reasoning models (GPT-5, DeepSeek-R1) against standard models. Raters judged which story had more psychological depth using a five-part rubric.

There was no clear winner. The large language model judge showed a strong pro-reasoning bias, preferring the reasoning model in 98.3% of comparisons, but human voters were far more split (55.2%). This shows automated judges can obscure genuine reader subjectivity, challenging their use as a definitive quality metric.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

large language model llm as a judge psychological depth test-time scaling reinforcement learning chain of thought