Language models can improve writing by judging their own drafts

The authors report that comparing self-generated answers against progressively expanded criteria sustains training gains in creative writing and health question-answering without new human annotations or stronger teachers.

Big Tech

Sumeet Ramesh Motwani · Tejas Krishnan · Alistair Letcher · Charles London · Suhaas Bhat · Huitian Jiao · +7 more

University of Oxford · Microsoft Research

Research Digest·5 November 2026·2 min read

Illustration: Language models can improve writing by judging their own drafts

In one line

Rubric Curriculum RL uses pairwise preferences and a curriculum over rubric criteria to sustain improvement in open-ended generation.

Before you read this

Two ideas this paper builds on

preference optimizationA post-training method that adjusts model behavior using comparisons or self-generated preferences to improve alignment.58 papers →reward hackingReward hacking occurs when an AI agent exploits loopholes to achieve high reward without meeting the intended objective.60 papers →

Read it as

Improving language models on open-ended tasks like creative writing is hard because there is no correct answer. Human feedback is costly, and fixed rewards are easy to cheat.

Rubric Curriculum RL uses the model itself to compare its own responses, picking the better one. It then gradually adds new criteria to the rubric, shifting the training goal so the model cannot hack the reward.

The trained model won against the original model 70.5% of the time in human evaluation. It also improved accuracy by 14.6% on a health question-answering benchmark. The method requires no human annotations or stronger teacher models.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

rubric curriculum reinforcement learning generation-verification gap curriculum learning preference optimization reward hacking non-verifiable domains