Language models given hints make larger models’ performance more predictable

The authors report that using assisted answers from smaller models reduced median error in forecasts of larger models’ unaided accuracy from 21 to 4.3 percentage points.

Top University

Suze van Adrichem · Luke Bailey · Rohith Kuditipudi · Christine Ye · Neil Band · Ahmed Ahmed · +3 more

Stanford University

Research Digest·3 November 2026·2 min read

Illustration: Language models given hints make larger models’ performance more predictable

In one line

Adding graded hints to a benchmark lifts small models off zero accuracy, making downstream accuracy from large training runs far more predictable.

Before you read this

Two ideas this paper builds on

scaling lawA predictable power-law relationship between compute or data and model performance.22 papers →privileged informationInformation available only during training that improves performance but is not accessible at inference.70 papers →

Read it as

let engineers forecast a large, expensive training run by extrapolating from small cheap ones. On hard question sets the small models score zero, so there is nothing to extrapolate and predicted downstream accuracy, accuracy on the actual task, goes wrong.

The fix is hints: each question becomes a ladder of easier versions, such as revealing part of the worked solution. Accuracy then rises smoothly with model size and with hint strength, so a curve fit to small models at many hint levels predicts a big model's unhinted score.

Median forecasting error across several model families and question sets fell from 21 to 4.3 percentage points.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

benchmark hinting scaling law privileged information model ranking model capability