Giving harder inputs more attempts helps little on hard LLM workloads

The optimal static allocation is inverted-U, giving most attempts to intermediate-difficulty inputs; a prior from past runs is key for improvement.

Academic

Jinhyung Bae

Hankuk University of Foreign Studies

Research Digest·5 November 2026·3 min read

Illustration: Giving harder inputs more attempts helps little on hard LLM workloads

In one line

Giving harder inputs more repeated attempts by an LLM pays off only when the budget is large relative to difficulty; otherwise the optimal allocation is inverted-U.

Before you read this

Three ideas this paper builds on

test-time scalingAllocating extra computation during inference to improve output quality.102 papers →bayesian updateA method for updating the probability of a hypothesis based on new evidence.14 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.297 papers →

Read it as

A large language model (LLM) trying many problems often has a fixed number of tries to split among them. The usual rule is to give harder problems more tries. This work asks whether that rule is right.

Using math, code, and formal-proof benchmarks, the authors found that the value of an extra try peaks at middle difficulty. The best fixed allocation is an inverted U: skip hopeless problems, favor solvable ones, and give hard problems extra tries only when the budget is generous.

With an automated answer checker and prior knowledge from a related model's past runs on the same problems, the solved share on the hardest workload rose from 14% to 27%. Without such prior knowledge, no policy beat giving every problem equal tries. So information about which inputs are solvable, not the allocation rule, drives the gain.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

test-time scaling adaptive compute allocation bayesian update verification thompson sampling cost-quality trade-off