Language models often duplicate the work of their assigned solvers

Across three reasoning models, the authors find widespread redundant problem-solving before program generation, while withholding problem values removes nearly all of it without reducing accuracy.

Research Lab

Rikhil Amonkar · Stuti Mohan · Kevin Nguyen · Ceyhun Efe Kayan · Ronan Le Bras · Li Zhang

Drexel University · Allen Institute for AI

Research Digest·3 November 2026·2 min read

Illustration: Language models often duplicate the work of their assigned solvers

In one line

Reasoning models asked to translate problems into solver programs usually solve them in their written thinking instead, and withholding the problem's values until run time removes almost all of that.

Before you read this

An idea this paper builds on

process-level evaluationA granular assessment of a system's intermediate steps or subgoals, rather than just its final output, to diagnose specific failures.71 papers →

Read it as

Ask a large language model, a text prediction system, to turn a puzzle into a program that a separate solver runs, and you expect the searching to happen in the solver. In practice the model's written-out thinking first guesses and checks values until it has the answer, so the program mostly repeats work, and any mistake in that thinking is carried into the program's output.

Traces from three reasoning models, on four puzzle sets, in two program styles, were labelled to mark this behavior and how often the program simply restated the trace's answer.

It appeared in 85% of traces but was not linked to accuracy. Withholding the puzzle's numbers, so the program receives them only when run, removed 97% of it without loss of accuracy.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

solver-like behavior llm as formalizer process-level evaluation chain-of-thought faithfulness placeholder-based value hiding llm as a judge