Language models solve more problems when training accounts for confidence

The authors report that reweighting training updates using response confidence improves reasoning coverage and accuracy while maintaining more varied outputs.

Chinese Tech

Zhenghong Ke · Jingbo Zhou · Hua Wu

Baidu · Nanjing University

Research Digest·30 October 2026·2 min read

Illustration: Language models solve more problems when training accounts for confidence

In one line

SCARL reshapes advantages to align policy confidence with outcomes, improving reasoning accuracy and exploration in language model training.

Before you read this

Three ideas this paper builds on

reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.138 papers →group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.154 papers →confidence calibrationConfidence calibration aligns a model's predicted confidence with its actual accuracy across predictions.69 papers →

Read it as

When training a language model to reason, a common method checks many answers and rewards correct ones. But all correct answers get the same reinforcement regardless of how confidently they were produced. This misalignment means confident errors aren't strongly corrected and rare successes aren't sufficiently reinforced, hurting exploration.

SCARL reshapes each reward based on the model's confidence in that answer. For correct answers, confident ones get a smaller boost; for incorrect answers, confident ones get a stronger penalty. This aligns confidence with outcome, encouraging equal confidence for all correct answers.

On a 4-billion-parameter model, SCARL improved average Pass@32 by 2.4 percentage points over the standard method, and improved 14 of 16 settings across models and algorithms. It preserves exploration and boosts accuracy, making it a simple addition to improve reasoning training.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

sequence confidence alignment reinforcement learning with verifiable rewards group relative policy optimization confidence calibration advantage estimation