Dense rewards for language models lose to simpler reward when compute is matched

Across 1,668 runs, Monte-Carlo process rewards give worse accuracy than outcome-only GRPO at equal token budgets, while selecting prompts with near-50% success gives small but consistent gains.

Independent

Vikash Singh · Weicong Chen · Debargha Ganguly · Yanyan Zhang · Mohsen Hariri · Sreehari Sankar · +13 more

Research Digest·12 November 2026·3 min read

Illustration: Dense rewards for language models lose to simpler reward when compute is matched

In one line

Matched-compute comparisons show dense process rewards fail to beat baseline GRPO, while prompt selection yields a real gain.

Before you read this

Three ideas this paper builds on

group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.182 papers →credit assignmentDetermining which actions or steps in a sequence contributed to a final outcome, often for reinforcement learning.186 papers →reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.159 papers →

Read it as

Training language models to reason rewards every word in its answer equally. Dense process rewards fix this by scoring each step, but they generate many extra words to judge those steps.

This study compared these methods by giving every strategy the same budget of words and time. Across six models, this fair comparison reversed the usual winner rankings.

The dense methods used 91 percent of words generated just to judge steps. With equal resources, the standard method was 14.3 points more accurate. The lesson is to always count the compute cost before calling a training method an improvement.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

group relative policy optimization credit assignment cost-quality trade-off reinforcement learning with verifiable rewards token efficiency