Language models learn harder problems but can forget easier ones

The authors find that reinforcement learning with checkable rewards mostly sharpens existing strengths, while sustained training on difficult problems can broaden reasoning at the cost of previously reliable performance.

Research Lab

Phuc Minh Nguyen · Ngoc-Hieu Nguyen · Parshin Shojaee · Chinh Duc La · Duy Minh Ho Nguyen · Rui Zhang · +3 more

University of Notre Dame · The Pennsylvania State University · Virginia Tech · VinUniversity · University of Stuttgart

Research Digest·4 November 2026·2 min read

Illustration: Language models learn harder problems but can forget easier ones

In one line

RLVR training primarily improves easy problem performance while expanding hard problem reasoning causes forgetting of easy skills.

Before you read this

Three ideas this paper builds on

reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.145 papers →catastrophic forgettingLoss of previously learned knowledge when a model is fine-tuned on new tasks.7 papers →group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.168 papers →

Read it as

, or RLVR, trains language models by giving a yes/no signal for correct answers. A central debate asks whether this actually teaches new reasoning strategies or just polishes skills the model already has.

This paper explains the conflict by identifying "primacy bias": a tendency for the model to over-rely on its initial training and stay rigid, or "lazy". Experiments confirm RLVR mostly improves performance on easy problems while failing to adapt to harder ones.

Hard problems are the only ones that genuinely expand reasoning. However, training on them causes of easy skills. This reconciles prior findings where RLVR helped more on unfamiliar domains than on well-practiced math tasks.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

primacy bias reinforcement learning with verifiable rewards lazy training regime catastrophic forgetting group relative policy optimization neural tangent kernel