often use repeated sampling to solve hard reasoning problems, but this produces many near-duplicate attempts. The authors propose training a small concept generator with reinforcement learning to output diverse problem-solving strategies, which then condition a larger frozen answer generator. This approach improves pass@128 from 19.0% to 39.2% on a hard held-out math dataset, outperforms concepts from a larger untuned model, and transfers to other model families.
Training a concept generator improves LLM reasoning over repeated sampling.
A small model trained with reinforcement learning learns to propose diverse problem-solving concepts, boosting a frozen large model's accuracy on hard math problems.
Big Tech
Ismail Labiad · Matthieu Kowalski · Marc Schoenauer · Rémi Munos · Julia Kempe
Meta FAIR · Université Paris-Saclay · Inria · NYU Courant Institute
Research Digest··2 min read
In one line
Training a small concept generator with reinforcement learning nearly doubles a larger frozen model's reasoning pass@k on hard math problems.
Before you read this
Three ideas this paper builds ontest-time scalingAllocating extra computation during inference to improve output quality.117 papers →large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.329 papers →reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.163 papers →
Read it as
The authors propose a method where a small concept generator is trained using reinforcement learning to produce high-level concepts (e.g., hints, strategies) that maximize the downstream success of a frozen, larger answer generator.
The authors first strengthen an existing concept-guided sampling pipeline by generating all concepts in a single autoregressive trajectory (rather than iteratively), increasing diversity.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.