Reward Collapse in Reasoning RL
Investigating how reinforcement learning signals for reasoning tasks degrade with long traces, and methods to maintain reward discriminability.
10 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
How this thread developed
Oct 2026
Random rewards reveal three stages of LLM capability development
explains spurious-reward paradox via reachability and three stages of LLM capability development.
2 further papers
Oct 2026 · Amazon AGI, Virginia Tech
Token-level rewards make critic-free reasoning training faster and more accurate
Introduces token-level process rewards in GRPO to improve training accuracy and speed, directly addressing reward discriminability.
1 further paper
Oct 2026 · Harbin Engineering University, Tencent
Residual advantage uses teacher-student disagreement as bounded step reward
Proposes Residual Advantage using teacher-student disagreement as a bounded step reward to maintain discriminability.
Oct 2026 · Virginia Tech, Cisco
Co-evolving visual tasks keeps model training near its learning frontier
VICO co-evolves visual tasks to keep model training near its learning frontier, preventing reward collapse.
1 further paper
Oct 2026 · Institute for AI Industry Research (AIR), Tsinghua University, SIA-Lab of Tsinghua AIR and ByteDance Seed
Divergence geometry and estimation jointly improve language-model policy optimization
Proposes a generalized trust-region formulation with matched gradient estimators to mitigate reward degradation in long-trace LLM RL.
Oct 2026 · King Abdullah University of Science and Technology, Shanghai Jiao Tong University
RL gains in no-think mode stem from reactivating existing reasoning
Analyzes the mechanism behind RL gains in no-think mode, showing reactivation of existing reasoning patterns.
6 of 10 papers shown