advantage estimation
Reinforcement learning gives a reward only at the end of a chain of decisions. Advantage estimation asks which earlier actions deserve credit, comparing an action's outcome with what usually happens. In language models, each token or turn is an action, so this decides whether a reasoning step helped or hurt.
These papers fix flaws in standard estimators. Naive methods give every step equal credit, compare responses within random groups, or ignore that trajectories reach different states at the same step. Refinements estimate advantages at the level of state, turn, session, or token, using graphs, contrastive rollouts, multiple group contexts, bounded residuals, or twin critics. The shared result is more faithful credit assignment: better performance and training stability without costly generation.
- Papers
- 49
- Released code
- 3
- First seen
- Aug 2026
- Latest
- Nov 2026
45 papers in the last two months, against 4 in the two before.
Start here
Three papers to read first, in the order they came out- 1Aligning auxiliary rewards with task success improves multi-task language model training
CONCUR projects gated auxiliary rewards onto scores that agree with the primary reward, improving multi-task LLM reinforcement learning.
Oct 2026 - 2AI agents complete more multistep tasks by reusing related experience
Beyond-Local Policy Optimization improves long-horizon LLM agent training by reusing experience across tasks to estimate step-level credit without a critic.
Nov 2026 - 3Budgeting rewards per prompt helps language models train past reasoning plateaus
Reward budgeting prevents premature convergence in reinforcement learning for LLM reasoning by assigning a fixed reward budget per prompt.
Nov 2026
How fast it is moving
Often studied with
credit assignmentDetermining which actions or steps in a sequence contributed to a final outcome, often for reinforcement learning.34 togethergroup relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.27 togetherreinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.17 togetherWho is working on it
The papers
Most central to this idea first, not most recent.
- Chinese Techcs.LG
Budgeting rewards per prompt helps language models train past reasoning plateaus
Reward budgeting prevents premature convergence in reinforcement learning for LLM reasoning by assigning a fixed reward budget per prompt.
University of Cambridge, Qwen Business Unit, Alibaba · Nov 2026
- Big Techcs.CL
Aligning auxiliary rewards with task success improves multi-task language model training
CONCUR projects gated auxiliary rewards onto scores that agree with the primary reward, improving multi-task LLM reinforcement learning.
CMU, Google DeepMind · Oct 2026
- Chinese Techcs.AI
AI agents complete more multistep tasks by reusing related experience
Beyond-Local Policy Optimization improves long-horizon LLM agent training by reusing experience across tasks to estimate step-level credit without a critic.
Beihang University, Alibaba Group · Nov 2026
- Independentcs.AI
AI agents improve at search and automation when trained around complete replies
Multiscale Policy Optimization improves long-horizon LLM agents by aligning credit assignment and policy updates at the turn level.
Oct 2026
- Chinese Techcs.CL
Residual advantage uses teacher-student disagreement as bounded step reward
Residual Advantage improves RLVR by using a bounded, centered teacher-student residual as a step-level credit allocation signal.
Harbin Engineering University, Tencent · Oct 2026
- Chinese Techcs.CL
New RL method trains AI agents to handle multiple sessions effectively
Mosaic uses hierarchical advantage optimization to outperform broadcasting in multi-session agentic RL on BrowseComp by 10.1% and with fewer sessions.
HKUST (Guangzhou), Huawei · Oct 2026
- Chinese Techcs.LG
Averaging rollout contexts improves label-free language model training
Self-rewarding LLMs become more reliable when reward estimates are marginalized over random group contexts, improving zero-label self-evolution.
Shanghai Jiao Tong University, Tencent · Sept 2026
- Chinese Techcode
Trajectory graphs improve step-level credit assignment in agentic reinforcement learning
GRAFT uses trajectory graphs to estimate step-level advantages more faithfully than GRPO in multi-turn agentic tasks.
Shanghai Jiao Tong University, Tencent AI Platform Department · Sept 2026
- Big Techcs.AI
Online reinforcement learning on live iOS apps yields UI agents with 7x better cross-platform task success
Online RL with decoupled planning and grounding and dense LLM rewards lets small agents match frontier UI navigation models.
Apple · Oct 2026
- Academiccs.LG
Sparse probes improve credit assignment for language model reasoning
ProbePO sparsely evaluates uncertain reasoning prefixes and redistributes GRPO credit, improving accuracy while cutting training time versus dense probing.
Independent Researcher, University of Illinois Urbana-Champaign · Oct 2026
- Research Labcs.AI
Offline trajectories can train agents without environment interaction while limiting forgetting
SAPD improves LLM agent training by distilling offline trajectories with relative step advantages, requiring no environment interaction.
Meituan, University of Chinese Academy of Sciences · Oct 2026
- Chinese Tech
Segment-specific rewards improve reinforcement learning for tool-calling agents
Decoupling advantage estimation per segment resolves cross-segment credit misattribution in tool-calling RL.
Peking University, Shenzhen University · Sept 2026
- Big Techcs.CL
Credit assignment that prunes redundant steps improves reasoning accuracy and efficiency
RECAP assigns credit based on downstream dependency and correct-answer progress, improving accuracy and reducing reasoning tokens in large models.
George Mason University, Amazon, Inc · Sept 2026
- Top Universitycs.LG
Shared rollout states improve credit assignment for long-horizon agents
Cross-Rollout Bellman Closure improves long-horizon agentic RL by merging rollouts and computing a Bellman fixed point for step-level credit.
Beihang University, Zhongguancun Academy · Sept 2026
- Research Labcs.LG
Density-aware reweighting accelerates multi-reward RL for LLMs
Inverse-square-root weighting by active-group density strengthens sparse reward signals and speeds multi-objective LLM reinforcement learning while preserving competitive final performance.
University of Chinese Academy of Sciences, University of Minnesota Twin Cities · Oct 2026
- Big Techcs.LG
Training on failed patches helps coding agents recover more effectively
Training software engineering agents on recovery attempts conditioned on failed patches and verifier feedback improves resolution rates over standard GRPO.
NVIDIA, Tsinghua University · Oct 2026
- Chinese Techcs.AI
Twin critics calibrate token-level advantages from a single rollout, boosting RL training efficiency
Twin critics calibrate token-level advantages in reinforcement mid-training, improving mean benchmark performance by 7.8% and cutting training-step time by up to 63.4% versus critic-free methods.
Tsinghua University, Tencent · Sept 2026
- Chinese Techcs.LG
Self-privileged critic improves value estimation for RLVR
A self-privileged critic that uses verified same-prompt rollouts as contrastive evidence improves value estimation in RL for LLMs.
Peking University, Tencent · Sept 2026
- Top Universitycs.AI
AI agents picking from a menu succeed more often with less training
Reinforcement learning directly over compiled action menus eliminates variance from natural-language generation and invalid outputs.
University of Illinois Urbana-Champaign, National University of Singapore · Nov 2026
- Chinese Techcs.LG
Progress-aware advantage shaping resolves uniform credit assignment in agentic RL
Progress-aware advantage shaping improves critic-free LLM-agent training by withholding rewards from invalid actions and concentrating penalties on deviations that cause downstream stagnation.
Zhejiang University, Westlake University · Oct 2026
- Big Techcs.AIcode
Dynamic rubrics improve credit assignment for long-horizon agent training
DRACO uses dynamic rubrics to assign per-step credit in GRPO, improving long-horizon agent training without ground-truth rewards.
Carnegie Mellon University, IBM Research · Sept 2026
- Big Techcs.AI
GRPO Can Reward Lucky Guesses as If They Were Reasoning
GRPO's advantage estimator rewards guessed correct answers, and SIGNBALANCE fixes this by decoupling magnitude from group statistics.
Rochester Institute of Technology, Adobe Research · Sept 2026
- Big Techcs.LG
Language models can learn even when every sampled answer is wrong
Steering vectors added to hidden states during training improve LLM post-training, especially when distilling from thinking teachers.
Arizona State University, Amazon · Nov 2026
- Chinese Techcs.LG
Language models solve more problems when training accounts for confidence
SCARL reshapes advantages to align policy confidence with outcomes, improving reasoning accuracy and exploration in language model training.
Baidu, Nanjing University · Oct 2026
- Chinese Techcs.CV
Training world models for policy improvement beats optimizing pixel accuracy
Training world models to maximize the policy's downstream return improves VLA policies even when next-frame pixel MSE is 17% worse.
Hunyuan, Tencent, Peking University · Oct 2026
- Top Universitycs.CL
RL gains in no-think mode stem from reactivating existing reasoning
Reinforcement learning gains in no-think mode of hybrid LLMs largely come from reactivating the existing think mode reasoning.
King Abdullah University of Science and Technology, Shanghai Jiao Tong University · Oct 2026
- Academiccs.LG
Self-Credit rescales self-assigned credit to improve long-horizon agentic RL
Self-Credit improves long-horizon agentic RL by using model-cited critical turns rescaled across rollouts for dense credit assignment.
Meta, Duke University · Oct 2026
- Top Universitycs.CV
Training video models across counterfactual views improves evidence-grounded answers
Optimizing video MLLMs on behavior packs across counterfactual views improves accuracy, temporal reasoning, and abstention over single-response post-training.
Peking University, The University of Melbourne · Oct 2026
- Big Techcs.LG
Context-aware credit assignment improves language agents on long-horizon tasks
CCPO improves RL for LLM agents by weighting observation-matched peers by context and combining historical and future credit.
Stanford University, Texas A&M University · Oct 2026
- Chinese Techcs.LG
Criterion-aware credit assignment improves rubric-based reinforcement learning for language models
CoCA improves rubric-based reinforcement learning by 4.2% on medicine and science benchmarks through criterion-aware credit assignment.
Zhejiang University, Ant Group · Oct 2026
- Top Universitycs.LG
Multi-Fidelity Control Variates Stabilize Data-Scarce Reinforcement Learning
Multi-fidelity policy gradients stabilize data-scarce on-policy RL by using biased low-fidelity data as a variance-reducing control variate.
University of Texas at Austin, University of British Columbia · Oct 2026
- Chinese Techcs.CL
Teacher-guided training improves specialization and coordination in multi-agent models
MAS-OPD achieves the highest mean score on code and math benchmarks by using on-policy distillation with role-advantage specialization and privileged attribution.
University of Science and Technology of China, Tencent · Sept 2026
- Chinese Techcs.LG
Elastic GPU scheduling makes reinforcement learning faster for marathon agent tasks
QwenGyre enables online reinforcement learning for xlong-horizon LLM agents by elastic GPU reallocation and trajectory deduplication.
Alibaba Token Hub, Alibaba Group, University of Science and Technology of China · Sept 2026
- Academiccs.LG
Anchored hypergraphs provide stable credit assignment for multi-agent teams
HySTAR uses a stable anchored hypergraph to separate credit assignment from adaptive representations, improving MARL performance across multiple benchmarks.
University of Electronic Science and Technology of China, Independent Researcher · Sept 2026
- Independentcs.LGcode
Chunk-level critics reduce compute for reinforcement learning on long proofs
AC2 exceeds GRPO's peak validation score of 18.5% using 2.5x fewer decoding FLOPs.
Oct 2026
- Chinese Techcs.LG
Three diffusion RL methods unified as divergence-constrained reward maximizers
DiffusionRFT uses exact sparsemax projection to converge faster, train more stably, and achieve top performance in regression-based diffusion RL.
Tencent · Oct 2026
- Chinese Techcs.LG
Separating search and answer credit improves accuracy and evidence grounding
Separating search and answer credit improves both exact-match accuracy and evidence grounding in reinforcement-trained multi-turn search agents.
South China University of Technology, FiT, Tencent · Oct 2026
- Top Universitycs.CL
Interventional step credit prunes redundant chain-of-thought while preserving accuracy
Interventional step credit and IGPO reduce reasoning chain length by up to 62% while improving accuracy.
Harbin Institute of Technology, Shenzhen, University of Aberdeen · Oct 2026
- Research Labcs.AI
TRACE reduces multi-turn attack success while largely preserving model utility
TRACE reduces multi-turn attack success rates across five models and seven attacks, with utility drop at most 1.23 points.
PRADA Lab, King Abdullah University of Science and Technology · Oct 2026
- Research Labcs.LG
Muon optimizer boosts training success for long-horizon repository agents
Muon optimizer combined with token-averaged loss and auxiliary penalties improves training success in long-horizon RL for repository-level issue resolution.
Renmin University of China, Duke University · Oct 2026
- Big Techcs.CL
Verifying pivotal agent decisions improves reinforcement learning credit assignment
Distributing credit only to pivotal decision segments improves large language model agent reinforcement learning outcomes.
University of California, Davis, Microsoft · Sept 2026
- Top Universitycs.AI
Privileged supervision improves action-level credit for language-model agents
TASPO converts privileged process supervision into outcome-grounded action credit, improving agentic policy optimization by 10.6% over GRPO across three benchmarks.
Zhejiang University · Sept 2026
- Big Techcs.LG
Frozen LLM critics sharpen credit assignment for long-horizon agents
A frozen LLM value critic, policy-aware via in-context conditioning, improves step-level credit assignment in agentic RL, beating GRPO and PPO.
Apple · Oct 2026
- Chinese Techcs.LG
Adaptive test weighting gives software agents denser, more useful rewards
CodeLift dynamically calibrates test pass-rate rewards to create an adaptive curriculum for training SWE agents.
The Chinese University of Hong Kong, Shenzhen, Tencent · Oct 2026
- Chinese Techcs.LG
Contrastive branch training improves credit assignment for tool-using language models
Contrastive Branch Policy Optimization improves tool-integrated agent training by separating branch selection from credit assignment and using entropy screening with contrastive branch values.
Alibaba Group, Harbin Institute of Technology · Aug 2026
- Big Techcs.CR
Co-evolving code attackers and defenders improves executable vulnerability detection
Co-training attacker and defender populations with execution-verified rewards outperforms static training for code security.
AWS AI · Oct 2026
- Big Techcs.CL
Low-rank gradient sketches cut memory costs in language-model reinforcement learning
LoGRA reduces LLM reinforcement learning memory by up to 45.7% using low-rank gradient sketches and KL step control.
NVIDIA · Oct 2026
- Chinese Techcs.LG
Outcome-guided self-distillation improves LLM reasoning by adapting supervision
Outcome-guided divergence selection and prefix truncation improve on-policy self-distillation for LLM reasoning, consistently beating vanilla OPSD across model scales.
Southeast University, Huawei Noah’s Ark Lab · Oct 2026
- Academiccs.CV
Video RL improves by assigning credit to individual visual tokens
TVRL uses gradients from a frozen vision-language model to allocate token-level credit, outperforming scalar-reward GRPO on video generation.
Northeastern University · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.