Reinforcement learning

Balancing hidden quality dimensions reduces reward hacking in rubric-trained models

Xia, Xu and Shi show that rising rubric rewards can conceal declines in specific aspects of response quality, a pattern they call intra-rubric reward divergence. Across chat, science and medical tasks, their dimension-aware training methods produced more stable gains and generally better responses than conventional reward aggregation.

20 Oct·3 min

Adaptive training harness keeps rubric rewards useful as policies improve

Wang et al. address a weakness in reinforcement learning for open-ended generation: rubric-based rewards can stop distinguishing useful responses or reward superficial behavior as the policy changes. Their HarnessRL system diagnoses these failures and modifies the relevant part of the training setup, outperforming adaptive and fixed baselines across two model sizes on a new five-domain benchmark.

20 Oct·2 min

Conditioning actions on returns improves value learning in reinforcement learning

Swerdlow, Dong and Finn reverse the usual value-learning problem: instead of predicting returns directly from state-action pairs, their Generative Value Classifier models how likely an action is under each possible return. Across offline, online and offline-to-online continuous-control tasks, they report stronger performance than leading conventional and distributional value-learning methods.

Language models learn to propose improvements for other language models via RL with a learned world model

The authors propose LM2LM, a framework that trains a language model via reinforcement learning to generate effective proposals for improving other language models. By using a frozen LLM (a research world model) to predict proposal gains from real experimental records, they avoid costly full execution of every proposal. Across eight LM development environments, LM2LM enabled a Qwen3.8-27B model to outperform GPT-6-sol on held-out tasks while requiring 33% lower data-generation cost.

Bayesian belief framework jointly handles two uncertainties in process reward models

The authors propose BayesianPRM, which treats process rewards as random variables and maintains a Bayesian belief over a discretized reward hypothesis space. Two belief updates address the two identified uncertainty sources, improving reward accuracy and test-time scaling on multimodal reasoning benchmarks.

Sequence-level credit assignment improves on-policy distillation of reasoning models

The authors propose GR-OPD, a method that aggregates token-level feedback from a teacher into sequence-level credit for on-policy distillation of large language models. They show theoretically that broadcasting summed feedback yields an unbiased estimate of the reverse-KL gradient under on-policy assumptions, and empirically that it outperforms vanilla OPD in every tested setting, including gains of 10.9 and 11.5 percentage points on two mathematical reasoning benchmarks.

Separating search and answer credit improves accuracy and evidence grounding

Liu and colleagues show that reinforcement learning based only on final-answer correctness can make search agents more accurate while making their answers less grounded in retrieved evidence. Their method, Grounded Credit Assignment (GCA), adds small, verifiable corrections for finding and using evidence, improving both accuracy and the evidence-supported share of correct answers.

19 Oct·3 min

JAX-based Atari environments enable GPU-accelerated training and easy modification

The authors introduce JAXtari, a JAX-native reimplementation of the Arcade Learning Environment that runs entirely on GPU, removing the CPU emulation bottleneck. The framework achieves a 10x speedup over traditional ALE, allowing PPO agents to complete 100 million steps in under an hour. JAXtari also natively provides object-centric observations and a flexible modification system for controlled experiments.

Stochastic attention routing enables efficient RL alignment for diffusion models

The authors propose Routing Policy Optimization (RPO), a method that introduces stochastic exploration into diffusion and flow generators by randomly sampling attention routes during training while preserving the original deterministic sampler at inference. RPO uses a Gaussian velocity likelihood to score perturbations and a controlled two-rollout estimator to compute advantages, achieving better reward-aligned generation quality on text-to-image and text-to-video tasks while reducing training cost by up to 4.2x compared to Flow-GRPO.

Decoupling skills from expert policies improves multi-capability vision-language model training

The authors propose SCOPED, a framework that trains separate capability-specific experts via reinforcement learning, extracts explicit skills from their reasoning trajectories, and uses those skills to condition the experts during on-policy distillation into a unified student vision-language model. Across eight multimodal benchmarks, SCOPED enables a compact 8B model to achieve the highest average score among evaluated models, surpassing both the backbone and mixed-capability RL baselines.

16 Oct·2 min

A few clean labels can fix sample-wise noise in reward model training

The authors propose CleanRM, a reward modeling method that estimates sample-wise noise ratios from a small set of clean preference labels using Wasserstein optimal transport. They prove the resulting training objective is unbiased with respect to the clean risk, and show that it outperforms standard denoising baselines and improves downstream RLHF alignment.

Adversarial rubric repair improves self-training on open-ended expert tasks

Dai et al. address a central obstacle in self-generated training for tasks without objectively checkable answers: models can exploit incomplete grading rubrics rather than produce genuinely useful work. Their Recursive Reward Improvement method adversarially tests and repairs each rubric before training, producing substantial gains on medical benchmarks and some transfer to unseen professional domains.

Calibrated rewards curb answer degradation during verifiable instruction training

Yoon and colleagues show that programmatic instruction-following verifiers can award full credit to responses that satisfy output constraints while barely addressing the underlying request. Training against these verifiers amplified the problem: standard reinforcement learning with verifiable rewards improved constraint scores but sharply reduced judged response quality. Their proposed method, CaLIF, preserved answer quality while still improving constraint satisfaction across five model configurations.

Imagined goals from geometric horizon models boost behavior cloning generalization

Lawson and colleagues propose TD-Aug, a data augmentation method that trains a geometric horizon model with temporal difference flows and uses its samples as imagined goals for goal-conditioned behavior cloning. Across OGBench tasks requiring compositional stitching, this improves success rates by 25 to 39 percentage points over standard BC, without any environment interaction.

16 Oct·2 min

Backpropagating verifier rewards improves reasoning in continuous flow language models

The authors propose CFRL, a post-training framework for continuous flow language models (CLMs) that denoise token embeddings over multiple steps. By training a critic to estimate verifier rewards from intermediate states, they bypass the non-differentiable decoding step and enable pathwise gradient backpropagation. On four program-verifiable reasoning tasks, CFRL reliably improves a 105M-parameter CLM and outperforms baseline methods of the same type.

One-step generative policy matches Boltzmann target for online MaxEnt RL

Luo et al. introduce DriftSpread, a one-step generative policy for online maximum-entropy reinforcement learning. By deriving drift and spread updates from the negative MaxEnt objective, the method trains an actor to sequentially move actions toward higher values and then broaden their distribution, achieving both strong control performance and efficient training on MuJoCo benchmarks.

Experience replay boosts RL training efficiency for large language models by reusing trajectories

The authors propose ReFlow, a system-algorithm co-designed reinforcement learning framework that uses experience replay of recent trajectories to amortize the high cost of rollout generation in LLM post-training. By implementing a decoupled architecture with a dual replay data cycle, elastic resource pool, and adaptive scheduler, ReFlow improves training throughput by up to 2.06× over state-of-the-art synchronous and asynchronous RL systems across single-turn reasoning and multi-turn agentic tasks, while maintaining comparable or better model performance.

Gradient norm ranks attention heads for efficient sparse and continual RL fine-tuning

The authors propose using gradient norm to rank attention heads in language models, enabling sparse reinforcement learning fine-tuning that updates only a few heads. They show that tuning the highest-ranked heads consistently outperforms lower-ranked ones and often matches denser baselines across math, instruction-following, and coding tasks. For continual learning, they partition heads across tasks to reduce interference, improving retention without sacrificing adaptation.

Turning failed reasoning attempts into training signal improves GRPO on hard problems

The authors propose Replay-Selected Hindsight Distillation (RSHD), which lets a reasoning model generate a corrective note after a failed attempt without seeing the answer, re-solve the problem with that note, and keep the replay only if it outperforms the original group. This recovers supervision from groups where every rollout fails, cases where standard GRPO provides no gradient. Across math and science benchmarks, RSHD consistently improves over GRPO, with largest gains on the hardest problems, and adds no inference-time overhead.

Unified platform enables GUI agent training across mobile and computer environments

The authors introduce MAI-Gym, an open-source platform that unifies GUI agent evaluation, data collection, and reinforcement learning (RL) training across mobile, computer, and web environments. Using a shared rollout pipeline and automated task synthesis, they train MAI-Gym-9B, a unified 9-billion-parameter agent that achieves 78.4% on MobileWorld and 76.2% on OSWorld-Verified, surpassing other open GUI agents of similar size. The platform scales to 20,000 parallel environment instances and supports ten established benchmarks.

Equal reward weighting can steer language models toward easier shortcuts

Li and colleagues show that combining a primary reward, such as correctness, with an easier auxiliary reward, such as brevity, can raise the aggregate training objective while reducing task success. Their Trust-Anchored Gating method prevents auxiliary-only outcomes from receiving credit and substantially limits this failure in mathematics and program-synthesis experiments.

RL gains in no-think mode stem from reactivating existing reasoning

The authors trained Qwen3-1.7B, 4B, and 8B models with GRPO in no-think mode on the DAPO-Math-17k dataset. As accuracy improved, responses grew longer and more reflective, and internal activations shifted toward the model's think mode. Causal interventions showed that replacing a low-dimensional think-related activation subspace removed 75% of the RL accuracy gain, establishing that much of the improvement comes from reactivating existing reasoning rather than learning new strategies.

15 Oct·3 min

Muon optimizer boosts training success for long-horizon repository agents

The authors evaluate a reinforcement-learning recipe for repository-level issue resolution that pairs the Muon optimizer with single-rollout asynchronous policy updates, token-averaged loss, truncated importance sampling, and auxiliary penalties. Across two model families, the full recipe achieves higher training success than the SPO baselines.

Hindsight assigns token credit from one verified language-model rollout

Chen and colleagues propose Hindsight-Privileged RLVR, a reinforcement-learning method that extracts token-level credit from a single completed attempt. Across mathematics, code generation and embodied-agent tasks, it reduced rollout requirements and training time relative to eight-sample GRPO, although neither method was consistently more accurate across domains.

Hyperbolic geometry beats extreme network depth in goal-conditioned maze navigation

Klein et al. modify contrastive reinforcement learning by measuring distances between learned representations along hyperbolic geodesics rather than Euclidean straight lines. On hard maze tasks, this geometric bias let comparatively shallow agents represent corridor-following routes more effectively than much deeper Euclidean networks.

Context-aware credit assignment improves language agents on long-horizon tasks

Han et al. introduce Contextual Credit Policy Optimization (CCPO), a reinforcement learning method that assigns step-level credit from sparse terminal rewards without training a separate value model. Across ALFWorld and WebShop, two simulated long-horizon agent benchmarks, CCPO reached up to 97.4% and 79.4% success, respectively, using Qwen2.5-Instruct models. The authors report gains over GRPO of as much as 21.6 and 18.5 percentage points.

Symplectic latent dynamics prevent rollout contraction in visual world models

Yalovetzky, Balestriero and LeCun replace an unconstrained JEPA predictor with a learned, action-conditioned Hamiltonian flow over position-like and momentum-like latent variables. On conservative and actuated visual pendulums, the resulting model preserves latent volume by construction and predicts long-horizon velocity more reliably than the baseline.

15 Oct·3 min

Advantage guidance makes supervised policy training competitive with offline RL

Zhou and Levine present Advantage Guidance, a policy-learning method designed to retain supervised learning’s stability while improving beyond the behavior in an offline dataset. The method trains a policy to respond to estimated action advantages, then uses contrastive guidance at inference to favor high-advantage actions under a constraint that keeps the policy close to the data distribution.

Frozen LLM critics sharpen credit assignment for long-horizon agents

Zhu et al. use a frozen LLM as a critic, a model that estimates the likelihood of future success, to identify productive actions, mistakes and recoveries within completed agent trajectories. Across three benchmarks, their method outperformed GRPO and PPO, with particularly large gains on exploration-heavy tasks where trajectory-level feedback failed.

14 Oct·2 min

Specialized GFlowNet mixtures find more modes without extra oracle calls

Younis and colleagues replace a single Generative Flow Network with multiple independently trained GFlowNets encouraged to specialize in different regions of a compositional search space. Across synthetic landscapes and a genomic design task, their mixture continued discovering modes as the population grew to 64 policies, while conventional data-parallel training saturated.

Routing gradients can isolate reward hacking for removal after training

Ward and colleagues introduce Gradient Routing Adapter Fine Tuning, or GRAFT, for reinforcement learning with misspecified rewards and imperfect behavior monitors. Across toy environments, coding tasks and model sizes from 135 million to 32 billion parameters, they report that routing suspicious updates into sacrificial parameters can preserve task improvements while allowing learned reward-hacking behavior to be removed after training.

Criterion-aware credit assignment improves rubric-based reinforcement learning for language models

The authors propose CoCA, a criterion-aware credit assignment framework for rubric-based reinforcement learning (RL) of large language models. Instead of compressing rubric scores into a scalar reward, CoCA computes signed credits per criterion and distributes them to tokens based on rubric-induced distribution shifts. Across medicine and science benchmarks, CoCA achieves an average gain of +4.2% over standard rubric-based RL.

Generative models as proposal distributions improve long-horizon decision making

The authors propose Generative Trajectory Planning (GTP), a model-based framework for online decision making that uses a learned generative model as a proposal distribution over action sequences. Instead of optimizing sequences from scratch, GTP samples candidates from a diffusion prior and refines them using learned dynamics, reward, and value models, enabling efficient trajectory-level reasoning. On continuous control and long-horizon manipulation benchmarks, GTP matches or exceeds state-of-the-art baselines and remains stable where strong baselines degrade.

14 Oct·2 min

Progress-aware advantage shaping resolves uniform credit assignment in agentic RL

The authors propose ProAS, a plug-and-play framework for critic-free reinforcement learning of LLM agents. By leveraging an Agentic Reference Graph to annotate action validity and task progression, ProAS applies Masking Unreasonable Incentives and Asymmetric Penalty Redistribution to reshape trajectory-level advantages. On ALFWorld and WebShop, ProAS boosts success rates by up to 16.7 points and accelerates training convergence up to 2.44x across multiple model scales and baselines.

Reward models for vision-language AI can be trained without human preference labels

Li et al. build a vision-language reward model with no human preference labels: a linear head on an intermediate layer of a frozen Qwen3-VL backbone, trained to rank web captions above perturbed versions. At 32B it beats GPT-4o by 6.6 points on VL-RewardBench and is level with it on two other benchmarks, at about one thousandth of the compute of fine-tuning. The comparison distribution, not benchmark rank, determines what the reward actually rewards.

Self-judged trajectories improve SWE agent training beyond binary verifier rewards

The authors introduce Behavioral Self-Rewarding (BSR), which reuses the current policy to compare trajectories sharing the same test outcome, producing auxiliary rewards for reinforcement learning with verifiable rewards. On SWE-bench Verified, SWE-bench-Live Lite, and Terminal-Bench 2.1, BSR improves average accuracy for Qwen3-14B and Qwen3-32B over verifier-only RL, without any new environment interactions.

Recursive synthesis turns unsolved problems into useful reinforcement learning data

Lu et al. argue that reinforcement learning with verifiable rewards depends less on dataset size than on whether problems produce a useful mix of successful and failed attempts. Their MagicBeanStalk method converts hard seed problems into new tasks matched to a model and rollout budget, substantially outperforming training on a 78,000-problem math corpus.

Agents keep more options open by seeking states rich in skills

Ji and colleagues connect empowerment, an agent’s capacity to control future outcomes, with unsupervised skill learning. They use skill-learning methods to estimate how many distinct behaviors remain executable from each state, then train a separate policy to reach states with more options. In complex environments, including Crafter, this objective produced behaviors such as collecting resources, using tools and building shelter.

Adaptive curriculum training lets vision-language models solve hard visual Sudoku puzzles

The authors introduce FG-Sudoku, a graded benchmark of visual Sudoku puzzles, and AdaC-GRPO, a curriculum reinforcement learning method that adaptively samples tasks within the model's Zone of Proximal Development. Applied to Qwen2.5-VL-7B, AdaC-GRPO raises accuracy on the hardest 9 by 9 tier from near zero to 0.219, approaching Gemini-2.5-Flash while closing 62.2% of the overall gap to Gemini-2.5-Pro.

Divergence geometry and estimation jointly improve language-model policy optimization

Cai and colleagues recast proximal policy optimization for language models as a choice along two axes: how policy change is measured and how that change is estimated from samples. They derive a generalized trust-region guarantee and report that their best configuration, KL divergence with Atomic estimation, improves average math and coding scores by 50% and 14%, respectively, over the standard formulation used in their experiments.

Trajectory-aware updates stabilize reinforcement learning for longer LLM reasoning

Cai and colleagues identify a weakness in long-horizon LLM reinforcement learning: small policy changes at each token can compound, causing the updated model to encounter states poorly represented by the policy that generated its training data. Their Trajectory-Aware Policy Optimization method explicitly estimates and constrains this accumulated mismatch, outperforming established alternatives across eight reasoning benchmarks.

Intervening During Rollouts Helps Agents Learn Beyond Their Initial Limits

Chen and colleagues introduce Rollout-Time Intervention (RTI), a reinforcement-learning method that turns selected failed agent rollouts into successful training trajectories using evaluation rubrics. Across five benchmarks and two model backbones, the authors report that RTI outperforms standard reinforcement-learning baselines and eventually increases unaided task success, rather than merely improving performance through intervention at test time.

Rewards Barely Drive Learning in Direct In-Context Reinforcement Learning

Kwon et al. test whether language models using stored trajectory-reward pairs actually learn from rewards at inference time. They find that rewards are recognized but exert little influence: most improvement comes from the presence and task relevance of trajectories, resembling ordinary in-context learning more than reinforcement learning.

10 Oct·2 min

Separating visual and language reasoning phases improves vision-language model accuracy

The authors propose SPLIT-RL, a staged reinforcement learning method that trains a vision-language model's visual reasoning (VR) and language reasoning (LR) capabilities in separate phases. By using the VR output as a bottleneck and applying claim-level advantage computation, they achieve accuracy gains of 1.4 to 6.1 points over standard GRPO across five model families, while also improving both VR and LR individually.

10 Oct·2 min

Residual advantage uses teacher-student disagreement as bounded step reward

The authors propose Residual Advantage (RA), which treats the teacher-student probability residual as a bounded one-step reward, subtracts the student's own expected value to form an advantage, and centers it within each response. This allows the teacher to redistribute credit among tokens while the verifier sets the overall response label. Combined with GRPO or REINFORCE++, RA improved Avg@8 by 1.7–3.6 points and Pass@8 by 3.9–6.3 points across three mathematical benchmarks.

10 Oct·3 min

Safety filter separates task and safety learning in robot control

Zhang et al. introduce FAITH, a model-free reinforcement learning framework that decouples safety enforcement from task optimization using a learned safety critic and feedforward filter. The task policy is trained through the filter on task reward alone, and when no safe action exists, the filter selects the action with minimum predicted peak harm. In simulations and on a physical Unitree G1 humanoid, FAITH achieves high safety rates while retaining most of the unfiltered task return.

10 Oct·3 min