News without the noise
Reinforcement learning methods, RL for language models and agents, reward modelling, preference optimisation
207 articles · page 3 of 5
Ji and colleagues give geometric and temporal interpretations of empowerment, an information-theoretic measure of how strongly an agent can control its future state. They show that maximizing potential empowerment directs agents toward central, broadly accessible states in tabular Markov decision processes, but they also construct settings where high empowerment provides little ability to adapt to new rewards.
Martinez et al. address a basic ambiguity in independent multi-agent reinforcement learning: a poor reward does not reveal whether the agent chose badly, a teammate responded incompatibly, or an opponent acted effectively. Their CASTLE framework compares predicted consequences for alternative actions and improved final scores over fully decentralized baselines on three benchmark tasks.
Zhang et al. present LoGRA, a reinforcement-learning post-training method that represents selected weight gradients as compact low-rank sketches instead of storing dense gradients and optimizer states. Across reasoning experiments, the authors report memory reductions of up to 45.7 percent without lower task performance, and sustained training of a 27B-parameter model under a hardware allocation where dense Adam could not fit.
Macina, Kapur and Sachan redesign reinforcement learning for language-model tutors around what a simulated student can solve after a tutoring dialogue. Their central result is that measuring learning on a new problem improves transfer, but does not by itself stop answer-giving: explicit binary constraints are still needed to distinguish teaching from telling.
On Tai Tang studies online reinforcement learning for multimodal deep-research agents at unprecedented scale: 128k context and 75+ tool-interaction turns. Directly extending conventional RL to this regime proves inefficient due to weak early exploration and premature entropy collapse. The authors propose Long-MDR, a training recipe combining On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue, which stabilizes learning and yields a 9B-parameter agent that ranks first on five of six benchmarks at a 50-turn evaluation budget.
The authors propose RollVerify, a lightweight framework that augments partial-rollout reinforcement learning for large language models. By using a new off-policy shift metric (OPS) to detect stale trajectory segments, RollVerify performs sequence- and token-level verification to truncate and regenerate off-policy suffixes. In experiments on mathematical reasoning, it achieves accuracy comparable to fully on-policy training while reducing GPU training cost from 98.1 to 57.5 GPU days.
Yang and colleagues introduce Bag-of-Tokens Group Relative Policy Optimization, a modification of GRPO that uses token-level process rewards without learning a value network. Across React code generation and AIME mathematics, the method reached target performance in fewer training steps than GRPO and several variants, although most experiments used only one random seed.
The authors analyze why self-distillation in reinforcement learning can fail even when the teacher performs well, and derive a condition for the teacher's update to improve the student. They propose Joint On-Policy Learning and Teaching (JOLT), which jointly trains a single policy as a privileged teacher (with KL regularization toward the student) and an unprivileged student (via dense on-policy distillation). Across math reasoning, code generation, tool use, and terminal tasks, JOLT+ outperforms GRPO by 4.2 to 9.9 percentage points and uses substantially fewer tokens on MATH.
Huang and colleagues examine how intermediate retrieval feedback should be incorporated when training search agents with reinforcement learning. Across seven question-answering benchmarks, assigning evidence credit directly to the search action that produced it outperformed training based only on final answers, raising average F1 by 3.09 points and multi-hop F1 by 4.81 points.
Cheng and colleagues studied reinforcement learning across Qwen and Gemma models, comparing performance and sampled solution traces before and after training. They find that RL changes which reasoning strategies a model tends to select, producing cross-domain transfer on some tasks but forgetting on others. This reweighting can also raise one-shot accuracy while reducing coverage over 1,024 attempts.
The authors introduce ExPPO, a lightweight modification to GRPO that reshapes advantages using prompt-relative, length-normalized response surprisal and prompt pass rate. ExPPO preserves verifier polarity while assigning higher credit to rarer correct responses, leading to better exploration. Experiments across three language model backbones show improved pass@K, sustained entropy, and increased correct-mode diversity.
Huang, Xu and Li study why language models trained with reinforcement learning from verifiable rewards can improve initially, then suffer abrupt accuracy declines. They argue that common objectives progressively make distinct reasoning strategies inaccessible, and report that preserving multiple approaches with their Mesh Learning method improves results across five reasoning benchmarks.
The authors propose LMOPD, a multi-teacher distillation method that uses a lexicographic priority order to route each student rollout to the specialist for the highest-priority unmet objective. The chosen specialist's token-level correction is then projected against higher-priority experts to avoid harmful interference. On math benchmarks with 30B models, LMOPD retained over 90% of top-priority gains in a four-expert setting, outperforming baselines.
Kao et al. show that in verifier-free reinforcement learning for LLMs, the posterior probability of a correct answer conditioned on a reasoning trace collapses to a low-variance interval as the trace lengthens, making rewards nearly indistinguishable. This undermines group-relative policy optimization like GRPO, which relies on reward differences across rollouts. The authors propose RLCPR, which improves stability and token efficiency via uncertainty-aware sampling and concentration-aware regularization, outperforming existing baselines by up to 4 percentage points on six benchmarks.
The authors show that a completed reinforcement learning (RL) training history contains structure that can be exploited to construct policies stronger than any checkpoint visited during training. They propose SURGE, which spectrally decomposes the weight updates from a high-accuracy anchor checkpoint and a competitive donor checkpoint, retaining the anchor's dominant component while incorporating complementary information from the donor. The method achieves state-of-the-art results on multiple reasoning benchmarks, exceeding the best native checkpoint without additional training or per-query inference cost.
The authors show that standard RLVR training, while improving single-sample accuracy, increases error correlation among samples, degrading majority vote. They propose adapter thickets—training K independent LoRA adapters on random data shards—which outperform a single trained adapter in all tested settings, maintaining vote quality even at large vote budgets.
Shen and colleagues propose MatrixReward, a reward construction method for reinforcement learning on open-ended tasks where no single correct answer exists. Training Qwen3-8B with these rewards produced an average score of 63.02 across four benchmarks, 1.23 points above the strongest pairwise baseline.
On-policy self-distillation (OPSD) provides dense token-level supervision but suffers from noise and instability. The authors analyze forward and reverse KL divergences and find that teacher reliability degrades on longer incorrect trajectories. They introduce Outcome-Guided OPSD (OG-OPSD), which applies forward KL on correct trajectories and reverse KL on incorrect trajectories only up to a prefix determined by cumulative teacher entropy. OG-OPSD consistently outperforms vanilla OPSD and strong baselines on mathematical reasoning, multimodal reasoning, and out-of-distribution tasks across Qwen3 models from 1.7B to 8B parameters and Qwen3-VL-2B.
Gan and colleagues introduce Reasoning State Propagation (RSP), a process reward model that explicitly connects the validity of successive steps in a reasoning trajectory. By combining limited step-level annotations with more scalable final-outcome labels, RSP improved reasoning search and reinforcement learning over representative process reward baselines.
Li and colleagues address a weakness in generative reward models: choosing the preferred response correctly does not guarantee that the accompanying critique is sound. Their EnGRICH framework uses a training-only MetaCritic to assess critique evidence and correctness, improving reward-model performance across seven benchmarks while leaving the deployed model unchanged.
The authors conducted a large-scale study on preference distillation, comparing reject sources from smaller frozen models versus the student's own responses. They found that across student model sizes from 7B to 72B, rejects from smaller Base models consistently produced stronger student performance on code generation and mathematical reasoning, while requiring less compute. Theoretical analysis and interventions identify task structure and limited reference policy coupling as key properties of useful rejects.
The authors propose Flash-OPD, a method for on-policy distillation that stops each training trajectory individually based on observed teacher-student compatibility events rather than using a fixed rollout length. Across diverse teacher-student settings, it achieves 2.2x to 7.5x speedups over standard on-policy distillation while maintaining or improving accuracy.
Saini and colleagues introduce MEND, a reinforcement learning method for post-training text-to-image flow models without KL penalties, frozen reference models or advantage weighting. In reported experiments, MEND improved several reward benchmarks more quickly than Flow-GRPO, ReFL and DiffusionNFT while explicitly limiting how far individual samples move.
The authors develop an information-theoretic model showing that post-training performance scales as Θ(√min(log M, K)), where M is the number of preference comparisons and K is the divergence budget. They validate this scaling law with extensive experiments using a 70B gold reward model and smaller proxy models, achieving R² between 97% and 99%. The result provides a principled way to set divergence limits based on available preference data, clarifying the trade-offs in reward optimization.
The authors formalize 'predictive sufficiency' to measure what access to the true latent context adds over history alone for next-step prediction in model-based reinforcement learning. They find that in some settings the agent's behavior implicitly identifies the context, leaving no headroom for a context encoder, while in others residual headroom persists but learned state representations recover it only partially.
Yang and colleagues treat reinforcement learning for joint audio-video generation as a changing credit-assignment problem: the model must determine both where each reward should update the network and how strongly that reward should count. Their Adaptive Reward Routing method tracks cross-modal influence and reward conflicts during training, producing consistent improvements over fixed-routing and fixed-weight baselines in modality quality, semantic alignment and audio-video synchronization.
Fan et al. present TIPS, a reinforcement-learning method that trains a model to examine a reasoning trace, label its steps and judge its final outcome, while rewarding only the correctness of the outcome judgment. Using 3,200 outcome-labeled trajectories, their best 4-billion-parameter model reached 85.2 F1 on ProcessBench, although it remained behind o1-mini.
The authors introduce MFPG-PPO, an extension of the multi-fidelity policy gradient framework to actor-critic methods. By redesigning sampling, advantage estimation, and control variate construction, they preserve cross-fidelity correlation and prevent variance inflation. MFPG-PPO improves upon standard PPO trained on high-fidelity data alone across simulated locomotion tasks, matching the performance of PPO with 16x more data on the hardest task, and enables stable learning on a physical Franka arm with only four episodes per update.
The authors propose Dependency-Aware Group Policy Optimization (DepGPO), which constructs command dependency graphs from execution traces and traces backward from the resources inspected by the task verifier. By assigning credit to relevant writes and their supporting reads along these paths, DepGPO redistributes trajectory advantages across steps, leading to improved task performance and training stability on complex terminal tasks.
The authors introduce BVER, a curriculum learning algorithm that grows intermediate start states from the goal and intermediate goals from the initial state distribution, biasing both toward unexplored regions. On three simulated robotic tasks—maze navigation, quadrupedal box climbing, and robot-arm ring-on-peg transfer—BVER learns faster than all compared reference-free curricula. In the box-climbing task, it reached 95% success on a 0.4 meter box in roughly 65% fewer training iterations than the best alternative, and was the only reference-free method to learn climbing a 0.7 meter box.
Osband reframes supervised learning as sequentially allocating a limited update budget across examples. The resulting horizon loss interpolates between cross-entropy early in training and the exact gradient of expected accuracy near the end, improving ImageNet top-1 accuracy by 1.0 to 2.4 percentage points under the tested flat-learning-rate setup.
Bertil Braun trained a chess self-play system from random initialization on one eight-GPU node for 2.5 days, producing a 6.32-million-parameter model that measured 3,251 benchmark Elo at 100,000 searches per move against a fixed-node Stockfish 13 ladder. The result integrates search allocation, prioritized replay, progressive model sizing, and throughput engineering into a complete learning loop.
The authors identify a 'self-confirming superposition trap' in reinforcement learning, where a lower-return policy persists because every globally optimal code assigns overlapping directions to features rarely co-occurring under the current policy, but which interfere under a better alternative action. They theoretically characterize conditions for such traps in a two-step model and bound the distortion required to reverse action rankings, showing that fitting weights from visitation can sustain the trap despite exact returns. Experiments with neural PPO, MiniGrid, DMControl, and DreamerV3 on Crafter confirm the feedback and demonstrate that interventions preserving access to neglected states or protecting replay weights can improve control.
The authors present a general framework for last-iterate convergence in online constrained reinforcement learning with structured function approximation. They separate contraction of regularised primal-dual dynamics from actor approximation and policy evaluation errors, enabling model-free on- and off-policy learning. The framework achieves representation-dependent complexity for linear CMDPs and general function approximation, with improved target-accuracy dependence over prior work.
Wang and colleagues introduce CATCH, a controlled environment for studying reward hacking in coding reinforcement learning with verifiable rewards. By comparing outcomes from a deliberately vulnerable evaluator with an independent execution-based audit, the authors identify when models earn rewards without correctly solving the task. Their experiments show that hacking depends on both the model's starting behavior and the difficulty of obtaining legitimate rewards.
The authors identify a teacher-side distribution shift in on-policy distillation, where the teacher supervises prefixes from the student's trajectory distribution rather than its own. They introduce SCOUT, which adapts the teacher via outcome-based RL on student prefixes, improving distillation accuracy on math and code generation tasks.
Lee and colleagues compare learning from individual rewards with learning from reward differences between actions shown in the same context. Their finite-sample analysis shows that pairwise regression is advantageous when absolute rewards contain substantial context-specific disturbances, while pointwise regression can be better when the reward model is nearly correct and its lower variance matters more.
Jerge and Jana present CANOPY, a multi-fidelity bandit algorithm for searching tree-structured candidate spaces without assuming that nearby branches always have similar value. The authors prove that its extra cost scales additively with the number of smoothness violations, and report matched-budget gains across model routing, candidate identification, test-time search, prompt trimming, and prefix caching.
Wang et al. introduce DRM, a Diffusion Reward Model that recasts reward modeling as conditional density estimation over p(r|x,y) instead of point prediction. A lightweight Diffusion Transformer denoises Gaussian noise into a reward vector with no parametric family imposed, and N samples at inference form an empirical reward distribution that can be aggregated into a scalar, variance, or quantiles. Across five benchmarks, DRM matches or surpasses baselines under matched data and backbone, and recovers multimodal structure where conventional heads collapse to a single point.
A unified theoretical framework reveals that three prominent regression-based diffusion reinforcement learning methods—DiffusionNFT, FlowAWR, and RAM—are each solutions to a divergence-constrained reward-maximization problem, differing only in the convex generator defining the constraint. The authors identify approximations in prior work and propose DiffusionRFT, which uses an exact sparsemax projection, leading to faster convergence, more stable training, and top performance.
The authors propose Strategy-Diverse Reinforcement Learning (SDRL) to train open-source LLMs as adaptive meta-solvers for industrial-scale optimization. They show that combining solver-integrated reasoning, exact algorithms, and heuristic search with a hierarchical diversity reward allows the model to select the best strategy for each problem, outperforming DeepSeek-V4-Pro and GPT-5.5 on average across benchmarks and on industrial-scale tasks.
Zhang et al. introduce Trajectory-to-Step Policy Optimization (T2SPO), a method that extracts step-level supervision from historical successful trajectories to guide reinforcement learning of LLM agents. By pairing each state in a successful trajectory with its remaining number of steps, they train a frozen TabPFN regressor to predict progress in new rollouts; the change in predicted distance between consecutive states provides auxiliary credit for each action. Experiments on ALFWorld and WebShop with 1.5B and 7B models show that T2SPO consistently improves task success over GRPO, a baseline that only uses outcome rewards.
The authors propose Density-Aware Reward Aggregation (DARA), a method that calibrates reward contributions in multi-reward reinforcement learning for LLMs. DARA dynamically adjusts weights based on the frequency of active learning signals, leading to faster convergence on format and length compliance goals. In experiments, DARA achieved up to 26% fewer training steps on tool calling and up to 65% fewer steps on math reasoning compared to the baseline GDPO.
Nisioti and colleagues test whether neuroevolution, which optimizes populations of neural networks through weight perturbation and selection, can address the tension between learning new tasks and retaining old ones. They report that evolution strategies produced the most consistent stability-plasticity balance, while genetic algorithms adapted faster but forgot more.
The authors compare 14 base and post-trained model pairs on three interactive agent benchmarks, using a lightweight harness to let base language models call tools and act across multiple turns. Post-training generally raises pass@1, the chance of succeeding once, but often reduces pass@K, the chance that at least one of several attempts succeeds.
Zhu and colleagues address a weakness in agentic reinforcement learning: a final success or failure score does not reveal which action caused the outcome, and provides no comparative signal when every rollout receives the same result. Their FAULT method diagnoses intermediate errors, estimates their relative costs from observed outcomes, and redistributes terminal credit across individual steps.
Ye and colleagues test whether reinforcement learning with verifiable rewards improves visual perception or merely teaches models to exploit nonvisual shortcuts. They find that benchmark gains can occur without visual training data, while tasks designed to make looking necessary substantially improve target discovery and transfer to independently constructed grounding evaluations.
Kong et al. demonstrate that a conditional energy-based model trained purely on random exploration trajectories, without action or reward labels, can support long-range planning. The model learns temporal log-density ratios at multiple horizons, which a separate planner queries to evaluate candidate actions via a one-step dynamics model. The method achieves long-range maze planning from states and images, and extends to egocentric navigation and manipulation planning from suboptimal data.