News without the noise
Reinforcement learning methods, RL for language models and agents, reward modelling, preference optimisation
207 articles · page 5 of 5
Wang et al. show that Group Relative Policy Optimization (GRPO) can assign large advantages to rollouts that reach correct answers by guessing rather than reasoning. Their alternative, SignBalance, removes dependence on the reward composition of each rollout group and reportedly improves bounded-answer mathematics and search-agent performance.
Gandhi et al. introduce DRACO, a reinforcement-learning method for training agents when task success cannot be checked programmatically. On AppWorld and the out-of-domain τ-bench, its dynamically generated rubrics and step-level credit redistribution outperformed the base model and several sparse-reward or rubric-based alternatives.
Zhou et al. study how reinforcement-learning credit should be distributed across multi-turn agent trajectories when only the final outcome is verified. They find that the decisive factor is verifier information density—the share of prerequisite steps whose correctness is observable—and that broad coverage outperforms targeted credit in the low-information regime typical of terminal-state evaluation.
The authors introduce SAGE, a reinforcement-learning framework that uses an expensive vision-language model as a temporary teacher rather than a permanent policy. Across sparse-reward visual reasoning and navigation tasks, selectively requested guidance improved learning in several environments, sometimes producing policies that surpassed the teacher and required no model calls at evaluation time.
Yang et al. address a credit-assignment problem in reinforcement learning for agents: a verified task outcome is typically applied uniformly to every decision in a long interaction. Their TASPO method uses privileged information from successful experiences to weight individual actions, improving performance over GRPO by 10.6% across three agentic benchmarks and transferring better to unseen tasks.
Meng et al. introduce HARTS, a training system that compresses shared prefixes in agentic reinforcement-learning rollout trees and schedules the remaining work across microbatches and data-parallel replicas. On rollouts generated from SWE-bench tasks, it delivered 4.81–4.87× speedups for forward, backward, and gradient computation while maintaining numerical and early-training behavior close to trajectory-wise training.
Wang et al. introduce Contrastive Branch Policy Optimization (CBPO), a reinforcement-learning method that separates where to sample alternative continuations from how to assign credit to tokens. Across ten mathematical-reasoning and knowledge-search benchmarks, the authors report the highest macro-average accuracy among the evaluated policy-optimization and branch-based methods at two model scales.
Huang et al. replace per-query iterative action optimization with a learned trajectory prior conditioned on current and goal images. Across four goal-conditioned pixel-control benchmarks, the authors report higher success rates and an order-of-magnitude reduction in planning time.
Han et al. analyze preference optimization for flow-matching generative models, showing that reward-driven updates can push outputs away from the pretrained data manifold when the induced displacement has a component normal to that manifold. Their proposed ThermoDPO-weighted objective improves both a toy benchmark and text-rendering metrics for Stable Diffusion 3.5 Medium.
The authors introduce Skill Self-Play (Skill-SP), a framework where LLMs generate and solve challenging tasks by dynamically routing through learned skills. In a reinforcement learning loop, the system co-evolves a proposer, solver, and skill controller, showing consistent improvements on tool-use and reasoning benchmarks, even turning around initially misaligned models.
Hu et al. introduce Molt, a PyTorch-native training framework for agentic reinforcement learning that prioritizes researcher readability and end-to-end algorithm traceability. In benchmarked asynchronous training runs, Molt achieved statistically comparable performance to a state-of-the-art Megatron-based stack, demonstrating that leanness does not compromise scalability.
The authors propose TRACE, a credit-assignment method for long-horizon agentic RL that derives per-action rewards from the log-probability of the gold answer under a frozen reference model, using temporal-difference learning. On the closed-web BrowseComp-Plus benchmark, TRACE raises Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6, and enables pure RL training without cold-start supervised fine-tuning or live-web data.
Li et al. propose CompactionRL, a reinforcement learning method that trains LLM agents to perform long-horizon tasks by compressing interaction histories. Applied to open models, it yields consistent gains on code agent benchmarks, with GLM-4.5-Air reaching 66.8% Pass@1 on SWE-bench Verified.
The authors present PROVE, a framework for reinforcement learning (RL) on multi-step tool use that overcomes three key obstacles: lack of realistic execution environments, ungrounded synthetic queries, and reward-induced verbosity. Training four different LLMs with PROVE yields significant gains on three multi-turn tool-use benchmarks, with improvements up to +10.2 points on BFCL Multi-Turn.
The authors introduce EvoTrainer, a framework that treats the training harness itself as an evolvable component in LLM reinforcement learning. By co-evolving policies and harness heuristics through empirical feedback, EvoTrainer matches or exceeds human-engineered RL baselines across three domains, with the largest gains in long-horizon agentic software engineering.