News without the noise
Reinforcement learning methods, RL for language models and agents, reward modelling, preference optimisation
207 articles · page 1 of 5
The authors introduce world2vec, a method for learning mechanistic world models from observations and actions without rewards. It faithfully captures dynamics over long prediction horizons with at least an order of magnitude fewer parameters than DreamerV3, TD-MPC2, and LeWM, and achieves state-of-the-art control in continuous environments from pixels.
Reinforcement learning from verifiable rewards (RLVR) struggles on hard problems because unsuccessful rollouts yield no learning signal. The authors show that combining two forms of teacher guidance, trajectory-level intervention (correcting student paths) and token-level supervision (dense feedback on each step), dramatically improves performance. On hard logic games, PIVOT nearly doubles mean avg@8 from 19.5% (best baseline) to 35.6%.
The authors show that the conventional sequence of training stages for AI models, from pretraining through fine-tuning and reinforcement learning, can be expressed as a single maximum-likelihood algorithm. Their unified trainer, which mixes static datasets with model-generated data in one continuous process, improves accuracy and task coverage over staged training at the same compute budget, while preserving broader capabilities.
Ke and colleagues examine a blind spot in reinforcement learning that rewards language models for verifiably correct answers: the update does not account for how confidently the model produced each response. Their confidence-aware adjustment improved 14 of 16 model-and-training combinations and increased the share of problems solved within 32 attempts.
Chai et al. present a two-stage training framework that jointly decides which intermediate states a GUI agent should practice and how far to roll out from them. Stage I performs one-step reinforcement learning at uncertain 'pivot' states selected via action disagreement, the variance among multiple sampled GUI actions. Stage II then progressively expands to suffix and root rollouts as the policy improves. On the MobileGym and CUA-Gym-Hub benchmarks, the method reduces required environment interactions by 56–61% while achieving better task completion rates.
The authors train UI navigation agents via online reinforcement learning directly on live iOS apps. Their decoupled architecture separates planning from grounding, using an LLM judge to provide step-level dense rewards alongside task completion rewards, optimized with PPO and neighbor-conditioned value estimation (GiGPPO). The resulting 8B agent nearly matches Gemini-3.1-Pro on iOS tasks and achieves seven times the success rate of supervised fine-tuning on Android tasks.
The authors propose Reinforced Adversarial Learning (RAL), where a judge is trained to distinguish reference outputs from generator outputs, and the generator is reinforced to maximize the judge's belief it is a reference. Across arithmetic, code, chemistry, and scientific abstract generation, this approach matches or surpasses training with explicit verifiable rewards, even though the reward is never seen. On NIH grant abstracts, RAL produced outputs that human experts and LLM judges found harder to tell from real ones than frontier model outputs.
The authors diagnose why on-policy self-distillation (OPSD) plateaus: the self-teacher, a prompted version of the student, often produces unreliable token-level guidance. They introduce Teacher-Evolution Self-Distillation (TESD), which trains the teacher with rewards based on task correctness and context independence, and adaptively down-weights samples with unreliable teacher guidance. TESD consistently outperforms current OPSD and sequence-level reinforcement learning methods across model sizes and datasets.
Zhang et al. introduce CoCA, a closed-loop framework for rubric-based reinforcement learning in which the scoring criteria evolve along with the language model's capabilities. By contrasting rollouts generated with and without rubric guidance, the method identifies which skills are already acquired, which are elicitable with scaffolding, and which need further training. Across seven benchmarks, it reaches baseline-level performance in roughly half the training steps while improving final quality.
The authors introduce CoHARL, a population-based reinforcement learning framework that co-evolves an LLM agent's policy and its executable harness, the code that controls tool use and feedback. On SWE-rebench, CoHARL achieved a 27.6% mean resolved rate, outperforming harness-only search by 6.4 percentage points and policy-only training by 7.3 points.
Wang et al. introduce Policy-Compatible Reinforcement Learning (PICO), a training method for reasoning problems on which every sampled answer fails. Rather than directly copying expert solutions, it uses expert-guided decoding constrained by the student model’s own probabilities, recovering verified successes that can supply otherwise missing learning signals.
Leng and colleagues study a conflict in reinforcement learning: rewards for properties such as response length can undermine the reward for actually completing a task. Their CONCUR method adjusts auxiliary scores to respect task success, with both variants exceeding GDPO by more than four points on the paper’s benchmark average for each of three models.
Guan et al. present PerceptRL, a training framework that targets visual mistakes in multimodal language models, which process images alongside text. The authors report that combining error-focused question answering with training for precise visual tool use improves average accuracy by 15.7 percentage points over the base model using the same tools.
Ryu and colleagues study a failure mode in reinforcement learning: rewarding models for covering a checklist can encourage longer answers with unnecessary or incorrect material. Their FORK reward also scores whether response content is necessary and factually correct, producing shorter answers that evaluators prefer more often than those trained on coverage alone.
Zheng and colleagues test whether small reinforcement learning experiments can predict the results of much larger training runs. In two procedurally generated environments, their fitted scaling laws forecast performance, resource allocation and hyperparameter choices, although Craftax also reveals local departures from the overall scaling trend.
The authors propose Self-Evolution Policy Optimization (SEPO), a reinforcement learning framework in which a large language model uses ground-truth labels as privileged hints to unlock its own exploration, then treats the resulting context-conditioned predictions as a teacher to distill the new solution into the standard policy. Across scientific, tool-use, and logical reasoning benchmarks, SEPO shows higher convergence ceilings and better training efficiency than strong baselines, including in settings where baselines stagnate or fail entirely.
Ren et al. propose Multiscale Policy Optimization (MPO), a reinforcement-learning method for language-model agents that complete tasks through multiple interactions. They report consistent gains over strong baselines with 4B and 30B-A3B models by organizing learning around complete assistant turns without discarding differences among the tokens within them.
DeCastro et al. introduce Dream2Assist, a model-based reinforcement learning framework that infers a driver's discrete intentions and conditions assistive control on them. In human-subject studies across racing and urban driving, the intent-conditioned policy improved torque alignment and reduced collision rates compared to a no-intent baseline.
Hu and colleagues introduce Info-GRPO, a training method designed to stop reasoning models from narrowing prematurely to a small set of strategies. Across four model backbones, the authors report AIME24 gains over standard GRPO ranging from 1.66% to 4.17%, alongside a comparison reaching the same majority-vote accuracy with eight times fewer samples.
The authors identify a 'Frequency Conflict' in test-time reinforcement learning (TTRL) for tasks that admit multiple valid solutions: generation frequency simultaneously serves correctness estimation (favoring frequent outcomes) and diversity preservation (favoring rare outcomes). They propose MS-TTRL, which decouples these roles by using outcome-level self-verification as an independent correctness signal and reserving frequency for diversity reweighting. Across two benchmarks, four multiple-solution tasks, and three backbone models, MS-TTRL improves the Recovery Rate by 19.13 percentage points on average over the best label-free baseline and approaches the performance of the supervised IPS-GRPO.
The authors investigate why BF16 reinforcement learning for LLMs can collapse despite identical weights, and show that selectively aligning rounding locations between inference and training is enough to stabilize training. Across four Qwen-family models, aligning RoPE, SwiGLU, and residual RMSNorm prevents the collapse seen in the default BF16 baseline.
The authors propose SMM-AC, a regularised RL algorithm extending soft mellowmax to continuous control, and evaluate it against PPO, SAC, and TD3 on MuJoCo benchmarks and real robots (Franka arm, ANYmal-D). They find that regularised methods, particularly SMM-AC, yield higher real-world returns and maintain stronger performance under perturbations to observations, environment, and dynamics, demonstrating that regularisation-induced robustness translates to more reliable real-world deployment.
Che, Ivanovic, and Pavone propose Short-to-Long (S2L), where smaller-discount source value networks supply features to a behavior learner whose discount is unchanged. It improves over matched baselines on Atari, online control, and offline MuJoCo, and ablations confirm the role of each design choice.
The authors studied generative reward models (GenRMs) as online preference providers for policy optimization in creative writing. They swept judge format, judge size, policy size, training duration, and DPO regularization, and evaluated policies under a fixed 32B Gold evaluator. The central finding is an evaluator-rewarder gap: static judging accuracy does not determine downstream optimization quality.
De Kuthy Meurers and colleagues trained a language model to produce better moral arguments using rewards from its own comparisons of competing answers. The model became more decisive and less sensitive to option order, but these improvements did not translate into more reliable moral judgments on external benchmarks.
Li and colleagues introduce GIFT, a training method that selects which teacher signals a reasoning model should follow while learning from correctness rewards. The authors report better average mathematical reasoning scores than eight baselines across four model scales, using gradient geometry to filter guidance before gradually withdrawing it.
Wang and colleagues introduce Trace2World, a system that reconstructs an app’s behavior from recorded interactions before generating a runnable training environment. The authors report that separating behavioral inference from implementation, then checking simulations against the original recordings, improves both workflow fidelity and the transfer of training gains to real devices.
Lee and colleagues introduce Self-Assessment Reinforcement Learning (SARL), a training framework that teaches language-model search agents to recognize gaps in their retrieved evidence. The authors report a 24.3% relative improvement over their strongest baseline, with larger average gains on questions requiring multiple reasoning steps.
Kang et al. train video world models to improve the robot-control policies that learn from their simulated trajectories, rather than simply to predict accurate images. Their DRILL-IMG method improves mean task success over two comparison methods, even though its next-frame pixel prediction error is 17% higher.
The authors present TWICE, a method for credit assignment in agentic RL that generates counterfactual rollouts sharing the same Gumbel noise as the factual episode. This coupling isolates the effect of each decision, reducing variance and cutting the number of rollouts needed to detect an advantage. On seven question-answering benchmarks, TWICE improves the average exact match of GRPO by 5.3 points and 4.5 points on two model sizes, while using 68% fewer auxiliary tokens than a prior method.
Dong et al. collected roughly 1,000 pairwise judgments from 87 participants who compared ChatGPT responses generated with and without their own personal memories. The authors found that memory-conditioned responses were not consistently preferred. They introduce a memory-aware reward model that achieved 72.8 percent preference prediction accuracy, significantly outperforming standard baselines.
Liu, Wei, and Zhou study episodic linear Markov decision processes with unknown transitions, adversarial losses, and bandit feedback. They present the first polynomial-time algorithm that attains sqrt(K) regret, matching the information-theoretic optimum while remaining computationally tractable.
The authors modify GRPO to incorporate failure-conditioned retries and assign credit using a stochastic shortest-path formulation. They find that ReAPO improves first-attempt accuracy on math and science reasoning tasks compared to GRPO and recent baselines, and achieves peak accuracy with fewer generated tokens.
Deng and colleagues investigate whether reward hacking can be forecast before launching a particular reinforcement-learning run. Their simulator, SimHack, predicts when detectable hacking will dominate sampled coding attempts, reporting 11.3% relative error in median onset predictions across unseen training recipes.
Zhang et al. train multimodal models to describe a selected image region or video event, then locate it again using that description. Rewarding successful reconstruction improves localization scores and encourages video captions that distinguish a particular event from similar occurrences.
Li and colleagues show how preference training can improve response-ranking metrics while pushing a language model toward uncontrollable repetition. They identify a shared shift in the model’s internal representations that relative ranking losses barely register, then use interventions to show how it can make one token dominate generation.
The authors introduce Self-Credit, a method that uses self-generated summaries with citations to assign credit in long-horizon agentic RL, refined by comparing actions across successful and failed rollouts. On BrowseComp-Plus, it improves by up to 37 points over base models and beats strong baselines by 5-8 points. Similar gains appear on GAIA and xBench-DeepSearch.
The authors ran dense hyperparameter sweeps for direct preference optimisation (DPO) across five Qwen3 model sizes and three Llama models, evaluating on eleven benchmarks and two LLM judges. They find that optimal learning rates decrease overall with scale, that configurations transfer poorly across model sizes and preference datasets, and that validation loss is a weak guide to downstream performance.
Wen et al. address two problems in reinforcement learning for long-horizon agents: successful trajectories are difficult to sample, and final-outcome rewards do not reveal which intermediate steps were useful. Their TrajEvolve framework uses counterfactual tests to label whether steps are necessary, trains a discriminator from those labels, and refines historical trajectories into higher-quality training examples.
Liu et al. introduce SCALE, an actor-critic framework for long-horizon language agents that predicts the probability of eventual task success from every interaction prefix and converts changes in that probability into action-level credit. Across five agent benchmarks and three model backbones, SCALE improves task performance over standard RL baselines in most settings, with gains up to 4.1 percentage points in success rate. The authors also show that conventional return-based critics degrade at ranking actions for decisions far from terminal supervision, a problem SCALE directly targets.
Zhao and Yu identify a failure mode in agentic on-policy distillation that cannot be explained solely by students drifting into states their teachers cannot handle. Teachers can complete tasks from high-disagreement states, but their next-token scores still tolerate delaying actions that cause students to exhaust their turn budgets. The authors’ scorer-repair method improved task success across every tested teacher, student and environment combination.
Hu et al. address a weakness in reinforcement learning from verifiable rewards: a single score for a completed answer gives no direct indication of which reasoning steps helped or hurt. Their method, ProbePO, samples continuations from a small number of uncertain intermediate states and uses the resulting verifier scores to refine credit, outperforming GRPO across three language models and six benchmarks.
The authors show that reinforcement learning with verifiable rewards (RLVR) reduces reasoning diversity due to finite-sample dynamics, not the objective itself. They propose Targeted Adaptive Replay (TAR), which stores diverse correct solutions from the base model and replays them only when a mode is being suppressed, achieving better balance between performance and diversity across six math benchmarks.
Garg et al. present a method for estimating empowerment, the range of future states an agent can bring about, without collecting new environment interactions. By jointly learning an empowerment-seeking policy and its discounted state-occupancy distribution from offline trajectories, they produce intuitive empowerment maps and improve subsequent exploration in goal-conditioned reinforcement learning.
Tian and colleagues developed RARE, a reinforcement-learning method that updates rubric criteria as a model’s capabilities change. On three video-script types, its best sample-level variant improved average evaluation scores by 3.82 points over training with fixed per-query rubrics, while largely preserving performance on earlier requirements.
Guanda Li and colleagues propose interventional step credit (ISC), a causal measure of how much each chain-of-thought step influences the final answer, and use it to train models via interventional guidance policy optimization (IGPO). On two model scales across nine benchmarks, IGPO reduced average response length by up to 62% while improving macro-average accuracy by up to 8.4 percentage points over base models.
Sun, Wang and Li address a weakness in Group Relative Policy Optimization for video: rollouts normally see identical frames, so they may all succeed or fail and provide little comparative learning signal. Their FrameGRPO method varies frame density within each rollout group and selectively rewards successful lower-frame trajectories when every full-frame reference rollout fails. The authors report consistent gains over standard GRPO across seven video benchmarks.
Remonda et al. present Prior-Informed Adaptation, an offline-to-online reinforcement learning framework that treats simulation data as a biased starting point rather than a faithful substitute for reality. In racing experiments, the approach adapted faster and failed less often than comparison methods, while its real-car deployment required under one hour of on-vehicle interaction.