Fri 30 Oct
LiveGet it free
ZOTPAPER

News without the noise

Reinforcement learning

Reinforcement learning methods, RL for language models and agents, reward modelling, preference optimisation

207 articles · page 4 of 5

AllAgentsLanguage modelsReinforcement learningVisionGenerative mediaSpeech & audioRobotics & embodied AISafety & securityEvaluation & benchmarksEfficiency & systemsCode & softwareAI for science & healthTheory
Viewgridlistcompact
Reinforcement learning

Online distillation cuts needless verbosity in RL-trained models

The authors identify and quantify the 'length-scaling tax'—excess response length on already-solved problems that arises during reinforcement-learning (RL) post-training. They propose Length Self-Distillation (LSD), which routes solved prompts to on-policy distillation using an exponential moving average of the online policy as teacher, while retaining the original RL objective for unsolved prompts. LSD reduces LST from 19.0% to -3.7% on single-turn reasoning and from 31.4% to 13.7% on multi-turn agentic tasks, achieving comparable or better accuracy than vanilla RL.

·3 min
Reinforcement learning

Training agents on their own trajectories accelerates reward-based learning

Qiao et al. introduce On-Policy Warmup (OPW), in which a language-model agent receives teacher supervision while following its own environment trajectories, then switches to reinforcement learning with verifiable rewards. In experiments with Qwen3-4B and GRPO on ALFWorld, OPW reached strong performance earlier and produced better average and final results than the compared initializations.

·3 min
Reinforcement learning

Temporal distance improves self-supervised goal-directed reinforcement learning

Bohlinger and Peters introduce ChronoSRL, a self-supervised reinforcement learning method whose learned representation measures distance in units of goal-reaching time rather than physical proximity or an unconstrained similarity score. Across seven locomotion and navigation benchmarks, the authors report faster learning and higher performance than contrastive and survival-based alternatives, including methods using larger networks.

·3 min
Reinforcement learning

Learning options by focusing on relevant features instead of whole states

The authors present an intrinsically motivated option discovery algorithm that learns abstract subgoals by attending only to a relevant subset of state features, rather than requiring the agent to recreate entire states. In three sparse-reward, image-based domains including Montezuma's Revenge, the method discovers options that transfer across tasks and significantly outperform prior state-reaching approaches.

·3 min
Reinforcement learning

Averaging rollout contexts improves label-free language model training

Wang et al. identify a source of noise in self-rewarding reinforcement learning: a response’s reward can change depending on which other model outputs happen to share its rollout group. Their Group-Marginalized Advantage Estimation method aggregates rewards across possible group contexts, producing more reliable policy updates without human labels. Across eight benchmarks and four base models, the authors report improved performance, stability and cross-domain generalization.

·2 min
Reinforcement learning

Off-policy sampling avoids suboptimal traps in reward-guided self-training

The authors study RE(S), a generalization of REINFORCE that updates the rollout distribution once every S gradient steps. They prove that for softmax policies in multi-armed bandits, RE(S) converges globally to the optimal policy at a Θ(1/T) rate for any S, and that off-policy sampling (S>1) avoids prolonged entrapment in suboptimal policies when the initial policy assigns low probability to the optimal action.

·3 min
Reinforcement learning

Self-privileged critic improves value estimation for RLVR

Liang et al. propose πPPO, an actor-critic framework for reinforcement learning with verifiable rewards (RLVR) that conditions the critic on contrastive evidence from successful and failed rollouts of the same prompt. This self-privileged formulation consistently improves value-estimation accuracy and outperforms standard PPO and critic-free baselines on challenging mathematical reasoning benchmarks, while also supporting smaller asymmetric critics.

·3 min
Reinforcement learning

Fisher-Rao prompt curricula make GRPO training more rollout-efficient

Okonkwo et al. analyze why group relative policy optimization wastes computation on prompts whose sampled answers are either all correct or all incorrect, producing no learning signal. They derive a prompt curriculum in Fisher-Rao arc-length coordinates and implement it in ARCUS, which outperformed standard GRPO and dynamic sampling across six mathematical reasoning benchmarks and three model backbones.

·2 min
Reinforcement learning

Self-generated failure summaries help models solve previously impossible tasks

Kirchhof et al. introduce RLTL;DR, a training method that turns verifier feedback from failed coding and tool-use attempts into short, model-generated insights for subsequent attempts. On tasks the base model could not solve in 128 tries, the method raised Pass@1 from 0 to 1 percent under standard training to as high as 31 percent, while retaining 12 to 13 percent when the insights were removed at evaluation.

·2 min
Reinforcement learning

Spatial gradient maps improve RL alignment of flow-based image models

The paper proposes SGA-Flow-GRPO, a reinforcement learning framework that assigns spatially differentiated credit to individual latent tokens in flow-based generative models. By reformulating the log-likelihood into token-wise importance ratios and deriving a continuous spatial credit map from reward gradients, the method outperforms standard Flow-GRPO in alignment quality while converging as fast as DiffusionNFT.

·2 min
Reinforcement learning

HiLoRe allocates GRPO activations between storage, compression, and recomputation using update exposure.

The authors propose HiLoRe, an activation-management method for GRPO training that uses the policy's analytic update coefficients to decide which states to store at high precision, which to compress, and which to recompute. Across five model-task settings with 2K responses, HiLoRe improves actor-update throughput by up to 13.5% over gradient checkpointing and 7.9% over the fastest evaluated baseline, with negligible downstream quality differences.

·2 min
Reinforcement learning

Empathetic AI learns to adapt support priorities as conversations evolve

The authors identify a fundamental mismatch in empathetic reinforcement learning: reward specifications remain fixed while support needs change. They propose CARE (Context-Adaptive Rubric Evolution), which generates turn-level rubrics that adjust the weights and criteria of three empathy dimensions. On the EMPA benchmark, CARE improves the EPM-Idx metric from 28.11 to 83.54 under Gemini-2.5-Pro, a gain of over 55 points, and consistently outperforms baselines across three independent judges.

·2 min
Reinforcement learning

Twin critics calibrate token-level advantages from a single rollout, boosting RL training efficiency

Qiao et al. introduce a twin-critic method for token-level credit assignment in reinforcement mid-training of language models. By training two critics and combining their advantage estimates with action-dependent weights learned via a conditional-moment objective, they reduce update drift and eliminate the need for repeated generation. Experiments show improved performance and substantially faster training.

·2 min
Reinforcement learning

Action-discriminative world models improve counterfactual planning for robot control

Qiu et al. identify a mismatch between how world models are trained and how they are used: factual prediction rewards matching observed transitions, while planning requires comparing unobserved alternative actions. Their AD-WM adds action-recovery objectives to a latent world model, improving closed-loop control without changing the model predictive control procedure at test time.

·2 min
Reinforcement learning

Anchored hypergraphs provide stable credit assignment for multi-agent teams

The authors introduce HySTAR, a MAPPO-based framework that uses an anchored overlapping sparse hypergraph as a temporally consistent value-decomposition basis, while adapting representations via a spatiotemporal encoder. This decoupling prevents structural target drift—the inconsistency in credit assignment when agent interactions change—and yields consistent improvements over MAPPO, value-factorization, and dynamic-grouping baselines across SMAC, GRF, Traffic Junction, and MPE environments. On the hardest SMAC settings, HySTAR achieves relative gains of 16.7% over MAPPO and 15.6% over HYGMA.

·3 min
Reinforcement learning

A framework predicts RL post-training outcomes without running reinforcement learning

The authors introduce PoEM, a method that predicts the output of reinforcement learning (RL) on a new reward function by linearly combining log-policies from models already post-trained on other rewards. They show that for linear reward combinations, the optimal policy is a log-linear combination of existing policies, and even for non-linear rewards, log-policies often span a low-rank subspace, enabling accurate approximation without additional RL.

·3 min