Aligning auxiliary rewards with task success improves multi-task language model training

Across three models and 15 benchmarks, the authors report that constraining auxiliary reward scores improves average performance over seven existing reinforcement learning estimators.

Big Tech

Jixuan Leng · Si Si · Fei Wang · Cho-Jui Hsieh · Jiong Zhang · Hsiang-Fu Yu · +1 more

CMU · Google DeepMind · Google · UCLA

Research Digest·28 October 2026·3 min read

Illustration: Aligning auxiliary rewards with task success improves multi-task language model training

In one line

CONCUR projects gated auxiliary rewards onto scores that agree with the primary reward, improving multi-task LLM reinforcement learning.

Before you read this

Three ideas this paper builds on

advantage estimationIn policy optimization, a method that calculates the relative value of actions using group statistics or baselines to guide learning.43 papers →credit assignmentDetermining which actions or steps in a sequence contributed to a final outcome, often for reinforcement learning.168 papers →reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.222 papers →

Read it as

Training a large language model (LLM) by on several tasks gives each task a main score and extra scores that are supposed to matter only for responses that already pass the main score. Standard advantage estimators treat every score equally, so an extra score can overturn the main score, or act alone whenever main scores are tied.

The authors propose CONCUR, a small adjustment to the estimator. It rewrites each extra score so that a response with a higher main score never receives a lower extra score, and responses at the main score's ceiling are scored equally. Fixed and adaptive variants handle auxiliaries that mostly agree with the main score differently.

Across math, code, instruction-following, and writing tasks, and several model families, CONCUR scored more than four points higher on average than the original estimator or than dropping extra scores. It preserves the main task's signal while still using what auxiliary scores reward.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

advantage estimation credit assignment reinforcement learning multi-task reinforcement learning gated auxiliary reward group relative policy optimization