Feedback criteria that track a model's growing ability halve its training steps

Co-evolving the scoring rubric with the policy lets language models absorb guided behaviors into normal responses and reach target performance in half the steps.

Chinese Tech

Chengsheng Zhang · Wenshen Xu · Yuchuan Wu · Zhiyuan Yu · Junfeng Fang · Tianlong Wang · +3 more

University of Science and Technology of China · Alibaba Group · Nanjing University · National University of Singapore

Research Digest·30 October 2026·2 min read

Illustration: Feedback criteria that track a model's growing ability halve its training steps

In one line

Co-evolving a reward rubric alongside a policy by tracking its capability frontier improves and accelerates open-ended LLM training.

Before you read this

Three ideas this paper builds on

co-evolving feedbackFeedback that evolves alongside the policy or components it guides to improve performance.38 papers →on-policy distillationA training method where a student model learns from a teacher using its own current policy's outputs, often improving reasoning and instruction following.108 papers →reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.231 papers →

Read it as

Training a large language model for open-ended generation usually needs a detailed scoring guide to provide feedback. If the rubric stays static while the model improves, the feedback becomes useless.

The proposed method, CoCA, contrasts normal responses with responses nudged by the rubric. This reveals what the model can only do with a hint. The rubric is adjusted to track this frontier, and the model either imitates the guided success or explores the frontier.

In tests across science and medical benchmarks, CoCA consistently beat standard methods, often reaching the same or better quality in about half the training steps. This shows that co-evolving the model and its feedback rubric saves compute and produces more capable text generators.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

co-evolving feedback rubric-based reinforcement learning capability awareness on-policy distillation reinforcement learning privileged information