Coding agents can learn neglected behaviors without sacrificing baseline performance

The authors report that textual feedback aimed only at failed turns helps coding agents learn tool formats and writing styles that ordinary reinforcement learning struggles to teach.

Top University

Kevin Frans · Alexander M Rush · Sergey Levine · Pieter Abbeel · Ashvin Nair

Cursor Research · UC Berkeley

Research Digest·4 November 2026·2 min read

Illustration: Coding agents can learn neglected behaviors without sacrificing baseline performance

In one line

Combining per-turn textual feedback self-distillation with RL stabilizes training and improves hard-to-optimize agent behaviors.

Before you read this

Three ideas this paper builds on

on-policy distillationA training method where a student model learns from a teacher using its own current policy's outputs, often improving reasoning and instruction following.115 papers →agentic reinforcement learningTraining approach that lets LLM-based agents learn multi-step tool use and planning via reinforcement learning, often with dense or per-tool rewards.109 papers →privileged informationInformation available only during training that improves performance but is not accessible at inference.71 papers →

Read it as

When training AI agents to perform multi-step tasks like coding or tool use, standard reinforcement learning often fails if the agent never tries the correct action, it cannot learn from rewards it never earns. This is especially problematic for specific behaviors like using a rare tool format, which the agent might never attempt.

The authors propose adding per-turn textual feedback, short hints generated by a judge model, directly into the agent's context only after a failed action. The agent then minimizes a loss that pulls its next action toward what it would have done had it seen the hint, while still receiving a standard reward for success.

This approach improved the target behaviors in a software-engineering agent without harming its primary coding performance. By limiting feedback to failed turns, the method avoided common training instabilities like collapsing diversity or shortening reasoning, making the overall learning more stable and effective.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

on-policy distillation agentic reinforcement learning privileged information knowledge distillation llm as a judge group relative policy optimization