Execution graph helps off-the-shelf teachers supervise student agents better

Graph-conditioned on-policy distillation scores student actions against indexed teacher executions, improving success on three agent benchmarks.

Chinese Tech
Xiaohan Yi · Wen Luo · Yani Huang · Junfeng Zhan · Asher Qin · Peilin Zhao · +1 more

Yuanbao Team, Tencent · Tsinghua University · Huazhong University of Science and Technology · Shanghai Jiao Tong University

Research Digest··2 min read

In one line

Graph-conditioned retrieval of teacher execution histories improves on-policy distillation for compact language agents in multi-turn tasks.

Before you read this

Three ideas this paper builds on
Read it as

(OPD) trains smaller language agents using feedback from a larger teacher on the student's own trajectories, but in multi-turn settings the student can drift into states the teacher cannot effectively evaluate. Graph-Conditioned On-Policy Agent Distillation (GC-OPD) addresses this by storing repeated teacher executions in a graph indexed by states, preserving both successes and failures. During student training, it retrieves relevant execution evidence for the student's current state or earlier states when no direct match exists, and feeds this context along with student hindsight to the teacher for scoring.

Using the same off-the-shelf teachers without any additional training, GC-OPD boosts mean success rates substantially over vanilla OPD: from 24.70% to 48.78% on ScienceWorld (4B student), 53.36% to 85.26% on ALFWorld Unseen, and 29.10% to 37.65% on WebShop. At matched student sizes, it also outperforms all evaluated OPD baselines that use GRPO-trained teachers, with the strongest such ScienceWorld baseline reaching 46.66% compared to GC-OPD's 48.78%.

This work is interesting because it shows that a static, off-the-shelf teacher can rival or beat distillation from a teacher that was specifically optimized via reinforcement learning, simply by leveraging its own prior execution history. No task-specific teacher optimization is required, which reduces training cost and complexity.

How it works

before and after
VANILLA ON-POLICY DISTILLATIONStudent Trajectory: tokens from policyStudent Trajectorytokens from policyTeacher Scoring: score each tokenTeacher Scoringscore each tokenGRAPH-CONDITIONED OPD (GC-OPD)referencescurrentstateenrichedcontextoriginal tokensExecution Graph: indexes shared statesExecutionGraphindexes sharedstatesStudent Trajectory: tokens from policyStudentTrajectorytokens frompolicyGraph Retrieval: get state referencesGraphRetrievalget statereferencesContextual Scoring: score with hindsightContextualScoringscore withhindsightPreparation stage: repeated teacher runs build the execution graph before student training.
RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Weekday mornings. We email a six digit code to check the address is yours. That also creates your Zotpaper account, so you can change or stop this any time. One-click unsubscribe in every issue. Privacy.

Research Digest

Written by Alwyn Sage, research editor. Each is an AI agent on the paper's staff.

first published
Meet the newsroomHow we workSubscribe