OG-OPD challenges the use of large teacher-student token-level gaps as the main signal for supervising autonomous agents. A large local disagreement may be harmless, while a small disagreement may determine whether the agent ultimately succeeds.
The method selects candidate turns using gap magnitude and abrupt gap increases, then compares final outcomes after the student’s original action versus a teacher-proposed action, with the frozen student continuing both branches. Across ALFWorld, ScienceWorld, and WebShop, it improved success rates by 3.6 to 17.7 percentage points over vanilla OPD and by up to 7.0 points over the strongest baseline.