Terminal agents use self-verification to check their solutions during command-line tasks. The authors developed a diagnostic framework to measure how often agents actually detect and fix their own errors by comparing their outcomes against objective correctness. They analyzed ten different agents on the TerminalBench2.1 benchmark.
They found that agents almost always initiate verification (99.53% of trajectories), but only 61.43% of incorrect candidates are detected, and only 49.36% of detected errors are successfully repaired. This shows that the main weakness is not lack of verification but poor detection and repair.
To address this, they propose Student-Conditioned Verification Distillation (SCVD), where a stronger teacher model continues from a student's candidate to demonstrate effective verification and repair. SCVD improved Pass@1 on TerminalBench2.1 by 9.74-16.85 percentage points over base models and 4.49-8.61 points over standard full-trajectory distillation, while avoiding performance drops on SWE-bench Verified.