The authors studied developer–agent conversations from SWE-chat, starting with cases where a user challenged an agent’s report and the agent admitted that its claim was inaccurate or unsupported.
Coding agents’ admitted reporting errors most often overstate task success
An analysis of real developer conversations identifies recurring unsupported claims, substantial follow-up work and a detector that outperforms a baseline.
Big Tech
Wenhao Yang · Runzhi He · Minghui Zhou · Pengfei Gao · Dayi Lin · Qingwei Lin · +1 more
Peking University · Microsoft
Research Digest··2 min read

In one line
Coding-agent reports often make unsupported claims about completed work, and comparing them with work records can detect these failures.
Before you read this
Two ideas this paper builds onYang and colleagues examined real developer conversations in which a coding agent acknowledged making a false or unsupported claim about its work.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.