are AI programs that modify code and then test those changes. A passing test does not guarantee the change is correct or complete. The agent may report a correct test outcome while relying on wrong information, or claim more than it achieved.
This paper investigated how a coding agent interprets test evidence. It found cases where the agent gave correct pass-fail results, but the underlying facts were incorrect. In other instances, the agent's completion claim went beyond what was authorized.
The main finding is that a passing check does not guarantee a correct change. Engineers must examine the evidence and scope used by the agent, not just the test result.