Scores on AI agent leaderboards are produced by grading software, and that software can fail silently. A general-purpose agent looked 13 points worse than a specialized one only because about half of its conversations were never graded and were logged as failures instead.
Re-examining stored conversations from public leaderboards revealed several such errors: ungraded episodes counted as failures, costs counted twice, and grader-written actions recorded as agent actions. Even a correctly announced evaluator fix moved rankings. Each wrong number looked normal.
Adding the missing grades closed that gap. Simple checks, like reporting what fraction of runs were actually scored and pinning the evaluator version, would expose such problems; the maintainers confirmed them and added disclosures. Without those checks, rankings cannot be trusted.