When engineers test AI agents on tasks like data analysis, they usually judge each answer pass or fail. This one-bit score cannot tell apart the best agents once they all succeed, making tasks seem too easy.
But the researchers show the issue is the scoring, not the tasks. They analyzed several and found top agents that tie on pass/fail often differ over tenfold in actual quality. Their solution is graded credit, which awards partial points and counts cost separately.
This matters because binary scoring hides real differences. Graded credit keeps benchmarks useful as agents improve, without requiring constantly harder tasks.