Grading proofs is more than checking the answer. Every step must be sound, one hidden gap breaks the argument. Experts are reliable but expensive to scale.
They built a benchmark of U.S. Mathematical Olympiad candidate solutions with expert grades and injected subtle errors, and used a self-critique pipeline called TROJAN to generate more. They evaluated many large language models as judges and proposed MABGrader, which treats grading as choosing among possible scores and repeatedly adjusts.
Current models still fall short of expert agreement. MABGrader improves on the previous best by 26.4% on a measure of grader agreement and makes 14% fewer grading errors. Reliable automated grading would make Olympiad-level evaluation scalable and catch flaws final-answer tests miss.