OracleLadder diagnoses multi-step reasoning failures by progressively helping a model: first with a roadmap of milestones, then with the milestone answers, while also testing each milestone independently. A deterministic symbolic verifier grades the outputs.
Across 354 NuminaMath problems and six models ranging from 8B to 671B parameters, composition was the largest failure category for every model, covering 33–48% of problems. After excluding cases flagged as grading errors by an LLM review, the range was 24–37%.
This matters because final accuracy hides why models fail. The diagnostic ranked the two strongest models differently from accuracy and showed that two RLVR runs with similar accuracy gains changed different reasoning capabilities. Related effects appeared on MATH500, AIME 2024/25, and a code-generation pilot.