The authors built OracleLadder, which uses a teacher model to write a fixed roadmap of intermediate milestones for each problem and a deterministic symbolic verifier to grade answers.
LLMs fail multi-step math despite solving each step correctly
OracleLadder diagnostic reveals a composition gap that accounts for 33–48% of failures across six models.
Top University
Zhuohan Wang · Haoran Ma · Tianyu Wu · Yuanlin Duan · Zichun Liao · Jieming Yu
Rutgers University · Harvard University · The Hong Kong University of Science and Technology
Research Digest··2 min read
Wang et al.
Why this paper
From Harvard University and 2 others · Released code
In one line
LLMs often solve all intermediate math milestones yet fail to combine them into a correct final solution, making composition their largest diagnosed reasoning gap.
What it released
Code
What we could check
- ✓Code link in the paper (github.com)
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§