What they did

The authors organized reusable mathematical skills into a structured taxonomy and aligned them with three stages of agentic behavior: planning, action, and feedback. Their evaluation suite covers both text-only and multimodal problems.

They also developed an automated pipeline that synthesizes problem-solving trajectories and generates fine-grained process annotations through controlled LLM rewriting. These annotations are intended to identify where a model's reasoning process succeeds or fails.

Key findings

• Models with similar final-answer accuracy displayed markedly different profiles across the assessed agentic capabilities.
• Process-level scoring distinguished failures in planning, execution, and feedback that outcome-only benchmarks would collapse into a single correct-or-incorrect result.
• The framework supported evaluation across both textual and multimodal mathematical settings.
• The results indicate that end-to-end accuracy is an incomplete proxy for a model's readiness to operate as a mathematical agent.

Why it matters

Mathematical agents must do more than produce correct answers: they need to formulate plans, execute steps, inspect intermediate results, and recover from errors. A capability-level benchmark could help researchers diagnose weaknesses more precisely and compare systems that appear equivalent under conventional answer-based evaluation.

Caveats

The abstract does not report the benchmark's size, model roster, human-validation procedures, or quantitative effect sizes. Because trajectories and annotations are produced partly through controlled LLM rewriting, their reliability and susceptibility to model-generated artifacts require careful validation; it also remains unclear how well the taxonomy transfers to open-ended mathematical work or real tool-using agents.