The evaluation separates two uses often conflated as “confidence”: estimating whether a question is easy for a model, and choosing the correct response among several responses to the same question. Across five reasoning models, three mathematical and scientific benchmarks, and four certainty metrics, token-level certainty was generally more effective for the first task.
Certainty also depends on token type and location. Difficulty information appears early in generation, while signals distinguishing correct from incorrect answers concentrate nearer the end. Using early certainty to allocate samples and late certainty to weight votes raised accuracy from 78.71% to 79.54% while reducing generated-token cost by 82.4% versus fixed 256-response majority voting.