Software increasingly relies on large language models (LLMs), programs that generate text, to solve problems. A model that can tell which of its own answers are right could check itself without a human.
The work extends a measure, the Vapnik-Chervonenkis dimension, which captures how finely a set of rules separates cases, to models that output probabilities. That yields a pair of scores: how sharply a model separates its own good answers from bad, and whether its confidence matches its success rate.
Testing small open models, each choosing between a pair of its own solutions and rating its confidence, the experiments showed that the sharper a model's self-judgment, the less its confidence matched reality. Stated confidence cannot be read as reliability.