Small language models (SLMs) are cheaper and faster than big frontier models, making them attractive for on-device tasks like using a terminal to fix files or run programs. But they succeed far less often. Why?
Researchers separated the failure into three skills: producing a correct answer, telling good answers from bad ones, and predicting whether an attempt will work. They compared a small model with a large one on two standard benchmarks.
With more tries, the small model got much better at producing correct answers, closing most of the gap. Yet even when given the same pool of candidate solutions, it still picked wrong ones more often, and it was more overconfident about its chances. So the real weakness lies not in generation but in judgment.