When large language models (LLMs) are pruned to become smaller, their accuracy on reasoning tasks often falls. But does this drop reflect a genuine loss of reasoning ability, or simply that correct answers become harder to sample?
The researchers compared original and pruned models by repeatedly sampling solutions. They measured the proportion of problems solved within a fixed number of attempts and compared the sets of problems each model could eventually solve.
Even heavily pruned models could often solve the same problems as the original, sometimes solving ones the original missed. However, achieving that required far more attempts and tokens, shifting the cost to sampling. This shows that pruning evaluations must consider both the range of still-solvable problems and the effort to reach them.