The authors evaluated language models across four code-generation benchmarks, comparing one-shot performance with multiple sampled solutions selected through execution-based verification.
Verified retries let smaller code models recover lost capability
Across four programming benchmarks, parallel generation and test-based selection allowed compressed or smaller models to rival much larger models within lower memory budgets.
Big Tech
Tomer Galanti · Josh Susskind · Eran Malach
Texas A&M · Apple
Research Digest··3 min read
Galanti, Susskind and Malach test whether repeated, verified attempts can compensate for using a smaller or more heavily compressed language model.
Why this paper
From Apple and Texas A&M
In one line
Verified retries from smaller models can match or beat much larger models on code benchmarks while using less memory, time, and energy.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§