The paper examines how reproducible evaluation conclusions are, using LLM-based prompt-structure inference as a case study. Eight model variants were ranked by reproducibility, but the full ranking did not survive resampling: only the bottom two models held rank reliably, while the top two held rank in only 68% of replicates. Defensible changes to analysis rules, like how repeated measurements were merged, moved four of eight rows and shifted the headline by 7 percentage points. The study also found that four of eight endpoints were no longer measurable within ten weeks. This demonstrates that small-sample LLM evaluations can appear far more definitive than the evidence supports, and the authors recommend reporting rank stability, sensitivity comparisons, and raw outputs alongside any ranking.
Small-Sample LLM Rankings Often Overstate Evaluation Reproducibility
A self-audit of eight models found that repeated outputs, model rankings and headline conclusions were sensitive to sampling and analysis choices.
Industry
Skelf Research
Research Digest··2 min read
In one line
LLM evaluation rankings often look definitive but are highly unstable under resampling and defensible analysis changes.
Before you read this
Two ideas this paper builds onRead it as
Sarkar repeatedly tested eight open model variants on prompt-structure inference, preserving 293 intermediate representations and auditing both output consistency and the resulting rankings.
The author evaluated eight open model variants from five families, ranging from 8 billion to 675 billion parameters, on recovering structured representations from prompts.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.