A release decision concerns the next rerun of a training recipe, not the mean of past reruns.
Mean benchmark accuracy misleads model release decisions; recipe cards certify tail risk
Auditing 2,124 fine-tuned models across eleven language models, the authors show that mean-based shipping violates a 10% deployment failure budget 48% of the time, while recipe cards reduce violations to 0.6%.
Top University
Jaewon Choi · Jihye Kang · Namhyuk Ahn
Seoul National University Hospital · Inha University
Research Digest··3 min read
The authors compare standard mean-accuracy reporting against the actual tail failure rate of fine-tuned model reruns.
Why this paper
From Seoul National University Hospital and Inha University
In one line
Mean accuracy is the wrong estimand for model release decisions; tail failure probability should be certified instead.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§