The authors developed a paired-run method to measure adaptive overfitting, meaning score inflation caused by repeatedly consulting the test set while choosing hyperparameters.
Test-set tuning modestly inflates scores but rarely changes model rankings
Across five classification tasks and 36,000 trained models, test-set optimization produced statistically significant overfitting without changing the top-ranked architecture.
Academic
Matteo Fregonara · Tom Viering · Jan van Gemert
Delft University of Technology
Research Digest··3 min read
Thread:Research Ideation Evaluation
Fregonara and colleagues directly compared hyperparameters selected using validation data with those selected using the test set across image and language benchmarks.
Why this paper
From Delft University of Technology · Part of Research Ideation Evaluation, now 4 papers
In one line
Honest test-set hyperparameter tuning mildly inflates reported performance but usually preserves model rankings when applied consistently across models.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks (3 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§