The authors audited the RVL-CDIP dataset along five axes: label validity, train-test split integrity, spurious visual cues (Bates stamp identification codes), distribution narrowness, and OCR engine dependency.
Systematic audit reveals pervasive data quality issues in RVL-CDIP document classification benchmark
The authors show that 10.5% of labels are erroneous, 35.2% of test set has duplicates in training, and OCR choice affects accuracy more than model architecture.
Academic
Stefan Larson · Kevin Leach
Vanderbilt University
Research Digest··2 min read
Larson and Leach systematically audited the 400,000-document RVL-CDIP benchmark and found multiple defects: label errors, train-test leakage, spurious cues, OCR sensitivity, and sensitive personal data.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.