Large web-scraped datasets often have duplicates, causing unfair evaluation when the same song appears in both training and test sets. This is a problem in symbolic music datasets.
The researchers tested rule-based and neural methods to find duplicates in the Lakh MIDI Dataset of symbolic music files. They used a subset with known duplicates as a benchmark.
They identified at least 38,134 duplicates, about a fifth of the dataset. Filtered lists are provided to remove these. This improves reliability of training and evaluation for music generation models.