AI model scores on standard tests might not reflect true ability if the model has already seen the test questions during training. This is called benchmark contamination, and it's getting worse. Recently, some models have even reproduced complete test questions and answers they were never given, proving the contamination happened.
The authors studied several large language models, the kind that power chatbots and code assistants. They used a method called few-shot continuation, where the model is given a few examples and then asked to continue, to see if it would accidentally output test questions from its training. They checked 20 different tests across four publicly available base models.
They found that one model, ERNIE-5.5-Flash-Base, leaked 96% of a coding test and 85% of a knowledge test, giving away answers. When the authors subtracted credit for leaked questions, this model's score dropped by 16.6 points, changing its ranking from first to last. This means reported scores can be misleading, especially for models used as building blocks in other research.
