When you hire a contractor, you test their work with a sample that you know they haven't seen. This gives you a fair measure of quality. But for large AI models trained on vast internet data, it's nearly impossible to guarantee that any test sample is truly unseen. This breaks the usual way we evaluate models.
This paper asks whether it's possible to evaluate a black-box model, one you can only feed inputs and get outputs, without a fresh test set. The authors studied scenarios where the evaluator can only look at model outputs on the test set, and also where they can ask the model extra questions. They proved conditions for when accurate evaluation is possible.
The main finding is that reliable evaluation without a fresh test set is possible only when the data distribution is close to having small support, meaning the possible inputs come from a very limited set. If the model can be queried on additional points, evaluation becomes more powerful in some natural cases. This matters because it shows fundamental limits to trusting benchmark scores for large models.