Evaluating AI outputs often requires a second AI judge, but these judges can be slow and expensive. The most accurate ones use complex reasoning, like thinking step-by-step, which costs a lot of time and money when checking thousands of answers.
This paper tested 14 simpler, faster decision models, called System-1 models, that answer questions about a text in one pass, without generating any text themselves. These models were compared to four expensive reasoning judges on eleven standard benchmarks. The five strongest System-1 models were just as accurate as the middle-tier generative judge, and two were actually more accurate.
The key finding is that a good System-1 model can match a top reasoning judge on accuracy for evaluating answers when it is given a reference answer to compare against. It does so at roughly one-sixteenth the cost per evaluation. The best System-1 models were also better calibrated, meaning their confidence scores matched how often they were actually right.