AI systems are now used to review scientific papers, but it hasn't been clear how reliable they are at finding mistakes.
To test this, researchers created a : they added hundreds of objective errors of four types, general issues, math, numbers, and citations, at mild, clear, or severe levels into real papers from astrophysics and machine learning. They then measured how many errors twelve different AI reviewers detected and whether they wrongly flagged correct text.
The most important factor was the choice of AI model: with the same prompt, the best models caught about 50 percentage points more errors than the weakest. Changing to a longer prompt that encouraged tool use raised detection by over 30 percentage points. Tools that checked cited papers or ran calculations helped for specific error types. This gives a standard way to evaluate AI reviewers and shows their strengths and weaknesses.