Many conferences are using AI language models (programs that can read and write text) to help review research papers. But these models are updated every few months, which might change their behavior.
To check this, the researchers tested twenty different AI models from OpenAI, Google, and Anthropic on hundreds of papers submitted to a recent conference. They compared how often earlier and later versions of the same model recommended accepting a paper.
They found that later versions of two models accepted far fewer papers, with the acceptance share falling by 52 and 53 percentage points. No paper received a higher score from the later version. This means that AI scores are not stable across versions, and conferences must validate scores for each specific model version.
