The authors built a benchmark by mining OpenReview passages in which reviewers explicitly affirmed or disputed a submission's originality.
LLM novelty judgments shift sharply with minor evaluation changes
Across six judges, prompt wording could reverse verdicts on identical research ideas, while retrieval and additional reasoning offered little consistent improvement.
Big Tech
Noy Sternlicht · Simra Shahid · Peter Jansen · Daniel S. Weld · Pao Siangliulue · Tom Hope
Hebrew University of Jerusalem · Allen Institute for AI · Microsoft · University of Arizona · University of Washington
Research Digest··2 min read
Sternlicht et al.
Why this paper
From Microsoft and 4 others
In one line
LLM-based novelty judges are highly unstable; small prompt changes swing pairwise accuracy by over 50 points and can push judges below chance.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§