8 million Chatbot Arena votes using model lineage fixed independently of voting outcomes.
Similar language models draw more decisive votes than rankings predict
Across 1.8 million Chatbot Arena votes, closely related models violated the assumptions behind its Bradley-Terry ranking system, although correcting for this barely changed the leaderboard.
Independent
Soovadeep Bakshi · Apurba Bose · Aishwarya Bose · Sandip Nandi
Independent Researcher
Research Digest··3 min read
Bakshi and colleagues test whether Chatbot Arena's single-score ranking model accurately describes comparisons between closely related language models.
Why this paper
From Independent Researcher
In one line
Chatbot Arena’s Bradley-Terry scores understate how decisively voters distinguish closely related models, though correcting this barely changes leaderboard order.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ✓Reports numbers on named benchmarks (3 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§