Language models can climb leaderboards by up to 45 places

Fleisig and colleagues show that manipulating votes, prompts and model submissions can substantially alter preference-based rankings without improving the target model.

Top University

Eve Fleisig · Joachim Baumann · Dylan Hadfield-Menell · Dirk Hovy

UC Berkeley · Università Bocconi · Stanford University · MIT

Research Digest·13 November 2026·2 min read

Illustration: Language models can climb leaderboards by up to 45 places

In one line

Large-language-model leaderboards that aggregate pairwise preferences with the Bradley-Terry model can be manipulated by strategic voting, prompting, and nomination.

Before you read this

Three ideas this paper builds on

reward hackingReward hacking occurs when an AI agent exploits loopholes to achieve high reward without meeting the intended objective.65 papers →preference optimizationA post-training method that adjusts model behavior using comparisons or self-generated preferences to improve alignment.65 papers →large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.306 papers →

Read it as

Which AI model should you trust? Many leaderboards let people vote by choosing the better answer from two anonymous models. Those votes are combined into a ranking using the Bradley-Terry model, a standard method that estimates each model's strength from pairwise comparisons. This method is equivalent to an older voting scheme, the Borda count, which is known to be manipulable.

The researchers tested three cheap attacks on public data from LMArena, a popular leaderboard: voting strategically on matches that do not involve the target model, submitting prompts the target model handles especially well, and inserting filler models to make the target look better.

Nearly every tested model could be boosted. Each attack moved a model tens of places in the ranking, and combined they moved models up to 45 places. These leaderboards influence which models users pick and which get funded, so the rankings should not be treated as tamper-proof.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

reward hacking preference optimization large language model llm as a judge benchmark verification