Language models can score better without learning, just by changing vocabularies

Schaeffer and Koyejo show that a common text-compression metric can favor particular tokenizers even when models remain at chance-level accuracy.

Top University

Rylan Schaeffer · Sanmi Koyejo

Stanford

Research Digest·4 November 2026·2 min read

Illustration: Language models can score better without learning, just by changing vocabularies

In one line

Bits-per-byte and unigram-normalized loss are not tokenizer-agnostic, and no metric can be: choosing one changes which research question gets answered.

Before you read this

An idea this paper builds on

large language modelA neural network trained on massive text corpora to generate coherent text and perform diverse language tasks.259 papers →

Read it as

Language models chop text into chunks called tokens, and the chopper is a tokenizer. Bits-Per-Byte (BPB) is a common score: how many bits a model needs to encode one byte of text. People treat BPB as neutral, but it is not: even an untrained model scores better with some tokenizers just because of the chopping.

The authors challenge three beliefs. A lower BPB does not reliably mean better answers; a supposedly fair alternative metric is also biased; and no score can be fair across tokenizers, treat identical predictions alike, and prefer only the true pattern.

So tokenizer comparisons depend on the metric. Choosing a metric is choosing a research question, not just measuring quality.

How it works

stage by stage

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

tokenization language model evaluation proper scoring rules byte pair encoding large language model