Using a language model costs money and time per token, the pieces the AI reads as words or parts of words. Some languages like Hindi get split into far more tokens than English for the same text, making them slower and more expensive. This tax works against fair and affordable AI across the globe.
The researchers built a new tokenizer from scratch, training it on human writing in about 200 languages. They added a second stage that merges common multi-word phrases, like "we are" into one token. The final version, Havon v5, also works well on computer code.
The best version needs 6.1% fewer tokens than GPT-4o on English and fewer tokens on 161 of 204 languages, including all 25 languages of India, at the same vocabulary size. This could make AI cheaper and more even for speakers of many languages.