Language model APIs charge per token, the small units text is broken into. Because tokenizers are trained mainly on English, they split non-English languages into many more tokens, making them more expensive and slower.
Researchers measured several tokenizers, including GPT-4o's, GPT-4's, and ones built for Indian languages by Sarvam AI, on all Indian languages in a standard multilingual test set and on Hinglish, a romanized mix of Hindi and English. They compared token counts to English.
On GPT-4o's tokenizer, Hindi required 1.57 times the tokens of English, but Santali needed 13.7 times. This token tax means Indian-language users pay more for the same service. Indian-focused tokenizers like Sarvam's reduce the gap but don't fix it completely.