1B and mBERT.
Tokeniser emptiness, not inefficiency, cripples small models on Indic scripts
Pramod shows that five of eleven Indic scripts are read as raw bytes by TinyLlama, with zero UNK tokens and a 94% reduction in usable context window.
Academic
Anna Pramod
University of Leeds
Research Digest··2 min read
The paper demonstrates that the reported efficiency of small language models (SLMs) is measured in English and does not transfer to Indic scripts.
§