Telescopic Language Models (TLM) train a single Transformer to be a valid language model at every layer depth, using stochastic prefix supervision where one random truncated prefix is trained against the full next-token target alongside a full-capacity pass. On a 200M parameter suite trained on 20B FineWeb-Edu tokens, TLM reduces the area under the quality-budget curve by 43-44% relative to fixed-exit suites like MLMS, while matching full-capacity performance (perplexity 14.99 vs. 13.98 for a separate twin) and at 12% lower GPU cost. This makes it possible to serve multiple compute budgets from a single training run without architectural changes.
One nested language model works across every tested layer depth
Stochastic supervision made all 20 layer prefixes usable while preserving full-capacity performance.
Big Tech
Zhilin Guo · Boqiao Zhang · Hakan Aktas · Kyle Fogarty · Nursena Koprucu Aslan · Wenzhao Li · +11 more
University of Cambridge · University of British Columbia · Google
Research Digest··2 min read
Thread:Nested Language Models
In one line
Training nested-capacity Transformers with stochastic prefix supervision yields valid language models at every depth from one run.
Read it as
Guo and colleagues trained a 200 million-parameter Transformer whose computation can stop after any layer, yielding a continuum of smaller language models from one artifact.
The authors trained on 20 billion FineWeb-Edu tokens, using the same data stream across methods.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.