LIFT removes the feed-forward bottleneck in Transformers by training the model to predict, alongside the next token, an information-dense state produced by a pretrained teacher LM. The states are precomputed, so pretraining stays parallel; at inference the teacher is removed and the model feeds its own predicted states back. Across models from 135M to 1B parameters, LIFT beats standard Transformers on language modeling, reasoning, and procedural tasks under matched token budgets and matches or exceeds them under matched compute.
Latent feedback helps Transformers carry information across generation steps
LIFT models learn a recurrent hidden state from teacher distributions while retaining parallel pretraining.
Top University
Dor Tirosh · Ido Amos · Mor Geva
Tel Aviv University · The Hebrew University of Jerusalem
Research Digest··3 min read
In one line
Pretraining Transformers to predict teacher-derived latent states as well as next tokens, then feeding back their own predicted states at inference, improves language modeling, reasoning, and procedural task performance.
Before you read this
Two ideas this paper builds onlatent communicationLatent communication transfers information between models via internal states like KV caches instead of explicit text.16 papers →teacher forcingTraining technique using ground-truth tokens as step-by-step autoregressive input instead of the model's own generated history.4 papers →
Read it as
Tirosh, Amos and Geva introduce LIFT, a Transformer architecture that feeds a compact continuous state from each generated position into the processing of the next.
Standard autoregressive Transformers pass information between generation steps primarily through discrete tokens.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.
What it released
Code