Standard autoregressive Transformers pass information between generation steps primarily through discrete tokens.
Latent feedback helps Transformers carry information across generation steps
LIFT models learn a recurrent hidden state from teacher distributions while retaining parallel pretraining.
Top University
Dor Tirosh · Ido Amos · Mor Geva
Tel Aviv University · The Hebrew University of Jerusalem
Research Digest··3 min read
Tirosh, Amos and Geva introduce LIFT, a Transformer architecture that feeds a compact continuous state from each generated position into the processing of the next.
Why this paper
From Tel Aviv University and The Hebrew University of Jerusalem · Released code
In one line
Pretraining Transformers to predict teacher-derived latent states as well as next tokens, then feeding back their own predicted states at inference, improves language modeling, reasoning, and procedural task performance.
What it released
Code
What we could check
- ✓Code link in the paper (github.com)
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§