teacher forcing
During training, a model often learns to predict the next step by being given the correct previous steps, this is teacher forcing. It is like a student copying from a perfect answer key rather than working from their own earlier answers. While it speeds up learning, it can cause problems later because the model never practices recovering from its own mistakes.
The papers explore ways to soften or bypass these problems. One approach mixes a soft, model-generated hint with the ground truth to reduce over‑dependence on perfect history. Another trains the model to also predict a richer hidden state from a teacher, letting it learn to reuse its own outputs. For video generation, auxiliary objectives or residual predictions force the model to look beyond local shortcuts and consider genuine temporal evolution, improving quality without extra cost at inference.
- Papers
- 4
- Released code
- 1
- First seen
- Sept 2026
- Latest
- Oct 2026
4 papers in the last two months, against 0 in the two before.
Start here
Two papers to read first, in the order they came out- 1Latent feedback helps Transformers carry information across generation steps
Pretraining Transformers to predict teacher-derived latent states as well as next tokens, then feeding back their own predicted states at inference, improves language modeling, reasoning, and procedural task performance.
Sept 2026 - 2Soft tokens improve supervised training for mathematical reasoning
Soft Teacher Forcing improves supervised fine-tuning by mixing ground-truth embeddings with soft tokens from a single forward pass.
Oct 2026
How fast it is moving
Often studied with
autoregressive video generationGenerates video frames one at a time, each conditioned on previously generated frames.1 togetherdiffusion modelA type of generative model that learns to reverse a gradual noising process to create data.1 togetherflow matchingA generative training method that learns an ordinary differential equation mapping noise to data by matching velocity fields along probability paths.1 togetherThe papers
Most central to this idea first, not most recent.
- Big Techcs.CL
Soft tokens improve supervised training for mathematical reasoning
Soft Teacher Forcing improves supervised fine-tuning by mixing ground-truth embeddings with soft tokens from a single forward pass.
Google Deepmind · Oct 2026
- Top Universitycs.CLcode
Latent feedback helps Transformers carry information across generation steps
Pretraining Transformers to predict teacher-derived latent states as well as next tokens, then feeding back their own predicted states at inference, improves language modeling, reasoning, and procedural task performance.
Tel Aviv University, The Hebrew University of Jerusalem · Sept 2026
- Chinese Techcs.CV
Future chunk prediction improves physical plausibility in autoregressive video generation
Adding Future Chunk Prediction as an auxiliary training objective improves physical plausibility in autoregressive video generation without changing inference.
Nanjing University, ARC Lab, Tencent IEG · Oct 2026
- Chinese Techcs.CV
Causal video diffusion matches bidirectional quality without a teacher model
Conditional Residual Prediction reduces causal video models’ dependence on generated history, nearly matching bidirectional quality without a bidirectional teacher.
The Chinese University of Hong Kong, Shenzhen, Tencent Hunyuan · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.