← All concepts

teacher forcing

During training, a model often learns to predict the next step by being given the correct previous steps, this is teacher forcing. It is like a student copying from a perfect answer key rather than working from their own earlier answers. While it speeds up learning, it can cause problems later because the model never practices recovering from its own mistakes.

The papers explore ways to soften or bypass these problems. One approach mixes a soft, model-generated hint with the ground truth to reduce over‑dependence on perfect history. Another trains the model to also predict a richer hidden state from a teacher, letting it learn to reuse its own outputs. For video generation, auxiliary objectives or residual predictions force the model to look beyond local shortcuts and consider genuine temporal evolution, improving quality without extra cost at inference.

Papers
4
Released code
1
First seen
Sept 2026
Latest
Oct 2026

4 papers in the last two months, against 0 in the two before.

Start here

Two papers to read first, in the order they came out
  1. 1Latent feedback helps Transformers carry information across generation steps

    Pretraining Transformers to predict teacher-derived latent states as well as next tokens, then feeding back their own predicted states at inference, improves language modeling, reasoning, and procedural task performance.

    Sept 2026
  2. 2Soft tokens improve supervised training for mathematical reasoning

    Soft Teacher Forcing improves supervised fine-tuning by mixing ground-truth embeddings with soft tokens from a single forward pass.

    Oct 2026

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.