Standard teacher forcing trains a model to predict each token while conditioning on the preceding ground-truth tokens.
Soft tokens improve supervised training for mathematical reasoning
Soft Teacher Forcing mixes ground-truth embeddings with model-derived continuous tokens, improving results across four mathematics benchmarks and seven language models.
Big Tech
Mohammad Fahes · Cordelia Schmid · Anurag Arnab
Google Deepmind
Research Digest··2 min read
Fahes, Schmid and Arnab introduce Soft Teacher Forcing, a fine-tuning method that partly replaces ground-truth token embeddings with continuous representations derived from the model’s own predictions.
Why this paper
From Google Deepmind
In one line
Soft Teacher Forcing improves supervised fine-tuning by mixing ground-truth embeddings with soft tokens from a single forward pass.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (3 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§