The authors formulate language modeling with latent auxiliary variables that carry information between predicted tokens, so that one forward pass yields several mutually consistent tokens.
Parallel language models match autoregressive quality while decoding several tokens per pass
Draxler et al. train models up to 3B parameters and measure 2.3 to 5.7 times faster inference on math, coding and instruction-following benchmarks.
Top University
Felix Draxler · Robin C. Geyer · Daniel Geyfman · Farrin Marouf Sofian · Justus Will · Philipp Hildebrandt · +10 more
University of California, Irvine · ETH Zurich · Charité Berlin · Hasso Plattner Institute
Research Digest··2 min read
The authors introduce Parallel Language Models (PLMs), which generate several mutually consistent tokens in one forward pass by routing dependencies through latent auxiliary variables.
§