The authors analyze the long-term generation process of autoregressive transformers as the number of generated tokens tends to infinity.
Transformer text generation may never converge, but temperature ensures it
For a wide class of multi-layer transformers, the authors prove there exists a temperature above which generation converges to a unique distribution, though no single temperature works for all models.
Big Tech
Katy Blumer · Da-Cheng Juan · Jon Kleinberg · Ravi Kumar · Cyrus Rashtchian · Andrew Tomkins
Google Research
Research Digest··3 min read
The authors study the asymptotic behavior of autoregressive transformers generating text indefinitely.
Why this paper
From Google Research
In one line
Within a broad fixed-depth transformer class, sufficiently high model-specific softmax temperature guarantees long-run cache convergence, while lower temperatures can permit persistent nonconvergence.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§