The authors formalise autoregressive decoding as a regularised optimisation problem on the probability simplex.
Decoding as optimisation on the probability simplex unifies and composes sampling strategies
The authors formalise decoding as a regularised optimisation problem, recovering standard methods as special cases and enabling new decoders through compositional regularisers.
Chinese Tech
Xiaotong Ji · Ahmed Khaled Khamis · Rasul Tutunov · Matthieu Zimmer · Haitham Bou-Ammar
Huawei Noah's Ark Lab · UCL Centre for AI
Research Digest··3 min read
The authors reframe large language model decoding as an optimisation problem on the probability simplex, where the goal is to maximise expected model score subject to regularisers that encode distributional preferences.
Why this paper
From Huawei Noah's Ark Lab and UCL Centre for AI
In one line
Decoding is regularised optimisation over next-token distributions; composing regularisers yields trade-offs in quality and diversity unavailable from individual decoding objectives.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§