The authors reformulate masked discrete diffusion to incorporate a hierarchy level for every token position, available with zero or minimal extra annotation.
Masked diffusion language models improve by encoding position priority as hierarchy levels
The authors' HMDLM factorizes generation into hierarchy prediction and token prediction, with level-specific noise schedules, and beats strong baselines on language and reasoning tasks.
Big Tech
Parul Gupta · Satwik · Avni Aggarwal · Bhavesh Gurnani · Arnab Kumar Mondal · Dinesh Khandelwal · +2 more
Indian Institute of Technology Delhi · IBM Research
Research Digest··3 min read
Gupta et al.
§