The authors analyze how rounding rules for 4-bit second-moment quantization affect AdamW's update dynamics.
Redesigning 4-bit AdamW quantization in preconditioner space reduces validation loss gap
Two methods, ZIP-SR and ZE-EDEN, cut the loss gap to full-precision AdamW by up to 70% across models from 130M to 2.7B parameters.
Top University
Hanyang Li · Shao Tang · Daniel Thomas Braithwaite · Gregory Dexter · Leonardo Neves · Aman Gupta · +4 more
University of California, Berkeley · Nubank
Research Digest··3 min read
Li et al.
Why this paper
From University of California, Berkeley and Nubank · Released code
In one line
Preconditioner-space rounding for 4-bit AdamW state quantization reduces validation loss gaps by up to 70% versus TorchAO.
What it released
Code
What we could check
- ✓Code link in the paper (github.com)
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§