Clean replaces explicit gradient covariance accumulation with randomized Nyström approximations of the left and right Shampoo preconditioners.
Nyström sketching enables second-order optimizer with linear memory cost
Clean and its quantized variant Q-Clean achieve SOAP-level performance while using less memory than AdamW, and enable training a 13B model on a single 80GB GPU.
Top University
Beheshteh T. Rakhshan · Sahar Rajabi · Maziar Sargordi Shikai Fang · Guillaume Rabusseau · Sirisha Rambhatla
Université de Montréal · University of Waterloo · Independent Researcher · Zhejiang University · Canada CIFAR AI Chair
Research Digest··2 min read
The authors introduce Clean, a second-order optimizer that uses randomized Nyström sketching to approximate the preconditioners in SOAP, reducing memory complexity from quadratic to linear in model dimensions.
Why this paper
From University of Waterloo and 4 others
In one line
Clean uses Nyström-sketched curvature to achieve SOAP-like LLM optimization with linear-memory states, using less memory than AdamW and reaching its final performance 26% faster.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§