The authors construct a transformer whose forward pass exactly implements Lloyd’s algorithm, the standard iterative procedure that alternates between assigning points to their nearest centroid and recomputing centroids.
Smaller transformers can learn and execute k-means clustering
The authors compress cluster labels into binary codes, reducing transformer width while retaining an exact construction for Lloyd’s algorithm.
Big Tech
Charlotte Park · Kenneth L. Clarkson · Lior Horesh · Takuya Ito · Parikshit Ram
IBM Research · MIT
Research Digest··2 min read
Park et al.
Why this paper
From IBM Research and MIT
In one line
A smaller transformer with embedding dimension d+ceil(log2 k) can exactly execute Lloyd's k-means algorithm.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§