The authors trained four model sizes, from 140 million to 1.3 billion parameters, on two text corpora, using three random seeds for most configurations.
Halting heads can overstate compute savings in looped language models
Wu and colleagues find that training objectives, stopping rules and batch execution substantially reduce the apparent benefits of adaptive iteration counts.
Big Tech
Tsung-Han Wu · Minsuk Heo · Kenny Tan
Apple
Research Digest··3 min read
Wu and colleagues pretrained looped language models with up to 1.3 billion parameters to test whether halting heads, which decide when tokens stop receiving computation, deliver their promised savings.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.