The authors trained decoder-only language models at two scales (0.6B and 3B parameters) through a three-stage pipeline: pretraining on web and code data, long-context continual pretraining (CPT) with sequences up to 128K tokens, and supervised fine-tuning (SFT) for short-context benchmarks.
Wider attention heads improve long-context retrieval in language models
Under a fixed parameter budget, models with fewer but wider heads outperform those with narrower heads on retrieval tasks.
Industry
Seiya Tokui · Shoichiro Yamaguchi · Sosuke Kobayashi
Preferred Networks, Inc.
Research Digest··2 min read
Tokui et al. trained 0.6B and 3B parameter transformers from scratch, varying only the ratio of attention heads to head dimension while keeping total parameters fixed.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.