The authors compared ways of allocating the same mid-training token budget over a fixed amount of continued training.
When specialized training data helps depends on the learning rate
Under fixed compute and specialized-data budgets, Wen and colleagues find that changing the learning-rate schedule can reverse which data-mixing strategy performs best.
Top University
Haodong Wen · Kairong Luo · Tingkai Yan · Wenguang Chen · Kaifeng Lyu
Tsinghua University · Peng Cheng Laboratory
Research Digest··3 min read
Wen and colleagues investigate when language models should encounter the smaller, curated datasets used in mid-training, a phase following broad pretraining.
§