The authors modeled long-context performance as a function of three variables: non-embedding parameter count N, short-context pre-training tokens D_s, and LCCP tokens D_l.
Continuing long-context pre-training keeps improving models, and gains are predictable
The authors derive an empirical scaling law linking long-context performance to model size and separate short- and long-context token budgets, then validate it on a 100B-token training run.
Top University
Ziqing Qiao · Xiaoyue Xu · Sikui Zhang · Yinuo Xu · Bingxiang He · Chaojun Xiao · +1 more
Tsinghua University · OpenBMB
Research Digest··1 min read
Ziqing Qiao, Xiaoyue Xu, and colleagues at Tsinghua University and OpenBMB ask whether long-context continual pre-training (LCCP) yields sustained, predictable improvements as its token budget grows.
§