The authors compared repacked text with randomly sampled natural long documents, then tested several selection pipelines on the same long-document candidate pool.
Documents with long-range dependencies improve language models’ context training
Selecting natural documents by dependency-sensitive token statistics improved long-context performance more reliably than generic quality filters or LLM judgments.
Chinese Tech
Junlong Jia · Jiang Zhou · Xing Wu · Chaochen Gao Tinghao Yu · Feng Zhang · Lilin Wang
Hunyuan Team, Tencent
Research Digest··2 min read
Thread:Long-Context Training Data
Jia et al.
§