of large language models typically faces a scaling wall where additional serial compute yields little improvement. This paper shows that instead of extending a single training run, one can allocate the same compute budget across multiple independent trajectories, each starting from the same pretrained checkpoint but with controlled recipe variations. The best checkpoints from each trajectory are selected and averaged, first within each trajectory and then across trajectories. This approach, called Trajectory Soup, outperforms the best single-trajectory average under matched budgets and continues to improve as more trajectories are added. The gains persist after subsequent fine-tuning, suggesting a practical way to overcome mid-training saturation.
Averaging checkpoints across shorter training runs beats one long mid-training run
The authors show that distributing a fixed mid-training budget over several forked branches and merging their strongest checkpoints yields downstream gains where serial training has saturated.
Chinese Tech
Zhehao Huang · Changxin Tian · Qingyuan Yang · Kunlong Chen · Ziqi Liu · Zhiqiang Zhang · +2 more
Ant Group · Shanghai Jiao Tong University
Research Digest··2 min read
In one line
Distributing mid-training compute across multiple independent training trajectories improves LLM performance more than a single long run.
Before you read this
An idea this paper builds onRead it as
The authors propose Trajectory Soup, a strategy that forks multiple independent branches from a shared pretrained checkpoint, averages the strongest checkpoints within each branch, and then averages the resulting anchors across branches.
The authors frame compute allocation as a choice between trajectory count and trajectory length.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.