mid-training
Mid-training is a stage in language model development where a model that has already been pretrained on broad web data is further trained on smaller, curated datasets. This phase aims to improve performance on specific tasks without the computational cost of full retraining.
The papers reveal that enhancing models after pretraining requires deliberate design. For mid-training itself, using multiple independent training trajectories or aligning data schedules with learning rate decay can overcome plateaus and reduce loss. Organizing training data around key capabilities like evidence grounding yields broad gains. Alternatively, leveraging model failures for data selection or adding modular components can also improve performance without extensive retraining.
- Papers
- 5
- Released code
- 0
- First seen
- Oct 2026
- Latest
- Oct 2026
5 papers in the last two months, against 0 in the two before.
Start here
Two papers to read first, in the order they came out- 1Averaging checkpoints across shorter training runs beats one long mid-training run
Distributing mid-training compute across multiple independent training trajectories improves LLM performance more than a single long run.
Oct 2026 - 2Compiling agent trajectories into targeted long-context tasks boosts model capabilities
TrajLong compiles agent trajectories into long-context training tasks targeting evidence grounding, cross-evidence aggregation, and temporal state maintenance, improving LLM agent performance.
Oct 2026
How fast it is moving
Often studied with
agent skillsLearned capabilities that enable AI agents to perform specialized tasks with improved effectiveness.1 togethercontinued pretrainingAdding new domains or modalities to an existing pre-trained model by training it further on curated data beyond its initial pre-training.1 togetherfailure attributionIdentifying which component or step in a system caused an observed failure, often using structured trace representations.1 togetherThe papers
Most central to this idea first, not most recent.
- Chinese Techcs.CL
Compiling agent trajectories into targeted long-context tasks boosts model capabilities
TrajLong compiles agent trajectories into long-context training tasks targeting evidence grounding, cross-evidence aggregation, and temporal state maintenance, improving LLM agent performance.
Hong Kong University of Science and Technology (Guangzhou), Tencent · Oct 2026
- Chinese Techcs.LG
Averaging checkpoints across shorter training runs beats one long mid-training run
Distributing mid-training compute across multiple independent training trajectories improves LLM performance more than a single long run.
Ant Group, Shanghai Jiao Tong University · Oct 2026
- Top Universitycs.LG
When specialized training data helps depends on the learning rate
Optimal training requires designing data and learning rate schedules jointly, as their interaction affects final loss.
Tsinghua University, Peng Cheng Laboratory · Oct 2026
- Academiccs.RO
Stereo module lifts 2D vision-language-action models to 3D for robot manipulation
ExStereo augments pre-trained 2D Vision-Language-Action models with 3D perception using stereo vision, improving robotic manipulation success rates.
University of Southern California · Oct 2026
- Top Universitycs.CL
Language model RL failures can guide targeted pretraining repairs
Failure-linked n-grams in RL rollouts can retrieve targeted pretraining data that improves reasoning more than equal-sized randomly sampled data.
Princeton Language and Intelligence · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.