← All concepts

mid-training

Mid-training is a stage in language model development where a model that has already been pretrained on broad web data is further trained on smaller, curated datasets. This phase aims to improve performance on specific tasks without the computational cost of full retraining.

The papers reveal that enhancing models after pretraining requires deliberate design. For mid-training itself, using multiple independent training trajectories or aligning data schedules with learning rate decay can overcome plateaus and reduce loss. Organizing training data around key capabilities like evidence grounding yields broad gains. Alternatively, leveraging model failures for data selection or adding modular components can also improve performance without extensive retraining.

Papers
5
Released code
0
First seen
Oct 2026
Latest
Oct 2026

5 papers in the last two months, against 0 in the two before.

Start here

Two papers to read first, in the order they came out
  1. 1Averaging checkpoints across shorter training runs beats one long mid-training run

    Distributing mid-training compute across multiple independent training trajectories improves LLM performance more than a single long run.

    Oct 2026
  2. 2Compiling agent trajectories into targeted long-context tasks boosts model capabilities

    TrajLong compiles agent trajectories into long-context training tasks targeting evidence grounding, cross-evidence aggregation, and temporal state maintenance, improving LLM agent performance.

    Oct 2026

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.