High-quality text is running out, so language models will soon be trained by passing over the same data many times. Reusing data keeps fitting it but also bakes in noise, so validation loss (error measured on held-out text) eventually stops improving and climbs again.
Researchers trained small language models on fixed text and built a mathematical theory of the process, tracking how errors correlate across repeated passes and across separately trained models.
They found that averaging the output scores of several independently trained models (logit averaging) suppresses that late rise in error. Validation loss bottomed out after roughly nine to fourteen passes, later as ensembles grew, and averaged models beat single wider or deeper ones at equal budget.