333B to 3B tokens, repeating epochs up to240 times.
Smaller batches and adapted recipes improve repeated-data pretraining
The authors show that optimizing batch size and training recipes for repeated epochs on fixed data yields substantial loss improvements, and that common GPU parallelism practices can be suboptimal under time constraints.
Top University
Sewon Min
UC Berkeley · Alien Institute for AI
Research Digest··2 min read
Thread:Optimizer Batch Scaling
This paper studies language model pretraining when unique data is finite and models must train repeatedly on the same data.
Why this paper
From UC Berkeley and Alien Institute for AI · Part of Optimizer Batch Scaling, now 3 papers
In one line
For repeated-data pretraining, smaller batches, repacked sequences, tied embeddings, and positive embedding weight decay improve validation loss; GPUs and time should be treated separately.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§