Pretraining Repetition Scaling
Investigates the scaling rules that govern the diminishing returns of repeating training tokens during language model pretraining.
2 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.