Training large language models on limited data is challenging. Using more computing can backfire. Prior methods froze settings like weight decay (a way to avoid overfitting) and ignored ensembling (combining multiple models), causing performance to drop as compute increased.
The researchers created formulas to adjust weight decay with model size and to add ensemble members. They tested on 100 million tokens of web text.
Their recipe reduced validation loss by over 0.15 nats (a 14% drop in perplexity, a prediction error measure) and improved steadily with more compute. This provides a practical guide for better models when data is scarce.