The authors fit a mathematical relationship predicting model loss, a measure of text-prediction error, from model size, training tokens, rewrite fraction and the number of passes over natural data.
Scaling laws predict when synthetic rewrites improve language model training
Roberts and colleagues account for rewrite-generation costs and show that models trained with rewritten text can outperform their rewriter on held-out text prediction.
Top University
Nicholas Roberts · Ahmed Ahmed · Suhas Kotha · Percy Liang · Yu Wang · Alex Renda · +1 more
Princeton University · Stanford University · Jane Street
Research Digest··2 min read
Roberts and colleagues develop a scaling law to predict how much rewritten text to include in language model pretraining, counting the computation needed to generate it.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.