The authors first conducted controlled experiments on a two-stage pretraining setup for knowledge and mathematics.
Loss reweighting improves LLM pretraining without altering data exposure.
A new method, LARE, automatically adjusts domain loss weights based on running statistics, yielding gains of up to 17 points on math benchmarks.
Independent
Jiazheng Li · Huaqing Zhang · Jing Xu · Tianle Cai · Jingzhao Zhang
Research Digest··2 min read
The authors demonstrate that adjusting the contribution of each domain's loss during pretraining, rather than changing data mixtures, can enhance joint capabilities.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.