Language models can stretch limited training data by masking inputs

Xu and colleagues report that an auxiliary masked-input training task improves models trained repeatedly on finite datasets, with gains estimated as equivalent to roughly 30% more unique data.

Top University

Zhiwei Xu · Shihao Wu · Hanseul Cho · Wei Hu · Yixin Wang

University of Michigan · KAIST

Research Digest·8 November 2026·2 min read

Thread:Compute-Efficient Scaling

Illustration: Language models can stretch limited training data by masking inputs

In one line

Masked-input regularization improves language model pretraining under data scarcity, and SoftQ scaling law better captures repeated data.

Before you read this

An idea this paper builds on

scaling lawA predictable power-law relationship between compute or data and model performance.29 papers →

Read it as

Training large language models usually needs massive text datasets. But unique text is growing slower than computing power, forcing models to reuse data. This can cause overfitting, where models memorize instead of learning.

The authors studied masked-input regularization (MIR), adding a loss on randomly masked tokens to standard next-token prediction. They also developed SoftQ, a that couples model and data size to better predict performance when data is reused.

Across models up to 1.4 billion parameters, MIR improved validation loss on top of strong weight decay. The improvement was worth about 1.3 times more unique training data. This matters because it helps get more from limited data.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

scaling law masked input regularization data-constrained pretraining weight decay autoregressive pretraining