When training language models, a common trick is to predict several upcoming words at once, not just the next one. This helps the model learn better and speeds up generation. However, the standard way of doing this mixes in a noisy signal that can hurt learning, especially as models grow larger.
The authors found a way to separate the useful teaching signal from the noise, and they recover the useful part without extra memory cost. Their method, called Soft-MTP, only needs a one-line code change and works with existing training frameworks.
In tests with models ranging from 167 million to 1.2 billion parameters, Soft-MTP consistently outperformed the standard approach: lower loss, better accuracy, and faster text generation. It even matched or beat a more expensive method, so it's a free improvement for anyone using .