When training a smaller language model to imitate a larger one, a standard method is to have the small model generate its own text and then learn from the teacher's corrections on that text. This is called . However, this method only teaches the small model how to improve paths it already takes, missing the better paths the teacher would have chosen.
The new method, called CoRD (coupled-rollout distillation), changes how the training text is generated. Instead of letting the small model generate its own text alone, CoRD blends the small model's choices with the teacher's likely choices. It keeps the small model's token when both agree, but replaces it with a token from the teacher when they disagree. This creates training text that includes states the teacher would reach, while changing as little of the student's own text as possible.
Across six small models from two model families, CoRD improved accuracy on five math benchmarks by 0.9 to 2.0 percentage points on single answers and 1.2 to 3.4 percentage points on best-of-eight answers compared to the standard on-policy method. The improvement was largest on the hardest problems the student had never solved before, and it required no extra data or training steps.