Teaching a smaller AI model by having it learn from a larger model's own outputs often works, but the larger model's predictions aren't always the best target. Existing methods that try to improve the target mix together two choices: which direction to adjust the target and how far to go. This can push the target too far from the larger model, hurting learning.
The new method, D^2-OPD, separates these two choices. First, it sets a hard limit on how far the target can stray from the larger model, so the target stays reliable. Second, it decides the adjustment direction by blending two signals: how the larger model changed after training and how different the smaller model initially is from the larger one. The smaller model then learns from this safer, better target.
On three math reasoning tests, D^2-OPD scored the highest across all settings. In the most challenging setup, it outperformed the strongest existing method by 9.45 percentage points. This matters because it allows smaller models to learn more effectively from larger ones, improving reasoning without needing more data or bigger models.