When training a smaller language model with help from a larger one, the larger model gives feedback on every word the smaller model writes. This feedback often emphasizes writing style over correct reasoning because style differences are larger.
This new method, SD-OPD, instead compares the student model to its own original pretrained version. It uses this difference to set update strength, and only keeps updates the teacher also supports. This lets training focus on reasoning improvement.
Across several model sizes, SD-OPD improved math reasoning accuracy by 7.76 to 10.37 percentage points over standard . This means better learning without needing larger teachers.