The authors ask whether self-distillation can get better teacher predictions not by giving the teacher extra context or instructions, but by changing the order in which the model executes its own layers.
A model can become its own stronger teacher by looping selected layers
LoOPSD searches for repeated layer blocks, then distills the looped model's token-level predictions back into the original student, improving reasoning accuracy across model families.
Chinese Tech
Wei Fan · Qingyue Yang · Chao-Hong Tan · Qian Chen · Wen Wang · Xiangang Li · +3 more
Anhui Province Key Laboratory of Digital Security · Token Foundry, Alibaba Group · Independent Researcher
Research Digest··2 min read
The authors introduce LoOPSD, an on-policy self-distillation method that constructs a stronger teacher from the same model by repeating selected layer blocks during inference.
Why this paper
From Token Foundry, Alibaba Group and 2 others
In one line
LoOPSD uses repeated layers as a teacher to improve student model reasoning through on-policy self-distillation.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§