The authors provide a theoretical analysis of top-k knowledge distillation, where practitioners retain only the teacher's top-k probabilities, renormalize them, and minimize a divergence on that subset.
Top-k distillation's renormalized gradient ignores tail mass
Khanbayov and Kurban prove that renormalization only normalizes, not safeguards, leaving student tail mass unconstrained.
Independent
Rasul Khanbayov · Hasan Kurban
Research Digest··3 min read
Khanbayov and Kurban rigorously show that the renormalized top-k distillation objective has zero gradient along directions controlling tail-mass variation, so a student can match the teacher on shown tokens while its tail mass drifts arbitrarily.
Why this paper
Independent · Released code
In one line
In top-k distillation, renormalizing the teacher's retained probabilities does not expose tail mass, so the student's gradient is exactly zero along tail-moving directions.
What it released
Code
What we could check
- ✓Code link in the paper (github.com)
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§