The authors address cross-tokenizer on-policy distillation, where a student model learns from a teacher with a different tokenizer.
Uncertainty-weighted credit improves cross-tokenizer distillation for LLMs
Authors' CAST framework redistributes chunk-level teacher advantages across student tokens and anchors unaligned positions, outperforming prior baselines on math, science, and code benchmarks.
Chinese Tech
Jingru Yang · Jinhang Wu · Minjing Shi · Shengfeng He · Xuyan Li · Man Tan · +1 more
Taobao & Tmall Group of Alibaba · Singapore Management University
Research Digest··2 min read
Jingru Yang and colleagues propose CAST, a shift-or-stay method for cross-tokenizer on-policy distillation.
Why this paper
From Taobao & Tmall Group of Alibaba and Singapore Management University
In one line
CAST improves cross-tokenizer distillation by assigning credit per student token using uncertainty and local teacher smoothness.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§