Reinforcement learning (RL) post-training often makes models longer-winded even on easy, already-solved queries. This wastes tokens and is called the length-scaling tax (LST). The authors propose Length Self-Distillation (LSD), which routes solved prompts to using an exponential moving average of the policy as a teacher, while keeping the original RL objective for unsolved prompts. This preserves concise behavior on easy queries without restricting exploration on hard ones. LST dropped from 19.0% to -3.7% on single-turn reasoning and from 31.4% to 13.7% on multi-turn agentic tasks, while accuracy was maintained or improved. LSD uses no external model and provides a practical way to control response length during RL training, which matters for deployment efficiency as reasoning models become more common.
Online distillation cuts needless verbosity in RL-trained models
Length Self-Distillation reduces excess response length on easy queries without sacrificing accuracy on hard ones.
Chinese Tech
Xu Wan · Wenyue Xu · Shengjie Zhao · Mingyang Sun
ByteDance Seed · Tongji University · Peking University
Research Digest··3 min read
Thread:Agent Harness Optimization
In one line
Length Self-Distillation reduces excess response length on already-solved queries without sacrificing accuracy.
Before you read this
Three ideas this paper builds onon-policy distillationA training method where a student model learns from a teacher using its own current policy's outputs, often improving reasoning and instruction following.131 papers →reinforcement learning with verifiable rewardsReinforcement learning that uses rewards from objectively checkable signals like correct answers or verification feedback.163 papers →group relative policy optimizationA reinforcement learning method that optimizes policy by computing advantages based on group-level reward statistics.188 papers →
Read it as
The authors identify and quantify the 'length-scaling tax'—excess response length on already-solved problems that arises during reinforcement-learning (RL) post-training.
The authors formalize the length-scaling tax (LST) as the percentage increase in response length on queries the policy already answers correctly after RL training.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.