Fixed-depth RVQ tokenizers allocate equal capacity per frame, wasting bits where refinements add little value. UniAdapt learns to predict the marginal acoustic utility of activating each additional RVQ layer per frame, then uses a causal primal-dual allocator to select depths under exact serialized-bit budgets. Across speech, music, and environmental audio, it reduces Log-STFT distortion by 1.07-4.39% at the same bitrate. Causal allocation improves three of four speech rates with zero budget violations and runs faster than real time. A listening test shows significant speech quality improvement (3.52 points) while music and environmental audio remain comparable.
Marginal utility allocation of audio tokens improves quality under strict bit budgets
UniAdapt learns where RVQ refinements are most valuable, reducing distortion across speech, music and environmental audio without increasing bitrate.
Chinese Tech
Mingyu Zhao · Jinchao Zhang · Zhiyong Wu
Tsinghua University · Tencent
Research Digest··2 min read
In one line
UniAdapt learns marginal utility of RVQ refinements to allocate variable depths under fixed bit budgets, reducing distortion without increasing bitrate.
Read it as
The authors introduce UniAdapt, a method that learns the marginal utility of residual-vector-quantization (RVQ) refinements on a frozen codec and allocates tokens under exact serialized-bit budgets.
Zhao et al. start from a frozen 48 kHz RVQ codec (UniStream) and train a causal controller to predict the acoustic marginal utility of each refinement layer per frame; an optional semantic head predicts semantic utility for speech utterances.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.