The paper introduces SemanTok, a flexible video tokenizer for autoregressive video generation. Unlike fixed-size tokenizers, SemanTok produces a variable-length token sequence where early tokens capture global semantics and later tokens add detail. This is achieved by feeding frozen DINOv2 features into the encoder and using lightweight heads to reconstruct those features from each token prefix.
SemanTok demonstrates strong performance across multiple autoregressive model sizes. A 201M parameter model matches or exceeds a VideoFlexTok model 3.4 times larger, and larger SemanTok models further improve fidelity. The method also maintains semantic alignment on out-of-distribution classes and provides better generation fidelity from shorter token prefixes, which are cheaper to predict.
This work is interesting because it addresses the efficiency challenge in video generation by allowing a variable token budget. The same prefix that is more semantic is also easier for an autoregressive model to predict, effectively separating semantic content from pixel detail and reducing computational cost.