The authors construct a self-supervised ridge-regression problem from input tokens, requiring no external labels.
Fitting regression models to tokens improves compression in two settings
Yang and colleagues report better long-context retrieval and comparable multimodal performance with 55.6% fewer visual tokens using input-specific regression coefficients as compressed representations.
Chinese Tech
Sihan Yang · Jie Huang · Hanxuan Li · Yifan Guo · Weigao Sun · Yiran Zhong · +1 more
The Chinese University of Hong Kong · Independent Researcher · Token Foundry · Alibaba Group
Research Digest··2 min read
Yang and colleagues introduce TTR-Compressor, which compresses tokens by fitting a regression problem to each input rather than pooling token features.
§