The authors propose FreeSolo, a native sparse attention with two branches: a short-range sliding window using RoPE within a fixed local context, and a long-range branch that selects keypoint cells via learned token key summaries (mean and maximum norm).
Sparse attention mechanism achieves zero-shot extrapolation to longer sequences
FreeSolo enables retrieval at 256× training length without long-context fine-tuning through a combination of short-range sliding window and long-range selection of keypoint cells.
Top University
Zekun Shi · Alastair James Arthur Price
National University of Singapore · University of Toronto · Sea AI Lab
Research Digest··2 min read
Zekun Shi and Alastair James Arthur Price introduce FreeSolo, a sparse attention mechanism that uses mean summaries of learned token keys for hierarchical search and maximum token-key norms for pruning.
Why this paper
From National University of Singapore and 2 others
In one line
FreeSolo achieves zero-shot length extrapolation with linear-time sparse attention.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ✓Compute or model size stated (params 800M)
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§