The authors constructed causal attention masks from point-hyperplane incidences over finite fields, with row-support VC dimension d as a tunable parameter.
Tunable attention masks enable subquadratic long-context sequence modeling
By controlling the VC dimension of attention mask row supports, the authors achieve linear or near-linear computation and constant-time streaming decoding.
Top University
Emile Anand · Abdullah Ateyeh · Archer Wang · Marin Soljačić
Georgia Institute of Technology · University of California, Berkeley · Massachusetts Institute of Technology
Research Digest··2 min read
The authors introduce Structured Matrix Attention (SMat-Attention), a family of causal attention masks parametrized by VC dimension d.
Why this paper
From University of California, Berkeley and 2 others
In one line
SMat-Attention uses VC dimension as a tunable parameter to control long-range token routing for efficient sequence modeling.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§