, "Newton" transmits more to "he" than to "she").
Contextualization principle: informative words are sources, less informative are sinks
The authors validate this principle in language models and reinterpret attention heads as information channels gated by singular vectors.
Top University
Hakaze Cho · Haolin Yang · Zhun Sun · Naoya Inoue · Benjamin Heinzerling · Kentaro Inui
RIKEN · Tohoku University · New York University · JAIST · MBZUAI
Research Digest··3 min read
Cho et al.
Why this paper
From RIKEN and 4 others
In one line
Less informative words absorb contextual information selectively, mediated by attention head singular vectors that gate information flow.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§