The authors analyzed 22 open-weight models spanning eight families using mechanistic interpretability to classify retrieval mechanisms (positional, lexical, and reflexive).
Positional encoding choice shifts retrieval from positional to semantic mechanisms
Hybrid architectures like SWA NoPE rely more on semantic retrieval than standard RoPE, altering long-context performance.
Academic
Eric Enouen · Sainyam Galhotra
Cornell University
Research Digest··2 min read
The authors studied how positional encoding (PE) choice shapes the internal mechanisms language models use for in-context retrieval across 22 open-weight models from eight families.
Why this paper
From Cornell University
In one line
Hybrid positional encodings shift language models from positional to semantic in-context retrieval, trading off performance.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§