rotary position embedding
Rotary Position Embedding (RoPE) gives language models a sense of token order by rotating each token's representation by a position-dependent angle. It uses fast rotations for nearby tokens and slow rotations for distant ones, enabling the model to capture both short and long-range relationships.
These papers show RoPE has shortcomings in long contexts. One work replaces its slow bands with data-dependent coordinates to improve performance. Another diagnoses two failure modes, semantic reversal and positional insensitivity, and offers a lightweight fix. A third method removes key-side RoPE in a compression scheme to speed up decoding.
- Papers
- 3
- Released code
- 0
- First seen
- Oct 2026
- Latest
- Oct 2026
3 papers in the last two months, against 0 in the two before.
Start here
Two papers to read first, in the order they came out- 1Data-aware rotary encoding reduces recency bias across diverse sequence tasks
DaRoPE replaces RoPE's slow frequency bands with data-learned coordinates to reduce recency bias.
Oct 2026 - 2Spectral theory diagnoses two failure modes in RoPE long-context models
RoPE's high-frequency components cause a tradeoff between semantic stability and positional sensitivity that can be measured and mitigated by frequency rescaling without training.
Oct 2026
How fast it is moving
Often studied with
mechanistic interpretabilityThe study of how neural networks compute through internal circuits, features, and causal interventions, rather than just input-output behavior.2 togetherfailure attributionIdentifying which component or step in a system caused an observed failure, often using structured trace representations.1 togetherkv cache compressionReducing the size of the key-value cache in transformers via selective retention, low-rank updates, or adaptive compression.1 togetherThe papers
Most central to this idea first, not most recent.
- Academiccs.LG
Spectral theory diagnoses two failure modes in RoPE long-context models
RoPE's high-frequency components cause a tradeoff between semantic stability and positional sensitivity that can be measured and mitigated by frequency rescaling without training.
Independent Researcher, University of Illinois at Urbana-Champaign · Oct 2026
- Big Techcs.CL
Data-aware rotary encoding reduces recency bias across diverse sequence tasks
DaRoPE replaces RoPE's slow frequency bands with data-learned coordinates to reduce recency bias.
Meta AI, Inria · Oct 2026
- Chinese Techcs.LG
Joint token-feature KV cache compression speeds long-context LLMs
SlimKV jointly compresses tokens and features in KV cache, achieving high compression with reconstruction-free attention and speedups.
University of Science and Technology of China, Nanyang Technological University · Oct 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.