← All concepts

rotary position embedding

Rotary Position Embedding (RoPE) gives language models a sense of token order by rotating each token's representation by a position-dependent angle. It uses fast rotations for nearby tokens and slow rotations for distant ones, enabling the model to capture both short and long-range relationships.

These papers show RoPE has shortcomings in long contexts. One work replaces its slow bands with data-dependent coordinates to improve performance. Another diagnoses two failure modes, semantic reversal and positional insensitivity, and offers a lightweight fix. A third method removes key-side RoPE in a compression scheme to speed up decoding.

Papers
3
Released code
0
First seen
Oct 2026
Latest
Oct 2026

3 papers in the last two months, against 0 in the two before.

Start here

Two papers to read first, in the order they came out
  1. 1Data-aware rotary encoding reduces recency bias across diverse sequence tasks

    DaRoPE replaces RoPE's slow frequency bands with data-learned coordinates to reduce recency bias.

    Oct 2026
  2. 2Spectral theory diagnoses two failure modes in RoPE long-context models

    RoPE's high-frequency components cause a tradeoff between semantic stability and positional sensitivity that can be measured and mitigated by frequency rescaling without training.

    Oct 2026

The papers

Most central to this idea first, not most recent.

Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.

rotary position embedding · Research concepts | Zotpaper