The authors separate RoPE’s frequency bands according to their wavelength.
Data-aware rotary encoding reduces recency bias across diverse sequence tasks
DaRoPE modifies RoPE’s slow frequency bands using bounded, context-dependent coordinates while retaining its standard treatment of faster bands.
Big Tech
Jarod Lévy · Mathurin Videau · Jad Yehya · Jean-Rémi King · Stéphane d'Ascoli · Thomas Moreau
Meta AI · Inria · Université Paris-Saclay
Research Digest··2 min read
Lévy and colleagues identify slow rotary frequency bands, whose wavelengths exceed the training context, as a weakness of Rotary Position Embedding during length extrapolation.
Why this paper
From Meta AI and 2 others
In one line
DaRoPE replaces RoPE's slow frequency bands with data-learned coordinates to reduce recency bias.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§