Data-aware rotary encoding reduces recency bias across diverse sequence tasks
Lévy and colleagues identify slow rotary frequency bands, whose wavelengths exceed the training context, as a weakness of Rotary Position Embedding during length extrapolation. Their alternative, Data-aware RoPE, performs best or comparably to other tested encodings across language models, synthetic tasks, music, genomics and neural signals, while reducing the tendency to favor recent tokens.
