The paper introduces Data-aware RoPE (DaRoPE), a positional encoding that modifies (RoPE) by keeping its fast frequency bands unchanged but replacing the slow bands—whose wavelengths exceed the training context—with bounded coordinates learned from contextual token representations. This makes the encoding depend on data content rather than just token distance.
The authors compare DaRoPE against several existing positional encodings across synthetic tasks, symbolic music, genomics, neural signals, and language models from 124M to 50B parameters. DaRoPE leads on non-text benchmarks, mitigates recency bias, and matches or exceeds RoPE in language modeling and length extrapolation while adding negligible parameters and maintaining FlashAttention compatibility.