The authors developed a theory quantifying two failure modes of (RoPE) in long-context models: semantic reversal, where position changes can reverse token preferences, and positional insensitivity, failure to distinguish adjacent positions. They introduced RoPE Profiler, a diagnostic toolkit that reuses cached query and key activations to measure these failures with Semantic and Positional scores. Across 49 long-context tasks, reasoning tasks predominantly suffer semantic reversal, while retrieval tasks suffer positional insensitivity. Guided by diagnostics, targeted high-frequency rescaling without retraining improved accuracy by up to 20 percentage points on Qwen3-8B and 25 points on Llama-3.1-8B-Instruct. This work provides a principled diagnostic and lightweight mitigation for long-context failures in widely used RoPE-based models.
Spectral theory diagnoses two failure modes in RoPE long-context models
Wu et al. derive a quantitative bound on context length and introduce a diagnostic toolkit that reuses cached activations to measure semantic reversal and positional insensitivity, enabling training-free gains of up to 25 points.
Independent Researcher · University of Illinois at Urbana-Champaign
In one line
RoPE's high-frequency components cause a tradeoff between semantic stability and positional sensitivity that can be measured and mitigated by frequency rescaling without training.
Before you read this
Three ideas this paper builds onThe authors developed a theoretical framework that relaxes prior assumptions of equal scales across RoPE frequency channels, better reflecting trained models.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.