Long-context language models often ignore information in the middle of a long input, a problem called "lost-in-the-middle." This happens partly because the models' internal position encoding (RoPE) weakens signals between distant tokens, making middle content harder to retrieve.
The researchers developed QGPM, a method that reads the model's own attention to identify which parts of the input are relevant to a given query. It then adjusts position encoding on fast-rotating planes to shift low-relevance tokens slightly forward, effectively pulling relevant evidence closer to the query without deleting or reordering any text.
On benchmarks, QGPM improved average accuracy over strong baselines, with the largest gains on retrieval tasks where finding specific information matters most. This method requires no training and works across different language models, offering a practical way to help models use their full context window more effectively.