Long-Context Attention Design
Explores how attention head configurations affect long-context retrieval and reasoning in transformers.
10 papers
Where this stands
The written synthesis of this thread is for subscribers. Subscribe.
Results across this thread
6 reported results from the papers in this thread.
The table is for subscribers. Subscribe to see every reported number side by side.
How this thread developed
Oct 2026 · Preferred Networks, Inc.
Language models retrieve information from long texts better with wider attention heads
Demonstrates that wider attention heads significantly improve long-context retrieval.
5 further papers
Nov 2026 · Tianjin University
Backward pass for long-context AI models runs up to 1.8 times faster
Reformulates the backward pass of delta-rule linear attention to speed up training of long-context recurrent attention models.
1 further paper
Nov 2026 · Writer AI Research
Overlapping indexer scan with attention speeds up long-context decoding by up to 14%
Introduces a schedule that overlaps indexer scan with attention to reduce decoding latency in sparse attention transformers.
Nov 2026 · University of California San Diego
AI chatbots fail long tasks due to competing content, not length
Shows that competing content, not length, is the main cause of long-context retrieval failures, informing attention design.
released code, data
4 of 10 papers shown