delta rule
- Papers
- 3
- Released code
- 0
- First seen
- Oct 2026
- Latest
- Nov 2026
3 papers in the last two months, against 0 in the two before.
Start here
The paper to read first- 1Backward pass for long-context AI models runs up to 1.8 times faster
ReWY reduces delta-rule attention's backward local matrix products from seven to three, speeding complete backward passes 1.15 to 1.81 times on NVIDIA GPUs.
Nov 2026
How fast it is moving
Often studied with
linear attentionAn attention mechanism that computes attention with linear, rather than quadratic, complexity in sequence length.2 togetherhybrid attentionCombining different attention types (e.g., linear and full) or positional/semantic strategies in one model to balance efficiency, accuracy, and memory.2 togetherThe papers
Most central to this idea first, not most recent.
- Academiccs.LG
Backward pass for long-context AI models runs up to 1.8 times faster
ReWY reduces delta-rule attention's backward local matrix products from seven to three, speeding complete backward passes 1.15 to 1.81 times on NVIDIA GPUs.
Tianjin University · Nov 2026
- Chinese Techcs.LG
Two-sided associative memory correction boosts linear attention performance
Gated Slot Attention-2 corrects both key and value memory in linear attention using complementary update rules.
The Hong Kong University of Science and Technology (Guangzhou), Tencent · Oct 2026
- Big Techcs.LG
Language models can match attention’s prediction quality with fixed-size memory
Cascaded linear memories trained via target propagation match attention and outperform existing linear and nonlinear recurrent memories.
Google Research, University of Southern California · Nov 2026
Concepts are extracted from each paper and reused across the corpus, so this page grows on its own as the desk reads.