Long-context language models increasingly use , which keeps a running state instead of storing every past token's attention weight. Training needs a backward pass, the step that computes weight updates; for delta-rule attention, that pass does avoidable matrix arithmetic.
The authors recast the backward computation as one triangular residual system, an algebraic rearrangement that shares an intermediate result and removes an unnecessary matrix inverse, and fused two gate computations. The model and its exact gradients are unchanged.
On NVIDIA graphics processors, complete backward passes ran 1.15 to 1.81 times faster across six shapes at the same precision. That matters because these operators fill many layers of current models, so training gets cheaper without redesigning the network.