The paper introduces Gated Slot Attention-2 (GSA2), which improves memory management in models by applying two complementary correction rules to a shared latent-slot memory. The first stage uses Gated Oja Rule-2 to correct key-to-slot associations, while the second uses Gated Delta Rule-2 to correct slot-to-value associations, with both stages decoupling erase and write operations. This design consistently outperforms strong linear-attention baselines on language modeling, reasoning, retrieval, and long-context tasks while maintaining linear-time training and constant-memory decoding.
The work addresses a key challenge in linear attention: how to efficiently update and correct the fixed-size recurrent memory. By combining insights from slot-based architectures and delta-rule memory corrections, GSA2 achieves better associative memory without sacrificing computational efficiency. Results show particular gains on RULER and LongBench benchmarks, with ablations confirming that the complementary correction directions are responsible for the improvements.