The Graph Machine (GM) is an architecture that maintains a large state (O(n)) but accesses only a constant number of tokens per step through dynamic pointers called edges, updated via a referral mechanism.
The authors replaced 75% of the dense Transformer layers in a 0.6B parameter model with GM sparse layers and pretrained from scratch on 15.7B tokens. They found that retrieving only 2 out of 4096 tokens per head barely hurts loss, and retrieving 4 tokens marginally improves loss over the dense baseline.
This is interesting because it suggests that most of the full attention matrix is unnecessary and that dynamic sparse routing can match or slightly exceed dense performance, potentially leading to more efficient large language models.