Transformers build a memory cache that expands with sequence length. Recurrent models avoid this but must choose: simple linear memories train fast because they use a single matrix but are limited in expressiveness, while nonlinear memories using a small neural network are powerful but much slower.
Toaster combines both by building a two-layer memory from two linear layers with an activation between them. It trains the hidden layer using target propagation, projecting the output error backward, so every layer can still use the fast training kernel.
The resulting model matches standard attention on a language modeling test. Its mean accuracy on 13 tasks is 51.84 percent, better than prior linear and nonlinear memories, while training nearly as fast as the simplest models.
