The work starts from an exact duality: a linear neural-network layer trained by gradient descent can be represented as linear attention over stored key-value pairs associated with its training history.
Growing neural networks learn by retrieving stored training examples
The authors derive principled training rules for deep layers that add a key-value memory for each example and use nonlinear attention at inference.
Big Tech
Maximilian Schlegel · Rajai Nasser · Seijin Kobayashi · Yanick Schimpf · Oliver Sieberling · Robert Obryk · +3 more
Google · MIT CSAIL · Yale University
Research Digest··2 min read
Schlegel and colleagues recast deep learning as retrieval from representations accumulated during training, rather than compression solely into fixed-size weight matrices.
Why this paper
From Google and 2 others
In one line
Replacing linear attention in neural networks with kernelized attention enables growing networks that store all training data.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§