The authors trained ReLU networks to add two integers modulo a prime, a standard setting for studying grokking, where training accuracy precedes generalization.
Networks can learn hidden structure while training loss stays flat
On modular addition, changing representations and improving held-out accuracy coexisted with nearly constant, positive training cross-entropy.
Big Tech
Akash Kumar · Karthik Sankaranarayanan
University of California, San Diego · Microsoft AI
Research Digest··3 min read
Kumar and Sankaranarayanan study whether a classification network can continue learning after achieving 100% training accuracy and entering a cross-entropy plateau.
§