The authors decompose each neuron's output h_i(x)=S_i(x)*M_i(x) where S_i∈{-1,+1} is the sign and M_i is the non-negative magnitude, for any sign-preserving activation (ReLU, SiLU, GELU, tanh).
Binary sign patterns of neurons, not their magnitudes, drive classification in deep networks
Göring et al. show that a k-NN classifier using only the sign of each neuron output matches full model accuracy, while randomizing magnitudes has little effect.
Top University
Niclas Göring · Shuofeng Zhang · Roi Holtzman · Yoonsoo Nam · Ard A. Louis
University of Oxford · KAIST
Research Digest··3 min read
Göring et al.
§