The authors derived an exact mean-covariance decomposition of direct feedback alignment updates.
Shared output errors can saturate networks trained with random feedback
The authors identify a rank-one update that drives tanh units into saturation and test interventions that reduce the resulting learning plateau.
Top University
Varun Reddy · Bernardo L. Sabatini · Houman Safaai
Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University · Howard Hughes Medical Institute · Harvard Medical School
Research Digest··2 min read
Reddy, Sabatini and Safaai analyze why direct feedback alignment, which trains hidden layers using fixed random projections of output errors, can stall near constant-predictor performance.
Why this paper
From Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University and 2 others
In one line
In direct feedback alignment, the shared component of output error drives tanh hidden units to saturate, causing a recoverable collapse that slows learning.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§