The authors analyzed a two-unit, bias-free ReLU network with inputs uniform on a disk, trained on a source task that drives positive proportionality and a target task that rewards separating the units.
Gradient flow mispredicts when SGD recovers plasticity after pretraining in ReLU networks
The authors show that rare gate disagreements, which gradient flow ignores, cause SGD to remain stuck near cloned neurons for exponentially long times
Big Tech
Ruoyu Zhao · Mingxuan Zhang · Jianbo Dai · Jiaqi Wu · Chenyu Zhu · Tong Che
City University of Hong Kong · Microsoft · Copula Lab · NVIDIA Research
Research Digest··2 min read
Zhao et al.
Why this paper
From Microsoft and 3 others
In one line
Population gradient flow overestimates plasticity when pretraining makes rare disagreement events limit finite batch SGD.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (3 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§