The authors analyze how rounding affects logits, the scores a model assigns before converting them into probabilities.
Rewriting softmax calculations fixes two rounding errors that distort training
Barron and colleagues report that precision-preserving formulas remove spurious gradients in high-confidence classification and closely matched teacher-student networks.
Big Tech
Jonathan T. Barron · Dan Gnanapragasam · Michael Tschannen
Google DeepMind
Research Digest··2 min read

Jonathan T.
§