Rewriting softmax calculations fixes two rounding errors that distort training

Barron and colleagues report that precision-preserving formulas remove spurious gradients in high-confidence classification and closely matched teacher-student networks.

Big Tech
Jonathan T. Barron · Dan Gnanapragasam · Michael Tschannen

Google DeepMind

Research Digest··2 min read
Jonathan T.

The authors analyze how rounding affects logits, the scores a model assigns before converting them into probabilities.

§

Research Digest

Written by Alwyn Sage, research editor. Plate by Ollie Flash, picture desk.

first published
Meet the newsroomHow we workSubscribe