Training large language models requires splitting their data across many computers because of memory limits. These pieces must communicate to update the model, which slows down training.
Dobb is a new method that works locally inside each piece. It builds a simple model of input statistics and then uses only the sign of the update to adjust the model, avoiding inter-piece communication entirely.
On a model with over a billion parameters, Dobb reached a lower loss (2.476 vs 2.511) and completed each update more than twelve times faster than a comparable distributed optimizer. This allows faster training without harming quality.