Training a language model across a slow network means workers must share updates. Waiting wastes time; continuing with out-of-date updates can make each worker’s copy drift away from the others.
LACCO, or Local Accumulation with Computation-Communication Overlap, lets workers keep computing while updates are exchanged. It then gives every worker the same combined update, rather than making some workers act on delayed information.
In language-model pre-training, it consistently beat other methods that overlap communication and computation, with little or no extra computing or memory cost. This can make training more efficient when network communication is a bottleneck.
