are trained with backpropagation, which coordinates all layers but forces each layer to hold its activations and wait for gradient propagation. Local learning removes this dependency by training each module with its own learning signal, but has not scaled to billion-parameter pretraining. SOLO (Shared-Output LOcal learning) replaces the private readouts used in conventional local learning with a shared read-only copy of the final module's readout, taken from the previous step. This transmits information from deeper modules without passing gradients between modules or reintroducing update locking.
Tested on Transformers from 340 million to 2 billion parameters pretrained on 15 billion tokens, SOLO approaches backpropagation accuracy, staying within one point in average zero-shot performance. The perplexity gap narrows as model scale increases. Without update locking, each pipeline stage holds activations for a constant number of micro-batches instead of a number proportional to the pipeline depth, freeing memory for larger micro-batches. This yields up to 1.44 times the throughput of the best measured pipeline backpropagation on the same partition.
SOLO is the first local learning method to demonstrate such memory and throughput gains in billion-parameter language model pretraining. It makes local learning a practical alternative to backpropagation for large-scale training, potentially reducing hardware costs and training time while maintaining model quality.