Modern language models sometimes use gates to control information flow. When input token embeddings become very small due to weight decay, these gates can vanish, causing invisible gradient spikes that overflow memory and discard batch updates during training.
The authors looked at 291 model repositories and found that over half of models with untied embeddings have this problem. They proposed two fixes: GroupClip, a simple change for already-trained models, and GateFloor, a small positive addition for models being trained.
GateFloor cut gradient amplification by 28 orders of magnitude in experiments. This prevents training stalls in tasks like adapting models to new languages, where before about one in five steps were lost.