Training large language models with quantized (low precision) weights, a technique called , usually requires keeping two full precision copies of each weight. This takes a lot of memory. Earlier attempts to reduce that memory by only tracking the most important directions for weight updates (low rank methods) lost too much accuracy. The problem is that quantization introduces small errors spread across all weight entries, not just the principal directions.
The authors noticed that even the discarded residual part of the weight update is useful if only its sign is kept. They built an optimizer, ResQAdam, that tracks the full momentum in half precision (BF16), applies the Adam rule only inside a low rank subspace, and adds a sign based update from the leftover part. This avoids storing a full second high precision state.
ResQAdam cuts the optimizer memory by about three quarters compared to standard AdamW, while losing less than 2% training speed. In tests on several models and quantization levels down to 2 bits, final accuracy was nearly the same as with AdamW. This makes high quality low bit quantization more practical for memory constrained hardware.