, Q-K, V-O, ReLU up-down projections).
No standard optimizer preserves training equivalence under Transformer reparameterization
Kim proves that gradient-based optimizers cannot keep functionally identical Transformer parameterizations equivalent after a single update, and proposes periodic rebalancing to fix it.
Independent
Bum Jun Kim
Research Digest··3 min read
, query and key) preserves the model's function but changes gradient scales, causing standard optimizers like AdamW to break equivalence after an update.
Why this paper
Independent
In one line
Standard optimizers do not keep functionally identical Transformers equivalent during training; SAGGS rebalances weight pairs to restore equivalence.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ✓Compute or model size stated (params 101M)
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (3 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§