Learning-rate-to-parameter-norm ratio predicts language model pretraining loss

Across varied training configurations, matching effective learning rate produced nearly identical loss trajectories despite different raw learning rates and parameter norms.

PaperChinese Techcs.LGarXiv:2608.24814v1 ↗

Zihan Liu · Ruiheng Zheng · Shaobo Zhang · Changxin Tian · Kunlong Chen · Zhiqiang Zhang · +1 more

Peking University · Ant Group

Research Digest·27 August 2026·2 min read

Read the paper →

What they did

The authors ran controlled language model pretraining experiments in which learning rates and parameter norms differed while their ratio—the effective learning rate—was matched. They tested whether the resulting loss trajectories aligned across optimizers, architectures, datasets, and model scales.

They also ablated normalization choices and the timescale over which learning rate and parameter norm vary. Interventions using weight decay and Hyperball, a norm-control method, tested whether these techniques affect loss mainly by changing ELR schedules. Finally, the authors reformulated a fitted functional scaling law using ELR instead of raw learning rate.

Key findings

Runs with matched ELR exhibited closely aligned loss trajectories even when their raw learning rates and parameter norms differed substantially.
Mean collapse errors were typically on the order of a few × 10^-3 across the tested settings, below seed-to-seed variation in a representative configuration.
Normalization design and the timescale of learning-rate and norm variation determined how precisely trajectories collapsed.
An ELR-based functional scaling law transferred across norm-control methods and accounted for “delayed acceleration,” in which norm control improves training only after an initial delay.

Why it matters

The results suggest that learning-rate schedules and parameter-norm control should not be analyzed independently: their ratio may be the more useful variable for predicting pretraining loss. This could simplify comparisons among optimization methods and make empirical scaling laws more transferable across training configurations.

Caveats

The evidence is empirical and establishes ELR as a predictive coordinate rather than a complete causal theory of optimization. Collapse precision depends on normalization and variation timescales, and the abstract does not establish whether the relationship extends to all model families, very large training runs, downstream performance, or generalization beyond pretraining loss.