This paper studies the precision floor of neural networks: the bit-width or noise level at which accuracy drops halfway to random chance. The authors find that this floor follows a power law in depth, meaning deeper networks generally need more bits—except when residual branches and pre-normalization are scaled appropriately. Across MLPs, CNNs, Vision Transformers, and pretrained GPT-2 models, the floor is governed by a single full-precision quantity called predictive amplification G, and the collapse constant Λ ≈ 1.45 is nearly invariant.
A key practical finding is that residual branches scaled by 1/√D and pre-normalization remove the depth penalty entirely, so deeper networks no longer require more bits. The paper also reports a QAT paradox: roughly doubles tolerable noise for shallow networks, but this benefit decays with depth, making the QAT depth law steeper than . These results provide a framework for predicting precision needs before quantization, reducing trial-and-error.