The paper introduces a new metric called effective depth (Deff) that measures how many statistically distinct states a transformer's residual stream actually produces as information flows through its layers. Rather than assuming that a model with L layers uses all L layers, this metric looks at how similar representations are between nearby and distant layers.
Applying this metric to sixteen decoder-only language models reveals that all of them fall below a reference value FL that would hold even if each layer added completely new information. The measured effective depths are extremely low, often under 2 layers. This is not because depth is wasted, but because correlated updates between layers are a normal and calibrated feature of trained transformers.