Transformers, the neural networks behind many AI language models, have attention heads that perform different roles. Understanding which head does what usually requires running the model on data, which is slow.
This paper finds a mathematical property called rho (ρ) that can be computed directly from the model's weights, without any data. Rho measures how much a head's internal geometry is shaped to work with rotary positional encodings (RoPEs), a common way to encode word order.
The authors show that rho predicts induction heads, specialized heads that copy patterns from earlier text, with a strong negative correlation (between -0.55 and -0.74). This means heads with lower rho are more likely to be induction heads. This work offers a new way to analyze model internals purely from weights, potentially saving computation and aiding interpretability.