When researchers try to understand what a neural network 'knows,' they often look at individual neurons, the smallest processing units inside the model. But a neuron's activation (its output number) changes with every different prompt, and past methods either used hand-written templates (which miss real-world variation) or random text (which is uncontrolled). Neither approach gives a reliable measurement that can be repeated or compared across different models or training stages.
The authors solve this by first defining exactly what population of prompts they will use: for a concept like 'for loop,' they list every property that can vary (identifiers, nesting, surrounding code) and draw each property at the same rate it appears in a reference code corpus. This makes the stimulus set both realistic and fully described. Then they measure each neuron's average activation across these prompts and divide by that neuron's own variability on ordinary input, so the result is a unitless number that means 'this many standard deviations above baseline.' They run the whole measurement twice to estimate noise.
The main finding is that this 'calibrated concept profile' gives a number with a unit for every neuron, a chance level for every comparison, and can track which neurons carry a construct across two trillion training tokens. This matters because it turns a subjective reading into a proper measurement that can be shared and trusted, without training a second model on top of the first.