This work provides theoretical foundations for probe-based learning of neural functionals. The authors prove that a neural network can be uniquely identified from a finite set of probe evaluations, requiring at most 2q+1 probes where q is the parameter dimension, under the assumption of analyticity. They further show that any neural functional can be approximated using such finite probes, establishing universality. Additionally, they demonstrate that using hidden-layer representations (not just final outputs) yields significantly more informative probes.
Motivated by these results, the authors introduce HiddenProbe, a simple architecture that learns from hidden probe responses. On benchmarks including MLPs and Transformers, HiddenProbe consistently outperforms existing probing methods and achieves state-of-the-art results. The code is publicly available.