Many AI models are trained to combine known operations, like sorting or filtering, to handle new tasks. Evaluating whether they truly learn this is tricky because some test problems may look new but are actually solvable using shortcuts learned during training. This makes it hard to know if the model genuinely understands composition or is just exploiting hidden similarities.
This paper systematically tested transformer models on sequential tasks made of smaller operations. The authors identified two key factors that cause overestimation: composition equivalence (different operation sequences that happen to produce identical final results) and module coverage (how often each operation appears in similar contexts during training and testing). They controlled these factors across different training setups.
The main finding is that when composition equivalences are removed, transformer performance drops from nearly perfect to near zero. This matters because many benchmarks inadvertently include such equivalences, leading to inflated claims about compositional reasoning. The results urge benchmark designers to define "unseen" tasks by their function, not just by surface sequence differences.