AI systems like large reasoning models (LRMs) are tested on puzzles where they must infer a rule from a few examples, like recoloring objects inside a shape. It is unclear if they learn the rule for that specific puzzle or a general concept, such as “inside a shape,” that applies across many puzzles.
The researchers built a generator that creates thousands of new puzzles grouped by 16 spatial and semantic concepts (e.g., “Inside and Outside,” “Count”). They tested several LRMs on 1,600 such puzzles, each requiring the model to solve 10 test inputs correctly. Even the best model succeeded on fewer than a quarter of the puzzles.
By analyzing the internal patterns the model uses to represent each puzzle, they found that puzzles cluster by the operation applied (like recoloring or removal) rather than by the higher-level concept. This suggests models learn reusable operation abstractions, improving their ability to generalize, but still struggle with puzzles requiring comparison of properties like shape or number.