ARC-TGI represents each transformation as a task-family generator that resamples colors, positions, object configurations, and grid sizes.
In-distribution accuracy does not predict abstract reasoning transfer in small models
Across more than 1,000 supervised fine-tuning runs on ARC-TGI, decoder-only, encoder-decoder, and mixture-of-experts models degraded sharply under grid-scale and cross-benchmark shifts.
Big Tech
Nur A Zarin Nishat · Jens Lehmann · Andrei Aioanei · Sahar Vahdati
Leibniz University of Hanover · TIB – Leibniz Information Centre for Science and Technology and University Library · Amazon
Research Digest··2 min read
Nishat et al.
Why this paper
From Amazon and 2 others
In one line
Small language models can score well on abstract reasoning benchmarks without acquiring transferable rules; performance degrades sharply under shifts such as grid-scale changes.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§