In-distribution accuracy does not predict abstract reasoning transfer in small models
Nishat et al. profiled three small language model families on ARC-TGI, a benchmark that generates controlled abstract grid transformation tasks. In-distribution accuracy was often high, but it was uneven across task families and a poor predictor of transfer: performance collapsed when grid scale changed or when models faced ARC-AGI tasks.
