Loop-native residuals let Transformers reason across tens of thousands of iterations
The authors introduce InfiLoop, a residual connection for looped Transformers that learns to selectively aggregate past recurrent states instead of overwriting them with each new iteration. On Sudoku-Extreme, a 7M-parameter InfiLoop model reaches 97.9% exact accuracy, and it continues to improve even beyond 20,000 effective steps, demonstrating that added loop depth can translate directly into stronger reasoning.
