The authors diagnose a bottleneck in looped Transformers, which repeatedly apply a shared block to increase reasoning depth.
Loop-native residuals let Transformers reason across tens of thousands of iterations
InfiLoop replaces the carry-last rule with learned attention and temporal decay, achieving state-of-the-art accuracy on reasoning benchmarks while scaling to over 20,000 test-time loops.
Independent
Pengxiang Li · Dilxat Muhtar · Di He · Guinan Su · Lu Yin · Shiwei Liu
Research Digest··2 min read
The authors introduce InfiLoop, a residual connection for looped Transformers that learns to selectively aggregate past recurrent states instead of overwriting them with each new iteration.
Why this paper
Independent
In one line
Looped Transformers keep improving past 20,000 test-time iterations when residual connections learn to retain and weight past recurrent states.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ✓Compute or model size stated (params 7M)
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§