Large language models often get better by adding more layers, but that means storing more parameters. Looping reuses the same layer weights multiple times, making the model deeper without adding new weights. This is helpful when memory is limited but comes with other costs.
The researchers trained many small transformers from scratch, varying how much of the model was looped and how many loops were used. They compared looping the entire transformer versus only a middle section. They also tried adding small, loop-specific adapters to let each loop learn slightly different transformations.
The best design depends on the main constraint. When parameter count is fixed, looping the full transformer gives the strongest quality. When training compute is limited, looping a middle block works better. Gains from adding more loops eventually saturate or reverse. Small loop-specific adapters help, especially with many loops, and a modified parallel version can run loops concurrently during text generation without losing quality.