UpLoop applies layer-wise recurrence: each selected layer performs multiple loops locally, preserving the pretrained depth hierarchy.
Upcycling pretrained LLMs with layer-wise loops breaks the two-loop saturation limit
UpLoop introduces hidden-wise gates on attention and MLP residual branches, enabling progressive refinement beyond two loops.
Chinese Tech
Tieyuan Chen · Huabin Liu · Yingtao Zhang · Shijie Li · Tengyu Pan · Weiyao Lin · +1 more
Inclusion AI, Ant Group · Shanghai Jiao Tong University · Tsinghua University
Research Digest··2 min read
The authors show that a non-looped pretrained LLM can be converted into a recurrent-depth model through post-training upcycling.
§