The authors initialized shallow vision transformers and progressively inserted layers by copying existing blocks using three strategies: Stacking, MIDAS, and LIDAS.
Progressive depth growth improves vision transformers with less compute
Growing shallow networks by inserting layers during training yields better representations and reduces FLOPs by up to 24.8% for MAE and DINOv3.
Big Tech
Amir Mohammad Karimi Mamaghan · Ferdinand Kapl · Andrea Dittadi · Karl Henrik Johansson · Kaitlin Maile · Johannes von Oswald · +1 more
KTH Royal Institute of Technology · Helmholtz AI · Technical University of Munich · Munich Center for Machine Learning (MCML) · Google
Research Digest··2 min read
The authors systematically investigate progressive depth growth in self-supervised vision transformers (MAE and DINOv3).
Why this paper
From Google and 5 others
In one line
Progressive depth growth in self-supervised vision transformers improves downstream performance while cutting pretraining FLOPs by 13.9-24.8%.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (3 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§