The authors divide each transformer’s residual stream into a dense core and ordered groups called shells.
A single transformer trains nested model sizes in one pass
ShellElastic uses structured weight sharing and width-aware parameter scaling to train multiple transformer sizes with a fixed computation graph.
Big Tech
Can Jin · Yan Kang · Yifan Gong · Chongjian Ge · Mingze Xu · Yuanjun Xiong · +2 more
Rutgers University · Adobe Research
Research Digest··3 min read
Jin and colleagues introduce an elastic transformer whose smaller sub-networks are exact prefixes of the widest model’s hidden states.
Why this paper
From Adobe Research and Rutgers University
In one line
ShellElastic trains nested transformer widths in one fixed-workload pass, while μ-Elastic preserves maximal updates and transfers hyperparameters across widths and supernet sizes.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§