The authors studied language-model configurations with dense reference sizes from 360 million to 3 billion parameters.
Reusing computations across model depth favors learning from the prompt
Across language models and controlled tasks, the authors find that growing or looping layers strengthens context-based learning and increases reliance on particular attention mechanisms.
Big Tech
Ferdinand Kapl · Kaitlin Maile · Stefan Bauer · Johannes von Oswald
Technical University of Munich · Helmholtz AI · Munich Center for Machine Learning (MCML) · Google, Paradigms of Intelligence Team
Research Digest··2 min read
Kapl and colleagues compare standard language models with models that gain layers through copying or repeatedly apply shared layers.
§