The authors translated seven frameworks, Star-free, AC⁰, TC⁰, C-RASP, MPC, BAPO and Sensitivity, into a common matrix of task predictions.
Seven transformer theories fall short of explaining what models learn
Controlled task comparisons expose mismatches between theoretical capability predictions, training success and generalization to longer inputs.
Big Tech
Awni Altabaa · Tobias Schnabel · Kiran Tomlinson
Yale University · Microsoft Research
Research Digest··2 min read
Altabaa, Schnabel and Tomlinson compare seven theories of transformer capability against experiments with small, task-specific transformers and pretrained language models.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.