The authors built Kolibri as a decoder-only transformer with hybrid attention (interleaving sliding-window and full-attention blocks) and sparse mixture-of-experts layers with 384 routed experts plus one shared expert per layer.
Open-weights German-English model claims Pareto frontier for quality and serving cost
Aleph Alpha's Kolibri activates only 4.4% of its 78.1B parameters per token while matching or beating comparable open models in English and German benchmarks.
Industry
Aleph Alpha · : · Moritz Althaus · Artur Baranowski · Michael Barlow · Fabien C. Y. Benureau · +62 more
Aleph Alpha
Research Digest··3 min read
Aleph Alpha presents Kolibri, a 78.1B-parameter mixture-of-experts transformer with 3.46B active parameters per token, trained on 24T tokens including a German-heavy data mix and released under Apache 2.0.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.