The authors study hybrid blocks that combine one full-attention layer with several efficient sequence layers, focusing on function composition, a task linked to multi-hop reasoning.
Multi-hop retrieval from a shared KV cache gives hybrid models strictly deeper composition power
The authors introduce MuRA, prove it computes m-function composition with a single block, and show that fewer retrieval hops require polynomial capacity.
Big Tech
Yam Eitan · Gilad Yehudai · Haggai Maron
Technion – Israel Institute of Technology · NVIDIA
Research Digest··3 min read
Eitan, Yehudai, and Maron analyze the expressive power of efficient hybrid sequence models, which pair a full-attention layer with cheaper recurrent or sparse layers.
§