Switching (SwiLA) is a new sequence layer that enhances the expressivity of linear attention without growing its memory footprint. It works by maintaining multiple linear regression components in its recurrent state, and each output dimension dynamically selects which component to use based on the input. This is derived from a test-time regression framework using online expectation-maximization.
Experiments on associative recall, in-context learning, and language modeling show SwiLA performs strongly, narrowing the gap to softmax attention and even surpassing it in several settings. This is interesting because it offers a practical middle ground between the efficiency of linear attention and the powerful nonlinear mapping of softmax attention, enabling models with fixed-size memory to handle long sequences more effectively.