Large language models use attention to focus on key parts of text, but full attention is slow for long inputs. Hybrid models use a faster in most layers and full attention in a few. Selecting which layers keep full attention is hard.
Earlier methods had flaws: they used approximations not matching the final model, and did not update model weights during selection. FlashMorph2 fixes this by evaluating actual hybrid configurations and jointly optimizing selection and weights.
FlashMorph2 improved scores by 2.42 points on a long-context recall test and 1.08 points on another long-document benchmark. This makes efficient hybrid models more accurate and practical.
