attention residual

"Attention residual" is a trick where a model passes not just the latest processed information but also older, summarized versions to later layers. Think of it like a student who, when solving a long math problem, keeps a running summary of earlier steps alongside the current one. This helps the model remember and use previous findings more effectively, preventing it from getting lost or forgetting crucial intermediate results.

Papers show this idea improves both training and reasoning. By routing "block summaries" across layers, one video model boosts its ability to use all layers equally well, generating high-quality video faster. Another model uses this method to improve information flow across its trillions of parameters, achieving better performance on complex tasks like coding and reasoning. A third paper shows that learning which past information to keep, instead of always using the latest, allows a tiny model to solve increasingly hard puzzles by effectively reasoning over many steps without forgetting.