LazyFlash keeps the forward pass, which computes the model’s outputs, fully dense: every token retains its standard attention interactions.
Skipping selected attention gradients speeds long-context fine-tuning while preserving dense inference
Deng and colleagues report up to 5.7× faster attention backpropagation and 1.7× faster overall training by sparsifying only the backward pass.
Big Tech
Keqi Deng · Ruchao Fan · Jinyu Li
Microsoft
Research Digest··2 min read
Deng and colleagues introduce LazyFlash, a method that reduces attention computation during long-context fine-tuning without changing how the model computes its outputs.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.