The authors pretrained a 450M-parameter transformer on 50B tokens using FlashAttention-3 with BF16 precision. Training was stable for 25B tokens, then the gradient norm grew a thousandfold and loss ended 0.2 nats above FP32 attention, without NaNs. They found two causes: a fused multiply-add in the forward softmax (treated as an extreme-input NaN case and never fixed) and a deeper issue—the softmax score gradient sums to zero along each row in exact arithmetic, but rounding to BF16 breaks this conservation. The broken sum allows the mean key to leak into the query gradient, and the leak grows as training proceeds and keys become large and attention sharpens. The authors propose GProj, which restores the zero sum after the BF16 cast with two rank-one corrections per row. This reduces median query/key gradient errors from 219% and 13% to 0.34% and 0.37%, on par with FP32 attention, with a 4.7% time increase per step. Matched from-scratch runs with GProj train to the same loss as FP32 attention, while both FlashAttention-3 and key smoothing destabilize.
Restoring softmax gradient symmetry prevents late BF16 attention instability
A row-wise projection corrected large gradient errors in FlashAttention-3 and matched FP32 attention during transformer pretraining.
Top University
Junlin Chen · Daize Dong · Huanwei Di · Haolong Jia · Jiawei Wu · Haotian Xie · +6 more
Rutgers University · Carnegie Mellon University · Oracle · New York University · MBZUAI
Research Digest··3 min read
In one line
FlashAttention-3 in BF16 causes gradient blow-up due to broken softmax score gradient sum conservation, fixable with gauge projection.
Read it as
Chen and colleagues trace a late-training failure in BF16 FlashAttention-3 to two numerical errors, including a previously overlooked violation of a softmax conservation law.
The authors pretrained a 450 million-parameter transformer for 50 billion tokens, comparing BF16 FlashAttention-3 with FP32 attention and modified backward passes.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.