They designed Echo, a 4-bit all-reduce for tensor parallelism in LLMs.
Echo enables fast 4-bit tensor-parallel all-reduce with controlled error accumulation
By restructuring the collective and feeding back rounding errors across calls, Echo speeds up LLM training and inference without significant quality loss.
Independent
Xingyu Xie · Xiaoyong Xie · Zhouchen Lin · Kim-Chuan Toh
Research Digest··2 min read
The authors present Echo, a 4-bit all-reduce for tensor parallelism that bounds quantization error accumulation across layers.
Why this paper
Independent
In one line
Echo is a 4-bit all-reduce that feeds rounding errors back into later calls to keep accumulated error bounded, achieving near 16-bit accuracy with up to 2x speedup.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ✓Compute or model size stated (hardware A100)
- ·No stated limitations found
- ✓Reports numbers on named benchmarks (8 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§