Qin evaluated all layers of two eight-billion-parameter models, examining six matrix-multiplication operand families: weights, activations, queries, keys, values and attention probabilities.
Lower precision beats structured pruning until GPU quantization limits are reached
Across three NVIDIA GPU generations, Qin finds that 2:4 sparsity becomes useful chiefly when no lower dense precision remains available.
Independent
Xiuyi Qin
Research Digest··2 min read
Xiuyi Qin compares two ways to accelerate large language models: reducing numerical precision and imposing 2:4 sparsity, which retains at most two values in each hardware-defined group of four.
What it released
Code
§