The authors ran a controlled benchmark on 73 OpenML classification tasks and 48 regression tasks.
Learned sparse expansions match dense ReLU layers across 121 tasks
Gradient descent on the expansion map makes top-k sparsified representations competitive with dense activations, with the learning advantage concentrated at narrow widths.
Big Tech
Alireza Asghari · Parikshit Ram · Kaushik Sinha
Wichita State University · IBM Research
Research Digest··3 min read
The authors evaluate learned expand-and-sparsify (EaS) representations, which lift inputs to a high dimension and keep only the k largest coordinates.
§