The authors propose a method to 'elastify' pre-trained transformers without retraining.
Differentiable soft pruning makes transformers elastic across computational budgets without retraining
By optimizing importance scores for each MLP neuron and attention head jointly across multiple sparsity targets, the method yields smoothly degrading performance up to 60% sparsity on 27 vision and text models.
Independent
Walter Simoncini · Dheeraj Varghese · Gertjan J. Burghouts · Cees G. M. Snoek
Research Digest··3 min read
Simoncini et al.
Why this paper
Independent
In one line
Pre-trained transformers can be made elastic by jointly optimizing individual importance scores for every neuron and attention head across multiple sparsity targets.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§