The authors propose CACTUS, a curvature-aware compression algorithm that minimizes a loss function combining layerwise output reconstruction with a gradient-based term penalizing deviation of per-sample gradients from the dense model.
Curvature-aware pruning preserves influence functions efficiently.
CACTUS, an ADMM-based optimizer, achieves order-of-magnitude speedups over prior methods while retaining gradient alignment and enabling data selection.
Big Tech
Mehdi Makni · Natalia Ponomareva · Rahul Mazumder
Massachusetts Institute of Technology · Google Research
Research Digest··3 min read
Makni et al.
Why this paper
From Google Research and Massachusetts Institute of Technology
In one line
CACTUS, an ADMM-based pruner, preserves influence functions while pruning LLMs, running over 28x faster than LLM-Surgeon with improved perplexity and gradient alignment.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (3 noted)
- ✓Reports numbers on named benchmarks (4 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§