The paper introduces two techniques to speed up feed-forward network (FFN) inference in large language models. Model casting is a mid-tuning recipe that swaps the SiLU activation for a ReLU-derived variant (R-S+) and adds an adaptive L1 penalty, inducing high activation sparsity. LoPA Gating is a new FFN design that reduces the FLOPs of the gating matrix, raising the theoretical speedup ceiling beyond 3x.
At matched model quality, LoPA Casting achieves a 3.2x FLOP speedup, compared to 1.6x for prior methods like top-p and TEAL. With custom kernels, the approach delivers an actual 3.31x speedup on GPU at 90% sparsity, surpassing the 3x limit of standard gating. The results show that unstructured activation sparsity can be practically exploited for inference, contrary to common belief.