The authors fit each expert projection in selected layers to a compact set of rotor parameters, based on a Clifford-algebra representation that approximates a dense projection using O(log² d) parameters.
Generating MoE expert weights on chip reduces inference memory pressure
FORGE replaces selected stored expert tensors with compact rotor parameters and dedicated hardware that reconstructs weight tiles during inference.
Big Tech
Chanyoung Moon · Travis Pence · Daisuke Yamada · Jiaqi Mo · Yanxin Zhang · Vikas Singh · +1 more
University of Wisconsin-Madison · Korea University · NVIDIA
Research Digest··3 min read
Moon et al.
Why this paper
From NVIDIA and 2 others
In one line
FORGE regenerates MoE expert weights on-chip from compact rotor parameters, replacing HBM weight loads and improving throughput 1.3-2.8x with some accuracy loss.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§