When you use a cloud service to run a large language model, the cloud sees your input. Homomorphic encryption (HE) lets you compute on encrypted data so the cloud never learns your secrets, but it is very slow. This paper tackles that slowdown.
The authors propose MOAI, a system that reorganizes how data is packed inside the encryption and creates new algorithms for key operations like Softmax and LayerNorm. These algorithms avoid costly rotations, the most time-consuming HE operation, and eliminate the need to switch between data formats.
MOAI reduces rotations by at least 1.7 times and cuts inference time by 52.8% compared to the prior best method. This makes practical secure inference faster, bringing privacy-preserving AI closer to real-world use.