In Mixture-of-Experts models, tokens are processed by a subset of experts spread across GPUs. Moving tokens between GPUs for expert processing involves two all-to-all communications. At small batch sizes, fixed overheads like synchronization dominate, making these communications slow even when little data is sent.
This work introduces NVLinkOneSided, a method that lets each GPU directly write tokens into the target GPU's memory without first exchanging metadata. Tokens are placed in a rank-major layout, and the compute kernels read them directly from that layout, avoiding extra data copies. Two synchronization protocols are provided: Fence for high throughput and Sentinel for low latency.
On Blackwell GPUs, the resulting Mixture-of-Experts layer is up to 2.4 times faster than existing solutions like DeepEP, even outperforming a fused implementation at decoding batch sizes. This improvement reduces latency for small batches, which is critical for interactive inference.