When a small device like a phone runs a language model with help from a cloud server, each round of communication adds a delay called a round-trip time. This delay hurts performance, especially on slow networks like cellular or satellite.
SpecRelay uses two strategies to reduce the impact of this delay. First, it keeps multiple draft token trees in flight at once, so the device can work while waiting for the cloud. Second, it decides per query whether using the cloud is worth it, based on the current network delay and predicted quality.
On benchmarks, under 250 ms of added round-trip delay, SpecRelay delivers up to 1.95x the throughput of a standard system while sending only 47% of queries to the cloud, achieving the same accuracy. This means faster and cheaper edge inference on slow networks.
