Chat and coding assistants need to deliver tokens at a steady pace. Even rare delays in the time between tokens cause annoying pauses, especially for long responses like reasoning chains. This problem, called decode jitter, happens because planning how much new work to mix into each decode step is based on average predictions, which often underestimate the real cost.
VelServe is a new scheduler that measures the actual slowest decode steps and leaves extra room in each step to avoid delays. It also prioritizes short prompts over long ones, so the overall wait for the first token stays low.
In tests on several large language models, VelServe served 2.4 to 2.9 times as many requests within their latency targets as the best previous scheduler, and at least 4.4 times as many on smaller models. This means providers can handle much more traffic without users noticing pauses.
