Speech transcription turns audio into text for voice assistants and translation systems. Running a single large model to transcribe many audio streams at once is computationally expensive, and the optimal way to group its operations changes as the number of concurrent requests varies.
This paper describes a serving engine that, for every processing pass, decides whether to run a large fused kernel (a single combined operation) or a chain of smaller fused kernels, based on pre-measured crossover points. It processes prompt and verification work together and adapts expert computation to the current request mix.
On an H100 GPU, the engine achieves up to 4.39 times the throughput of the tuned vLLM baseline, while maintaining nearly identical word error rate (1.91–1.93% vs 1.88–1.93%). This allows serving 8 times as many concurrent requests within a 400 millisecond latency budget, significantly improving the capacity of real-time transcription services.
