Applications built on large language models (software that generates text) often repeat the same opening instructions across many requests. Serving engines, the software that runs these models on graphics chips (GPUs), can cache that shared text instead of recomputing it, and memory is what limits how much work a chip can hold.
The two popular engines, vLLM and SGLang, cache this shared text at different granularities, and benchmarks of their out-of-the-box settings show big gaps, usually blamed on caching. But their defaults also differ on whether a long prompt is processed in pieces so other requests keep flowing. The team built a repeatable test setup and a synthetic workload of chained agent requests, then compared both engines as shipped and again with that one setting matched.
Matching it removed most of the capacity gap and much of the latency gap. What remains is a tail-latency advantage of 2.4 to 6.4 times at equal throughput, consistent with how one engine orders waiting requests.