The authors introduce SWE-Serve, a for evaluating AI agents on production inference engineering tasks. The benchmark consists of 53 tasks derived from recent production changes to the SGLang inference serving system, spanning six engineering families such as model support, decoding, and serving APIs. Tasks are executed in containerized environments on CPU or a single H100 GPU and evaluated with hidden functional and regression tests, including end-to-end (E2E) serving tests and performance gates.
Across 11 models and 31 configurations, the best-performing configuration achieved 75% mean pass@1. The key finding is a substantial gap between local correctness and production correctness: on 19 tasks with E2E coverage, model-serving E2E tests rejected roughly one-third of patches that passed every other test. The pass rate dropped from 69.4% (excluding E2E tests) to 45.9% (with E2E tests included) under the best configuration.
This benchmark is interesting because it makes the production correctness gap directly measurable, enabling the community to track whether future agents move beyond completing tasks locally to achieving production correctness. Existing benchmarks do not target production inference engineering in a repository-scale setting, making SWE-Serve a novel contribution.