The paper presents FDE-Bench, a of 136 deployment configuration tasks covering Docker, Compose, and Kubernetes in both greenfield (from scratch) and diagnosis-and-repair modes. Agents submit declarative artifacts that are replayed in a clean sandbox and evaluated by four gated programmatic checks: build, readiness, behavior, and conformance to the specification. The benchmark uses no LLM judge, only automatic checks, and includes a release gate to reject trivial solutions. Seven language models from four providers were tested using the same scaffold. The best model resolved 75% of tasks, while the worst resolved 52.9%. Readiness failures accounted for 110 of the 313 unresolved episodes, making it the largest failure stage. Repair tasks were notably easier than greenfield tasks, with a 30.7 percentage point higher resolution rate across all models. A case study with a human engineer directing an LLM achieved 92% resolution on a subset of tasks, compared to 72% for the autonomous system. This benchmark is interesting because it rigorously evaluates a critical but often overlooked capability of LLM agents: turning application code into a running, healthy system, and it pinpoints where agents most commonly fail.
LLM agents deploy repairs better than systems built from scratch
FDE-Bench tests whether agents can produce replayable Docker, Compose, and Kubernetes configurations that build, become ready, behave correctly, and meet specifications.
Top University
Weihang Ding · Junfei Zhan · Yueting Li · Qirong Guo
University of California, Berkeley · Imperial College London · The Hong Kong University of Science and Technology (Guangzhou)
Research Digest··2 min read
In one line
FDE-Bench reveals that readiness checks are the primary failure point for LLM agents in deployment tasks.
Before you read this
Three ideas this paper builds onbenchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →agentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real304 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.341 papers →
Read it as
Ding et al. introduce 136 deployment tasks evaluated through programmatic checks in freshly rebuilt environments, without an LLM judge.
The authors built FDE-Bench from hand-authored deployment scenarios covering Docker images, multi-service Compose stacks, and Kubernetes.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.