Datadog is betting that AI agents can write and run the tests that other AI agents' code needs. The observability company acquired Propolis earlier this year to add goal-based, agent-written test generation, aiming to relieve what it sees as the new bottleneck in software development: testing.
Nearly a year into the agentic coding boom, many teams now use AI to write software as their de facto approach, according to The New Stack. But the report notes that AI-written code does not necessarily compress the software development lifecycle on its own, because the critical bottleneck has shifted from writing code to testing it.
Anthropic's engineers are now shipping eight times as much code as before, and some engineers are shipping thousands of pull requests per month, the report says. Traditionally, testing that code has meant humans deciding which user flows need coverage, scripting tests for them, and reworking those scripts when the flows change.
Datadog bought Propolis earlier this year to attack that problem. Propolis' approach lets agents explore applications on their own, follow user flows, and generate tests from plain-language prompts. Combined with Datadog's observability platform, the system can test code while drawing on the logs and traces needed to vet system performance, without the fragility of hardcoded testing.
The company is set to discuss the approach in a webinar on November 4, with Ruxanda Lueck, a Datadog group product manager working on goal-based testing of AI features, and Matthew White, who co-founded Propolis and served as its chief technology officer.
Topics include how agents can test user flows that no one thought of, how goal-based testing persists when user flows change, and how to test AI features that return probabilistic, rather than deterministic, outputs. Datadog also acknowledges that scripted tests still have their place.
Analysis
Why This Matters
- Testing has become the bottleneck in software delivery as AI coding agents accelerate how much code gets written.
- Whether agent-based testing works reliably could determine how quickly AI-written code reaches production.
- The push raises questions about whether AI systems can meaningfully verify work produced by other AI systems.
Background
Agentic coding tools have moved from experiment to default practice across many engineering teams, dramatically increasing the volume of code produced. The new problem is that the rest of the software development lifecycle has not sped up to match. Testing remains largely a manual, script-writing task, and it now sits in the path of far more code than before. Datadog's acquisition of Propolis is a bet that testing can be automated the same way coding has been: by letting agents work from goals expressed in plain language, with observability data to ground the results.
Key Perspectives
Datadog and Propolis: Testing should be goal-oriented rather than script-bound. Agents can explore applications, generate and run tests from plain-language prompts, and use observability data such as logs and traces to verify performance. Scripted tests still have a role, but not as the primary way to keep pace.
Engineering teams: Teams facing AI-driven release velocity need to offload test-writing work or watch coverage fall behind. The appeal is real, but adoption depends on whether agents can handle complex user flows and changing requirements without constant human intervention.
Critics and skeptics: Agent-written tests may carry the same reliability risks as agent-written code, especially when the feature under test is itself probabilistic in nature. The webinar's own agenda, with its focus on testing non-deterministic AI outputs, signals this remains an unsolved problem.
What to Watch
- The November 4 webinar, which may reveal how much test-writing teams can realistically offload today.
- Whether goal-based testing proves durable as user flows and product surfaces change.
- How Datadog's approach handles AI features with probabilistic, non-deterministic outputs.
