Google has open-sourced AQuA (Ambient Quality Agent), a reference implementation that analyzes production AI agent sessions, verifies recurring failure patterns, and investigates root causes against deployed source revisions. The tool targets silent quality regressions such as agents ignoring user preferences or skipping required tool calls while returning successful responses.
AQuA runs alongside an agent in Google Cloud, outside its request path, reading telemetry from Cloud Trace, Cloud Logging, or BigQuery. It samples up to 1,000 recent sessions per run and processes them through five stages: sampling, reviewing, clustering findings, verifying candidate clusters, and tracking issues. Developers can configure domain-specific quality criteria through a goal.md file and add deterministic custom metrics.
Google evaluated 32 sessions from a multi-agent travel application, identifying 42 findings grouped into nine candidate clusters. The verifier rejected three unsupported clusters after checking up to three full transcripts per candidate cluster, leaving six verified issues. Two involved bypassing seat availability checks and losing a traveler's vegan preference during delegation. After developers modified prompts and replayed the sessions, seat-selection bypasses fell from 15 sessions to two, dietary-preference omissions dropped from seven to zero, and sessions passing all checks increased from five to 13.
For root cause analysis, AQuA compares failing sessions with an immutable source snapshot captured at deployment. Its diagnosis agent identifies relevant code or instructions, cites file and line ranges, and proposes changes when the cause lies in the repository. It can also attribute failures to external dependencies, tool handoffs, or retrieved data. AQuA does not apply fixes or open pull requests automatically. The Google blog post states, "An insight is its citations, not a self-reported confidence score." Findings link to supporting traces, and proposed root causes must reference valid lines in the deployed snapshot.
AQuA tracks findings as NEW, RECURRING, or automatically RESOLVED after 14 days without being observed. Sampling can miss rare failures, and replay may not reproduce external state. Reacting to the announcement on LinkedIn, Dmytro Aprelenko, engineering manager at Redcore, wrote, "Agent orchestration definitely needs better continuous diagnosis to catch these loops early." AQuA validates source references. Its open-source implementation helps teams integrate production agent diagnosis into development and evaluation workflows.
Analysis
Why This Matters
- Enables teams to catch quality regressions that don't trigger traditional infrastructure alerts, improving AI agent reliability.
- Provides a systematic approach to root cause analysis linking failures to specific code changes.
- Open-source nature allows wider adoption and community contributions to production AI monitoring.
Background
AI agents are increasingly deployed in production to perform tasks such as booking travel or managing workflows. However, agents can silently fail by ignoring user preferences or skipping steps, which standard monitoring tools may not detect. Google's AQuA addresses this gap by analyzing session telemetry and correlating failures with deployed code versions.
Key Perspectives
Google: says AQuA targets silent quality regressions that conventional infrastructure monitoring can miss, and its open-source implementation helps teams integrate production agent diagnosis into development and evaluation workflows. The blog post notes that the tool's strength is its citations rather than a self-reported confidence score.
Dmytro Aprelenko: engineering manager at Redcore, says agent orchestration needs better continuous diagnosis to catch loops early.
What to Watch
- Adoption of AQuA by teams developing multi-agent systems.
- Integration of AQuA findings into automated CI/CD pipelines for proactive remediation.
- Potential improvements to handle sampling limitations and external state reproduction.
Checked
12 claims in this story checked against 1 source: 12 supported.
Show each claim
- SupportedGoogle has open-sourced AQuA to diagnose production AI agent failures.InfoQ: “Google Open Sources Ambient Quality Agent (AQuA) to Diagnose Production AI Agent Failures.”
- SupportedAQuA targets silent quality regressions conventional infrastructure monitoring can miss.InfoQ: “The tool targets silent quality regressions that conventional infrastructure monitoring can miss, such as agents ignoring user preferences or skipping required tool calls while returning successful responses.”
- SupportedThe Google blog post says, “An insight is its citations, not a self-reported confidence score.”InfoQ: “The Google blog post states, An insight is its citations, not a self-reported confidence score.”
- SupportedDmytro Aprelenko, an engineering manager at Redcore, wrote on LinkedIn, “Agent orchestration definitely needs better continuous diagnosis to catch these loops early.”InfoQ: “Reacting to the announcement on LinkedIn , Dmytro Aprelenko , engineering manager at Redcore, wrote , Agent orchestration definitely needs better continuous diagnosis to catch these loops early.”
- SupportedAQuA runs alongside an agent in Google Cloud, outside its request path, and reads telemetry from Cloud Trace, Cloud Logging, or BigQuery.InfoQ: “AQuA runs alongside an agent in Google Cloud, outside its request path, reading telemetry from Cloud Trace, Cloud Logging, or BigQuery.”
- SupportedAQuA samples up to 1,000 recent sessions per run and processes them in five stages.InfoQ: “It samples up to 1,000 recent sessions per run and processes them through five stages: sampling, reviewing, clustering findings, verifying candidate clusters, and tracking issues.”
- SupportedGoogle evaluated 32 travel-application sessions and identified 42 findings grouped into nine candidate clusters.InfoQ: “Google evaluated 32 sessions from a multi-agent travel application, identifying 42 findings grouped into nine candidate clusters.”
- SupportedThe verifier rejected three unsupported clusters after checking up to three full transcripts per candidate cluster, leaving six verified issues.InfoQ: “The verifier rejected three unsupported clusters after checking up to three full transcripts per candidate cluster, leaving six verified issues.”
- SupportedTwo verified issues involved bypassed seat-availability checks and a vegan preference lost during delegation.InfoQ: “Two involved bypassing seat availability checks and losing a traveler's vegan preference during delegation.”
- SupportedAfter prompt changes and replay, seat-selection bypasses fell from 15 sessions to two, dietary-preference omissions from seven to zero, and sessions passing all checks rose from five to 13.InfoQ: “After developers modified prompts and replayed the sessions, seat-selection bypasses fell from 15 sessions to two, dietary-preference omissions dropped from seven to zero, and sessions passing all checks increased from five to 13.”
- SupportedAQuA compares failing sessions with an immutable source snapshot captured at deployment; its diagnosis agent identifies relevant code or instructions and cites file and line ranges.InfoQ: “For root cause analysis, AQuA compares failing sessions with an immutable source snapshot captured at deployment. Its diagnosis agent identifies relevant code or instructions, cites file and line ranges, and proposes changes when the cause lies in the repository.”
- SupportedAQuA tracks findings as NEW, RECURRING, or automatically RESOLVED after 14 days without observation; sampling can miss rare failures and replay may not reproduce external state.InfoQ: “AQuA tracks findings as NEW , RECURRING , or automatically RESOLVED after 14 days without being observed. Sampling can miss rare failures, and replay may not reproduce external state.”