What they did
The authors developed an evaluation method that tests each agent invocation for faithfulness and verifiability relative to the information it received. This local testing is designed to identify where errors enter a multi-agent pipeline rather than assessing only the final report.
They applied the method to three top-ranked open-source deep-research systems and classified errors into four types: hallucination, reliance on uncited input, uncited output, and insufficient citations.
Key findings
- Nearly every agent type made substantial mistakes; agents summarizing a single document were the main exception.
- Error patterns differed systematically by agent role, with orchestrator errors dominated by citation-related failures.
- In AI-Q, 84.7% of final-report errors originated at the orchestrator. About 31% of those were hallucinations, while the remainder were citation mistakes.
- Two simple interventions based on the diagnostics increased citation recall by 5% without degrading measured output quality.
Why it matters
Final-report citation scores show that a system failed, but not where or why. By assigning failures to specific agent calls and error categories, this method can guide targeted changes to orchestration, citation handling, or summarization instead of requiring broad modifications to an entire deep-research pipeline.
Caveats
The study covers three open-source systems, so its agent-level error distributions may not transfer to proprietary or differently structured architectures. Local evaluation also measures an invocation against its supplied inputs; it does not by itself establish that the original web sources are correct, complete, or trustworthy.