The paper introduces Q2D-Web, a for retrieval in production agentic RAG systems. It pairs a 190M-document web corpus with 70k agent-reformulated search queries in ten languages, drawn from real user interactions. Three sets of relevance judgments are provided: agent citations, production rankings, and a combined set with LLM-based judgments to reduce false negatives. The benchmark tests 13 retrievers (lexical, dense, late-interaction) and finds their rankings are stable across judgment sets but vary by domain, language, and query type. A subcorpus sampling method retaining one-third of the corpus preserves full-corpus model rankings.
Benchmark tests retrieval for agent-written queries across 190 million web pages
Q2D-Web evaluates 13 retrieval systems using roughly 70,000 multilingual queries reformulated from real production conversations.
Industry
Maximilian Schall · Sedigheh Eslami · Markus Krimmel · Antoine Chaffin · Louis Milliken · Bo Wang · +1 more
Perplexity AI
Research Digest··2 min read
In one line
Q2D-Web provides a large-scale benchmark with 70k machine-written queries and 190M documents for evaluating first-stage retrievers in agentic RAG systems.
Before you read this
Two ideas this paper builds onRead it as
Schall et al. introduce Q2D-Web, a benchmark designed to reflect the first retrieval stage of agentic retrieval-augmented generation systems.
The authors assembled a corpus of approximately 190 million web documents and 70,000 machine-written search queries in ten languages.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.