What they did
The authors assembled a corpus of approximately 190 million web documents and 70,000 machine-written search queries in ten languages. The queries were reformulated by agents from real user queries and their conversation threads in production systems, rather than written directly by benchmark annotators.
They evaluated 13 lexical, dense, and late-interaction retrievers against three fixed relevance sets: documents cited by agents, production ranking signals, and a combined set that supplements both with LLM judgments for previously unlabeled documents. They also tested whether carefully sampled subcorpora could approximate evaluation on the full collection.
Key findings
- The relative ordering of the 13 retrievers was largely unchanged across the three relevance-judgment sets.
- Retriever performance and rankings differed substantially across topical domains, query languages, and query types, indicating that aggregate scores conceal important variation.
- A subcorpus containing about one-third of the 190 million documents preserved the full-corpus model ranking under the combined judgments when selected using reciprocal rank fusion across pooled retrieval runs.
- Evaluating on that smaller corpus increased absolute Recall@1000 by 3–7 points, showing that sampling retained comparative rankings but made retrieval measurably easier.
Why it matters
Most public retrieval benchmarks pair many queries with relatively small corpora or large corpora with few queries, and typically emphasize human-written searches. Q2D-Web offers a larger-scale test of the machine-generated reformulations that retrieval components actually receive in agentic RAG pipelines, while its sampling results provide a less expensive—though imperfect—route to comparative evaluation.
Caveats
The relevance labels are proxies derived from agent citations, production rankings, and LLM judgments rather than exhaustive human assessment, so they may reproduce biases from deployed systems and automated graders. The sampled evaluation preserves rankings only under the reported combined judgments, inflates absolute recall, and may not generalize to new retrievers that were absent from the pooling process.