What they did

The authors built a probabilistic evaluation framework that jointly models whether retrieval succeeded, whether the generator answered or abstained appropriately given that outcome, and whether the user ultimately received a correct answer. This separates task success from generator success, or policy adherence.

They applied the framework to 27 combinations spanning three datasets, three retrievers, and three generators. They also modeled LLM-as-a-judge labels as calibrated noisy observations that can be combined with limited human annotations.

Key findings

• The conditional decomposition distinguished RAG systems that appeared equivalent under marginal, end-to-end metrics, revealing differences in behavior after retrieval success or failure.
• Across the 27 configurations, task correctness and appropriate generator behavior were not interchangeable: a system can reach the right user-facing outcome without consistently following the desired answer-or-abstain policy.
• Retrieval-success annotations carried more information than task-success annotations for estimating policy adherence; the authors explain this asymmetry information-theoretically.
• Calibrated LLM-judge outputs could be incorporated alongside human labels within the same Bayesian model rather than treated as equally reliable ground truth.

Why it matters

RAG evaluation often compresses retrieval and generation into a single accuracy score, obscuring where failures originate. This framework offers a way to compare pipelines diagnostically, allocate annotation effort more efficiently, and use inexpensive automated judgments while explicitly accounting for their errors.

Caveats

The reported experiments cover three datasets, three retrievers, and three generators, so the findings may not transfer unchanged to other domains, retrieval settings, or abstention policies. The abstract also does not report effect sizes or how robust the conclusions are to model assumptions and miscalibration of the LLM judge.