The authors introduce a Bayesian framework that models the dependencies between retrieval success, generator abstention, and answer correctness in Retrieval-Augmented Generation systems. They apply it to 27 configurations across three datasets, retrievers, and generators. The conditional decomposition shows that systems appearing equivalent under standard marginal metrics have markedly different behaviors. They also find that annotations of retrieval success are more informative for estimating policy adherence than task success annotations. This provides a more nuanced understanding of RAG system performance and error propagation, and it shows how to combine human and LLM judgments efficiently.
Bayesian framework exposes hidden differences among seemingly similar RAG systems
The model separates retrieval success, appropriate generator behavior, and user-facing correctness while incorporating both human and calibrated automated judgments.
Academic
Pius von Däniken · Felix Matthias Saaro · Mark Cieliebak · Jan Deriu
Centre for Artificial Intelligence · ZHAW School of Engineering
Research Digest··2 min read
In one line
A Bayesian evaluation model jointly examines retrieval success, abstention, and answer correctness to reveal hidden RAG system differences.
Read it as
The authors evaluate 27 retrieval-augmented generation configurations using a Bayesian model structured around how information and errors flow through a RAG pipeline.
The authors built a probabilistic evaluation framework that jointly models whether retrieval succeeded, whether the generator answered or abstained appropriately given that outcome, and whether the user ultimately received a correct answer.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.