Retrieval-based factuality evaluation decomposes LLM medical answers into atomic claims and verifies each against retrieved evidence. This work introduces a taxonomy-based diagnostic for the open-ended MedExpert dataset plus three closed-ended biomedical benchmarks, using LLM-as-judge to label retrieval quality and verifier reasoning failures without requiring gold answers or gold evidence. Across 4 retrieval methods and 6 frontier verifier LLMs, the pipeline recovers only a small fraction of clinician-flagged factual errors on MedExpert, an order-of-magnitude collapse relative to closed-ended benchmarks. Scaling model size, adding reasoning effort, expanding to authoritative web sources, and applying medical fine-tuning all fail to remove the failure modes. The value is in locating failures: retrieval-stage errors concern irrelevant evidence, clinical scope mismatch, trustworthiness, evidence polarity, and chunking, while verifier errors occupy six reasoning steps. These taxonomies give a model-agnostic way to see why end-to-end metrics hide catastrophic performance in open-ended medical evaluation.
Retrieval-based checks miss many factual errors in open-ended medical answers
Across multiple retrievers, evidence sources and verifier models, the authors find persistent failures that standard closed-ended benchmarks obscure.
Top University
Center for Language and Speech Processing · Johns Hopkins University
Research Digest··2 min read
In one line
Retrieval-based factuality evaluation of long-form medical answers fails because scaling retrievers, verifiers, reasoning, and medical tuning does not fix fundamental error modes.
Before you read this
Two ideas this paper builds onRead it as
Huang et al. examine why retrieve-then-verify systems struggle to identify factual errors in long-form medical answers.
The authors evaluated retrieval-based factuality checking on the open-ended MedExpert dataset and three closed-ended biomedical datasets.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.