The authors created MedEVM, a dynamic benchmark containing 1,050 cases across 24 disease systems.
LLMs often diagnose without correctly valuing available medical evidence
A dynamic benchmark shows that diagnostic accuracy can conceal premature, delayed, or easily redirected decisions, while evidence verification improves performance.
Chinese Tech
Kehua Feng · Yunsheng Lu · Yitong Qiao · Tiantian He · Lei Liu · Yue Shen · +2 more
Zhejiang University · Ant Healthcare, Ant Group
Research Digest··2 min read
Feng et al.
Why this paper
From Ant Healthcare, Ant Group and Zhejiang University
In one line
LLMs often submit correct diagnoses on insufficient or misleading evidence, and verifying evidential support before submission improves diagnostic accuracy.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§