MedEVM is a dynamic of 1,050 cases across 24 disease systems where evidence arrives turn by turn and an LLM must wait or submit a diagnosis. Testing 9 LLMs revealed four misalignments: models fail to calibrate evidence sufficiency when diagnosing, confidence does not guarantee timely submission, reordering evidence changes diagnoses, and misleading evidence still shifts diagnoses after sufficient evidence is present. EVM predicts errors, and preventing premature submission improves accuracy.
LLMs often diagnose without correctly valuing available medical evidence
A dynamic benchmark shows that diagnostic accuracy can conceal premature, delayed, or easily redirected decisions, while evidence verification improves performance.
Chinese Tech
Kehua Feng · Yunsheng Lu · Yitong Qiao · Tiantian He · Lei Liu · Yue Shen · +2 more
Zhejiang University · Ant Healthcare, Ant Group
Research Digest··2 min read
In one line
LLMs often submit correct diagnoses on insufficient or misleading evidence, and verifying evidential support before submission improves diagnostic accuracy.
Before you read this
Three ideas this paper builds onbenchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →verificationChecking outputs for correctness before accepting them, often to catch errors or reward hacking.341 papers →agent harnessA system that structures, verifies, monitors, or controls how an AI agent interacts with external tools, environments, or execution runs.186 papers →
Read it as
Feng et al. tested nine language models on MedEVM, where medical evidence arrives sequentially and models must choose whether to wait or diagnose.
The authors created MedEVM, a dynamic containing 1,050 cases across 24 disease systems.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.