KlinikeBench is a of 333 clinician-authored tasks that evaluate language models on interactive clinical assessment. Each task simulates a patient encounter where the model must gather history, request examinations, and record a diagnosis within a fixed turn budget. Scoring goes beyond diagnostic accuracy to include process steps like required tool use and must-ask coverage.
The benchmark was used to evaluate 31 models from 7 families. The best models, GPT-6-astra and Claude Opus 5, achieved success on fewer than 30% of tasks, even though they reached 90.7% diagnostic accuracy. This reveals a large gap between diagnosing from a complete chart and performing a full interactive assessment. Some models performed much worse in conversation than on static diagnosis.
The benchmark was developed with over 35 clinicians who authored cases and rated dialogues. Clinicians preferred the simulated dialogues over reference conversations. KlinikeBench provides a more realistic evaluation of clinical LM capabilities, highlighting that high diagnostic accuracy alone is insufficient.