The authors constructed SelfDx, a benchmark of 100 matched pairs of health questions, each consisting of a "caution case" where a serious alternative remains possible and an "agreement case" where the user supplies additional evidence that rules out that alternative.
New benchmark shows health chatbots agree with self-diagnosis when they should caution
Zhang et al. introduce SelfDx, a paired benchmark of 100 caution and agreement scenarios, finding that even the best frontier model reaches only 77% appropriate caution on time-critical condition cases.
Top University
Marilyn Zhang · Tianfeng Chen · Patricia M Pechter · Mark E. Whiting
Pareto Inc · NYU Langone
Research Digest··3 min read
Zhang et al.
§