OpenTumorBoard is a built from 611 real-world tumor board patient cases and 19,157 discussion turns transcribed from publicly available YouTube recordings. It evaluates LLMs in two settings: responding to a clinically significant question (SPECIALIST TURN) and simulating a full back-and-forth discussion to reach consensus on cancer care (BOARD SIMULATION).
Fourteen general-purpose and medical LLMs were evaluated. The best model scored 3.43 out of 5 in clinical equivalence to specialist answers and 2.78 out of 5 in alignment with recorded board conclusions. Supervised finetuning and on held-out test data improved performance, showing that real discussion trajectories can aid model adaptation. This work addresses a gap in benchmarking LLMs on realistic, multidisciplinary clinical decision-making.