Cancer treatments work best when matched to a patient's genetic profile. A tumour with a certain gene change might resist a drug, meaning the drug won't help. Doctors rely on expert-curated knowledge bases to know which gene changes cause resistance.
We tested ten language models, software that can answer questions, on over a thousand cases where a gene change should make a cancer drug fail. The models recommended the resisted drug in about a fifth to over four-fifths of cases, depending on the model. To be fair, we compared recommendations for matched drug-sensitive and drug-resistant cases.
The key finding: when resistance was confirmed by two independent sources, four models from three different companies recommended the wrong drug in at most 2.4% of cases. But models sometimes acknowledged the resistance yet still recommended the drug. Repeating a model's own resistance conclusion cut errors to under 3.5%, showing the models follow suggestions even when wrong.
