Tuesday 15 September 2026
Evening EditionRSS

ZOTPAPER

News without the noise

ZOTPAPER
Safety & Evals

Medical AI models often give right answers with disconnected reasoning

Across 14 language models, perturbing or removing visible medical reasoning usually did not alter answer accuracy.

PaperIndustrycs.AIarXiv:2608.24790v1 ↗
Mengzhu Xu · Jifan Gao · Xia Jiang · Yaoxin Wu · Xi Long

Eindhoven University of Technology · Dana-Farber Cancer Institute

Research Digest··2 min read
Read the paper →
The authors audited medical chain-of-thought by applying 30 clinically motivated perturbations to model rationales and questions across four medical QA benchmarks. They find that answers are frequently insensitive to meaningful changes in the displayed reasoning, suggesting that a correct answer does not necessarily make the visible rationale a faithful account of how it was produced.

What they did

The authors tested 14 language models on four medical question-answering benchmarks. Their audit used 30 perturbation operators—including severity reversals, negation flips, demographic swaps and evidence ablations—to edit either the question or the model’s chain-of-thought rationale.

They jointly measured whether the chain updated and whether the answer changed. They also corrupted chains, removed chain-of-thought prompting and asked two board-certified clinicians to re-annotate 197 perturbed questions.

Key findings

  • On clinically meaningful destructive edits, the panel-wide Chain-Decoupling Rate was 72.9%: in these cases, the chain did not register the edit and the answer did not flip.
  • Corrupting the chain-of-thought left answer accuracy unchanged, indicating that the displayed rationale often was not causally important to the prediction.
  • Removing chain-of-thought prompting did not reduce accuracy.
  • Clinicians judged the original gold answer defensible for 98.5% of the 197 re-annotated perturbations, reducing the likelihood that annotation breakage explains the results.

Why it matters

Medical users may interpret a plausible rationale as evidence that a model reached its answer through valid clinical reasoning. The audit challenges that inference and provides a reusable metric—the Chain-Decoupling Rate—for evaluating whether displayed reasoning responds appropriately to clinically relevant evidence.

Caveats

The evidence comes from four medical QA benchmarks rather than real clinical workflows. For closed-source models, hidden reasoning traces were unavailable, so the authors could assess only answer-side signals; these were consistent with decoupling but cannot directly establish how those systems internally reasoned.

§

Analysis

The study separates answer correctness from rationale faithfulness, two properties often conflated in medical LLM evaluation. Its perturbation-based approach is stronger than judging whether explanations merely sound plausible, although causal claims about internal computation remain limited when model internals are inaccessible.

newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.

Editorial ProcessSubscribe