A retrospective study across six matched model pairs and 12 medical specialties found that reasoning models made more accurate primary diagnoses than standard language models, while generating over 34% more false-positive diagnoses and giving less reproducible answers. The Nature Communications study tests diagnostic suggestions from electronic health records, rather than outcomes in live patient care.[1]
A benchmark built around longitudinal records
The researchers developed AIMS-LDR, a long-context benchmark based on real-world electronic health records. Longitudinal records are a more demanding input than a clean exam vignette: the model must identify which symptoms, tests and prior events matter, interpret their order and separate useful evidence from incidental detail. The study reports that the accuracy–safety pattern persisted across prompts, model settings and evaluation methods.[1]
The diagnostic trade-off in AIMS-LDR
| Dimension | Reasoning-model result | Clinical interpretation |
|---|---|---|
| Primary diagnosis | More accurate than matched standard models | Deeper search can improve the leading hypothesis |
| Differential relevance | More clinically relevant alternatives | Broader coverage may help avoid premature closure |
| False positives | More than 34% higher | Extra alternatives can create unnecessary work or intervention |
| Reproducibility | Approximately 40% lower | The same record may yield a different diagnostic set on another run |
Reproducibility is easy to underweight in a demonstration. Two fluent answers may both sound medically sophisticated, yet a system that changes its differential materially between runs is harder to audit and calibrate. In practice, a clinician needs to know whether a new diagnosis appears because the evidence changed or because the sampling process did.
A structured collaboration test with 15 physicians
The researchers also tested a workflow with 15 physicians in which the model broadened the diagnostic possibilities and the clinician checked their reliability. The authors report improved accuracy and fewer false positives in that controlled comparison. This is preliminary evidence for dividing the search and verification tasks, rather than simply asking a doctor to approve an AI-generated list.[1]
The workflow matters more than the slogan “human in the loop.” A clinician who merely signs an AI-generated list may inherit the model’s false positives. A useful division of labour gives the model a defined role—searching for overlooked possibilities—and gives the clinician an explicit task—testing whether each suggestion is supported, actionable and worth the burden it creates.
The physician study was small, and the records and decisions were evaluated under controlled retrospective conditions. The paper does not establish improved patient outcomes, faster diagnosis or lower resource use in live care. Prospective validation is required before treating the collaboration result as a deployment recipe.
Sources & further reading
- The accuracy-safety trade-off of large reasoning models in clinical diagnostics Nature Communications · 29 Oct 2026 · Accessed 30 Oct 2026