A doctor using a language model can reason better than a doctor searching conventional resources—and can also be pulled towards a deliberately wrong diagnosis. Both statements are supported by randomised experiments from an overlapping research programme. The apparent contradiction disappears once the quality of the advice and the design of the interface become part of the question.[1][2][3]
Three experiments, one design problem
| Study | Participants and cases | Comparison | Central result |
|---|---|---|---|
| Diagnostic assistance, published | 58 physicians; 342 completed cases | GPT-4o plus conventional resources versus conventional resources alone | 71.4% versus 42.6% mean reasoning score; adjusted difference +27.5 percentage pointsSource 1 |
| Automation bias, preprint | 44 AI-trained physicians; 264 cases | Unmodified GPT-4o suggestions versus suggestions with subtle errors in 3 of 6 vignettes | 84.9% versus 73.3%; adjusted difference −14.0 pointsSource 2 |
| Trust-calibration nudge, preprint | 72 AI-trained physicians; 432 cases | The same ChatGPT-5.1 recommendations, with or without an accuracy anchor and case-specific confidence signal | Nudged group +7.6 points overall; larger +10.3-point estimate on error cases in a post-hoc analysisSource 3 |
The first trial measured assistance, not training
In the published study, 60 physicians were recruited in Pakistan and 58 completed the experiment, split 29 to 29. Each worked through up to six expert-developed cases during a 75-minute session. The LLM group could use GPT-4o alongside conventional resources; the control group used conventional resources without AI features. Across 342 completed cases, the adjusted reasoning advantage was 27.5 percentage points.[1]
The LLM did not make the group meaningfully faster. Mean time per case was about 604 seconds with GPT-4o and 635 seconds without it; the adjusted difference was −6.4 seconds, with a confidence interval from −68.2 to 55.3 seconds and P=0.84. The measured benefit was a better answer, not a proven speed saving.[1]
All participants had completed a 20-hour AI-literacy programme. That means the trial does not isolate whether training caused the improvement. It compares tool access after a common preparation. This matters because “train the user” is often treated as a complete safety strategy, and the next experiment tested that assumption directly.[1][2]
A plausible wrong answer changed what trained physicians saw
The automation-bias trial again used six vignettes, but half of the treatment recommendations contained clinician-designed, subtle and consequential errors. Physicians remained free to open, modify or ignore the suggestions. When they were offered the flawed set, mean diagnostic-reasoning accuracy fell from 84.9% to 73.3%, an adjusted decline of 14.0 points. Top-choice diagnosis accuracy fell by 18.3 points.[2]
This was not a contest between medicine and an autonomous model. It was a test of how a recommendation changes human reasoning when it appears inside the workflow. A persuasive explanation can supply useful differential diagnoses and supporting evidence; it can also become an anchor that reorganises the clinician’s search around the wrong premise.
The third trial put friction only where confidence should fall
The nudge trial used ChatGPT-5.1 recommendations, again with deliberate errors in three of six cases. The intervention displayed two cues: a benchmark-accuracy anchor to calibrate expectations, and a case-specific mean confidence score from Claude Sonnet 4.5, Gemini 2.5 Pro Thinking and GPT-4o, translated into a traffic-light signal. The recommendations themselves were otherwise the same.[3]
The nudged group’s diagnostic-reasoning score was 7.6 percentage points higher than the control group’s, with a 95% confidence interval from 1.4 to 13.9 points. On the deliberately wrong cases, the reported 10.3-point benefit was post hoc and less certain. The result supports selective scrutiny, not a claim that three models voting together certify clinical truth.[3]
- Show where a recommendation came from and which evidence it relies on, rather than presenting fluent advice as an unqualified system answer.
- Make disagreement and low confidence visible at the moment of decision, when the clinician can still change the reasoning path.
- Preserve the physician’s initial assessment or require an explicit comparison before an AI suggestion replaces it.
- Log when the system was opened, which recommendation was shown and what the clinician accepted or rejected.
- Evaluate missed diagnoses, false alarms and patient outcomes in prospective workflows—not only vignette scores.
The sequence is more useful than a binary verdict on medical AI. Correct assistance can extend a clinician’s reasoning. Incorrect assistance can redirect it. A well-timed cue can recover some judgement without forcing scepticism into every case. The design goal is therefore not maximum trust or minimum trust. It is a system that helps a doctor know when the recommendation deserves another look.
Sources & further reading
- Large language model diagnostic assistance for physicians in a lower-middle-income country: a randomized controlled trial Nature Health · 6 Feb 2026 · Accessed 16 Oct 2026
- Automation Bias in Large Language Model Assisted Diagnostic Reasoning Among AI-Trained Physicians medRxiv · Accessed 16 Oct 2026
- Mitigating Automation Bias in Physician-LLM Diagnostic Reasoning Using Behavioral Nudges: A Randomized Controlled Trial medRxiv · Accessed 16 Oct 2026