A study comparing ten GPT-5-based configurations on 161 clinician-validated medical cases found that adding retrieval or agent workflows did not consistently improve diagnostic reasoning. The authors report benefits from explicit reasoning and good use of patient history, alongside failures in image interpretation, search and evidence integration.[1]
The comparison and its reported direction
| Measure | Value | Context |
|---|---|---|
| Case set | 161 | Clinician-validated real-world multimodal cases |
| Systems | Ten GPT-5-based configurations | Different reasoning, retrieval, image and agent arrangements |
| Assessment scope | Five dimensions | Diagnostic accuracy, reasoning quality, retrieval accuracy, evidence integration and robustness to low-quality inputs. |
| Strengths reported | Clear inputs and explicit reasoning | More specific, plausible differential diagnoses |
| Failure modes reported | Images and evidence integration | Visual mistakes and inconsistent retrieval could lower performance |
The missing comparison is the size of each gain or loss
The reported direction matters, but it does not tell a reader how far apart the ten systems were. Without the configuration-level results, this account cannot identify a numerical winning margin, rank the configurations or say that every tool-rich system performed worse. The finding is that complexity did not reliably deliver an improvement.[1]
Why an extra component can change the answer in either direction
A retrieval module can add a relevant clinical reference or introduce a mismatched one. An image module can add a useful observation or a visual interpretation error. The resulting system has to reconcile those additions with the history already in the case. More information helps only when the added information is interpreted and integrated correctly.
A useful follow-up comparison would hold the cases fixed and record which answers changed after each component was added. Count errors corrected and previously correct answers spoiled separately. That exposes whether a similar overall score hides a different pattern of gains and losses.
Illustrative arithmetic, not results from the study
| Change after adding a component | Invented count | Effect on correct answers |
|---|---|---|
| Previously wrong answers corrected | 12 | +12 |
| Previously correct answers made wrong | 9 | −9 |
| Net change | 3 | +3, not +12 |
The study concerns diagnostic reasoning on a fixed case set. Evidence that a configuration helps on those cases is distinct from evidence of patient benefit or safe autonomous clinical use. Its practical research lesson is to measure the contribution of each addition instead of treating a larger system diagram as evidence of a better diagnosis.
Sources & further reading
- The complexity paradox of generalist AI agents in medicine: exploring GPT-5 for real-world multimodal medical diagnosis npj Digital Medicine · 9 Oct 2026 · Accessed 14 Oct 2026