Clinician oversight does not reliably remove every error introduced by automated advice, according to a systematic review in npj Digital Medicine. The analysis, published on 23 October 2026, examined controlled and real-world studies of how clinicians responded when decision-support systems were wrong.[1]
Forty-six reports represented 41 independent cohort clusters, including 40 core cohorts. In three adjusted cohorts, the odds of a correct clinician decision were much higher when automated advice was correct than when it was wrong: pooled odds ratio 10.19, with a 95% confidence interval of 7.43–13.98. This compares two advice conditions in a small subset of the evidence; it is not a tenfold estimate of the benefit of adding AI to unaided care.[1]
What happened when the automated interpretation was wrong
Four real-world studies of automated electrocardiogram interpretation provided a different and operationally important observation. The review estimated that 26.6% of erroneous interpretations were not corrected or recognised. Yet the 95% confidence interval was exceptionally wide, 6.4% to 65.6%, and heterogeneity was high. The spread makes it misleading to quote 26.6% as the probability that any modern AI-assisted doctor will miss an error.[1]
Clinical-oversight evidence by study subset
| Evidence subset | Size | Reported result |
|---|---|---|
| Overall review | 46 reports; 41 independent cohort clusters | 40 core cohorts across varied technologies and settingsSource 1 |
| Adjusted advice comparison | Three cohorts | Odds ratio 10.19 for clinician correctness; 95% CI 7.43–13.98Source 1 |
| Real-world automated ECG interpretation | Four studies | 26.6% of errors uncorrected or unrecognised; 95% CI 6.4%–65.6%; I² 95.8%Source 1 |
| Patient-level adverse events | One retrospective cohort | Harm reporting was sparse across the reviewSource 1 |
The distinction between agreement and error interception matters. Agreement with a correct recommendation is useful; agreement with an incorrect one is a failure of the intended check. Evaluations that deliberately include wrong advice can measure that difference, whereas a high overall accuracy score may hide it.[1]
The authors found newer AI and language-model studies too methodologically diverse for a common estimate. A rule-based ECG interpretation, an imaging classifier and a conversational diagnostic assistant present different errors and opportunities for verification. The review therefore preserves distinctions between tasks and technologies when describing the evidence.[1]
Evaluating the review step itself
The review's practical implication is to measure oversight as part of the system. A clinical evaluation can record which erroneous recommendations are noticed, which are corrected and which reach the final decision. That makes it possible to compare interfaces and workflows on the specific function the human reviewer is expected to perform.[1]
Patient-level adverse events appeared in only one retrospective cohort. That leaves an evidence gap between observing a wrong decision and measuring its consequences for patients, even where the decision-level studies are informative.[1]
The key evaluation condition is what happens when the software is wrong. Measuring the full review workflow under that condition shows whether a clinician detects the error, corrects it and prevents it from reaching the final decision—the function that oversight is meant to perform.[1]
Sources & further reading
- Human oversight of erroneous clinical AI and decision support a systematic review and meta analysis npj Digital Medicine · 23 Oct 2026 · Accessed 27 Oct 2026