With access to GPT-4o, doctors scored higher in three countries. The control group reveals why

Access to a capable language model can improve a doctor’s score on a difficult clinical case, but that improvement needs to be measured rather than imagined. A multi-country randomised trial published in npj Digital Medicine assigned 249 physicians in Indonesia, Kenya and the Netherlands either access to GPT-4o or a control condition while they worked through clinical scenarios. The assisted groups performed better in all three settings. That is a real, measured positive effect on a test of physicians’ clinical reasoning, even though no patient was treated as part of the scoring exercise.[1][2]

The researchers reported the largest absolute gap in Kenya, where assisted physicians scored 18 percentage points above those without the model. Indonesia showed a 10.7-point difference and the Netherlands a 7.2-point difference. Confidence intervals for all three were positive in the paper. The cross-country pattern makes the result interesting for medical education and access to information, but the differences do not establish that a particular doctor would gain precisely that many points, or that the intervention would perform identically in a hospital ward.[1]

The comparator makes the result easier—and harder—to interpret

The control group had no access to usual external resources such as online guidelines or the internet. This matters enormously. The experiment compares a physician with an advanced reference and reasoning tool against a physician asked to complete the cases without comparable outside assistance. It does not establish that GPT-4o outperforms established clinical decision-support software, a curated medical reference service or a colleague using current guidelines. It is evidence for the value of model access under those test conditions, not a ranking of all ways to support a clinician.[1][3]

Clinical case scoring is nonetheless an important intermediate outcome. Experienced doctors make decisions under time pressure and incomplete information; tools that help structure differentials or flag an overlooked possibility deserve empirical investigation. Random allocation reduces some sources of selection bias between the groups. Testing across different economic and health-system contexts also exposes assumptions that can be hidden in a single-country benchmark. But translation into practice still depends on connectivity, medical terminology, local disease prevalence and available treatments.[1][2]

What would count as a patient benefit?

The study did not measure patient recovery, hospitalisations, mortality, prescribing mistakes or missed diagnoses in actual care. Harms were not assessed. Those limits should sit beside the headline scores, not in an unread footnote. Higher clinical-case performance can sometimes indicate better decisions, but an apparently persuasive model can also be wrong and alter how a clinician weighs evidence. Before routine deployment, trials must test whether the tool reduces consequential errors compared with current practice and how rapidly professionals detect inaccurate suggestions.[1]

There are further adoption issues. A model can assist reasoning and still generate fabricated citations, use guidelines that do not apply locally or obscure its uncertainty. Clinical confidentiality and data residency complicate access. Doctors may become over-reliant if the system’s fluent response earns more trust than its evidence. Conversely, a blanket refusal to evaluate the technology would miss a potentially scalable support mechanism for settings in which specialist consultation or updated references are difficult to obtain.[1][3]

The most useful next trial would compare GPT-4o-assisted care, an up-to-date conventional reference service and existing standard practice on actual clinical tasks, with patient-safety monitoring and independent review. It should stratify by experience and healthcare setting, report costs per consultation and record both improvements and harms. A follow-up after several months could determine whether physicians learn transferable skills or simply become accustomed to relying on the model.[1]

For now the result is best understood as a measurable gain in clinicians’ simulated-case performance in three countries. That is credible and potentially beneficial to patients if it can be reproduced safely in practice. It is not permission to substitute a chatbot for a qualified physician. This is why the evidence is promising: it identifies a practical capability worth testing under harder, real-world conditions rather than merely announcing that AI might someday save lives.[1][2][3]

Report an error or suggest a correction