Microsoft’s diagnostic AI reached 83% on staged complex cases

Microsoft’s diagnostic orchestrator reached the correct answer in about 83% of 304 difficult medical cases while choosing which questions to ask and tests to request along the way. Physicians given textbooks and internet resources reached about 36%. The comparison, reported in Science and described by Microsoft on 22 October, tests a consequential part of medicine: how to gather enough information to distinguish competing diagnoses.[1]

Microsoft calls the system MAI-DxO, short for Diagnostic Orchestrator, and the assessment Sequential Diagnosis Benchmark, or SDBench. The orchestrator uses virtual clinical roles that propose and challenge explanations as successive fragments of a case are disclosed. It is not simply asked a multiple-choice medical exam question with all the relevant findings already printed underneath.[1]

Two datasets, different clinical challenges

The difference between the two datasets matters. Journal case conferences deliberately select unusually challenging, educational puzzles and are not a representative sample of patients arriving at a GP surgery. Emergency-department records create a different test, but turning an existing case into a sequential simulation still changes what the participants experience. Real patients leave out information, contradict themselves, deteriorate while waiting, resist investigations or present with multiple competing concerns. A scripted reconstruction cannot capture all of that.[1]

Cost also influenced the system’s choices. Microsoft reports that a configuration designed to economise on investigations reached 81% accuracy on the difficult cases at less than half the comparison cost. These figures attach estimated costs to the tests requested in a simulation. Whether the same choices reduce hospital spending depends on what happens after the diagnosis, including treatment and any later investigations.[1]

Choosing the next useful question

The architecture changes the question being asked of AI. Rather than seeking one answer from one model, the orchestrator can produce several candidate explanations, interrogate their weaknesses and choose investigations with an eye to the information each could add. This resembles a disciplined diagnostic conference more than a chatbot providing instant advice. The researchers report that the method's benefit persisted with newer underlying models, suggesting that the organisation of reasoning contributes something beyond a single model's medical knowledge.[1]

Imagine a patient with breathlessness, fever and a rapid heartbeat. Several disorders might initially fit. A sensible next step is not necessarily every available test; it is the history, examination or investigation most likely to separate dangerous explanations. Such an illustration is not a case from the study, but it explains why ordering the right information in the right sequence can be as important as recalling a disease name.[1]

Moving from records to patients

Microsoft plans prospective work with Beth Israel Deaconess Medical Center in Boston in which patients interact with the system across different hospital settings. The move matters because a live interaction can reveal problems that a completed record conceals: unclear descriptions, interrupted conversations and changes in a patient’s condition.[1]

A useful clinical evaluation would connect diagnostic decisions to what follows: how quickly urgent conditions are recognised, which investigations are ordered and how clinicians respond to the system’s uncertainty. The retrospective study makes that next step worth testing. Its distinctive contribution is to assess the sequence of decisions that produces an answer, alongside the answer itself.[1]

Report an error or suggest a correction