Microsoft’s diagnostic AI reaches 83% on complex cases by choosing what to ask next

Microsoft’s diagnostic AI reached 83% accuracy on 304 complex medical cases by choosing questions and investigations as it worked towards a diagnosis. Generalist physicians achieved 36% with external reference resources and 22% without them in the same staged evaluation. The result, published in Science on 22 October, tests a skill that conventional medical quizzes largely skip: deciding what evidence to request next.[1]

MAI-DxO, the Microsoft AI Diagnostic Orchestrator, coordinates several virtual diagnostic roles and lets them revise an assessment as information arrives. On cases based on more typical emergency-department presentations, Microsoft reports 80% accuracy against 47% for physicians. Both comparisons use staged cases, not autonomous treatment of patients. Their value is that they expose the decision process behind an answer.[2][1]

The improvement is in the questions

Conventional medical question-answering scores often give a system nearly everything it needs before the examination begins. A fluent model can succeed at recognizing the correct answer while having no demonstrated ability to decide whether an additional scan is useful, whether a blood test is premature or when another test would merely add cost and uncertainty. Microsoft's SDBench attempts to expose that missing skill by revealing information in response to requests. Diagnostic accuracy is scored alongside the path through the case and estimated investigation costs.[1]

DxO is not merely a single model asked to respond in one burst. The orchestrator coordinates several virtual diagnostic perspectives, has them propose possibilities, question assumptions and argue over what evidence should come next. The procedure resembles a case conference in design, though it is not a room full of independently trained human specialists. Different roles may depend on the same underlying language-model capabilities. Multiple simulated voices do not guarantee independent information or independent errors.[1]

A more cost-constrained configuration retained 81% accuracy on the difficult NEJM cases at less than half the estimated investigation cost of comparison approaches. The estimate covers tests requested in the benchmark. It does not include the full costs of hospital care, such as staffing, delays, incidental findings and follow-up. The result shows a different accuracy-cost balance within the experiment.[1]

Why the human comparison needs careful reading

NEJM clinicopathological conferences were selected partly because they are exceptional diagnostic puzzles. A generalist doctor working alone, even with reference materials, is not the same comparator as a hospital team that includes the relevant specialists, laboratory support and opportunities to examine a patient. In an actual department, cases are handled collaboratively and patients can react, clarify and deteriorate. A static set of written cases cannot replicate all of that. The emergency-department-derived cases are closer to ordinary practice but remain staged evaluations rather than live treatment.[1]

Microsoft plans prospective evaluation with Beth Israel Deaconess Medical Center, including patient interaction and comparison with physicians’ independent assessments. That moves the work towards the practical questions left by written cases: how the system handles ambiguous descriptions, incomplete records and patients whose condition changes while the diagnosis is being considered.[1]

From a correct answer to a useful consultation

A well-structured question can make bad medical reasoning seem more professional. An AI could follow an impressively economical sequence to a confident but wrong conclusion, or request a test because it recognizes a familiar pattern in the case description. Clinicians may become less vigilant when a system gives apparently coherent explanations. Success requires measuring not only whether the final answer matches a reference but whether the model knows when the evidence is inadequate and makes its uncertainty useful to the person responsible for care.[2][1]

The contribution is a way to test and organise diagnostic reasoning over time. Instead of rewarding only the final answer, SDBench records the questions, investigations and estimated costs along the route.[1]

Report an error or suggest a correction