83%: The Diagnostic Score That Puts Microsoft's AI Ahead of Generalists

The debate over whether artificial intelligence can truly replace human judgment in medicine has intensified with the release of a new benchmark. On Thursday, Microsoft published a study in Science claiming its diagnostic system, MAI-DxO, correctly identified complex medical cases more than twice as often as generalist physicians. Mustafa Suleyman, the CEO of Microsoft AI, framed this achievement as a critical step toward what the company calls medical superintelligence, a state where AI surpasses contemporary human-led health systems in reasoning and coordination.

The system operates by coordinating multiple AI agents that function like a panel of specialists. These agents question the patient, order tests, and debate the diagnosis. In simulations using difficult cases from the New England Journal of Medicine, DxO achieved an 83% accuracy rate. By comparison, primary care doctors and general internists working individually with reference materials were correct only 36% of the time. However, a significant caveat exists: these complex cases are typically handled by specialists or teams in real life, and those professionals were not part of the comparison group.

Skeptics point to the data suggesting that most of the improvement stems from the underlying AI models rather than Microsoft’s orchestration layer. Microsoft tested DxO with models from OpenAI, Google, Anthropic, Meta, and DeepSeek, with the best results coming from OpenAI’s GPT-5 and GPT-5.5. The cost and accuracy trade-offs were mixed. With GPT-5, DxO reduced costs by about $3,000 per case but only improved accuracy by four points. With the newer GPT-5.5, accuracy jumped from 72% to nearly 87% on complex cases, yet gains were minimal for emergency department cases, where both versions scored around 85%.

The orchestration system appears to help less with top-tier models and more with weaker ones. For example, Gemini 2.5 Flash improved from 46.4% to nearly 69% with DxO, but the gap between weaker and stronger models remained larger than the boost provided by the system. Microsoft researchers noted that accuracy gains naturally shrink as models approach theoretical maximums, and a spokesperson argued that DxO helps optimize both accuracy and cost simultaneously.

Suleyman, who previously co-founded DeepMind, says Microsoft is working with the Mayo Clinic to train a healthcare foundation model for release next year. He predicts a coding-style breakthrough for medical records by 2027, allowing for highly accurate predictions of how a medical record might unfold. He expects healthcare models to become an order of magnitude better in capability within the next 24 to 36 months, driven by a flywheel of real-world usage and iteration.

To validate these claims, Microsoft has partnered with Beth Israel Deaconess Medical Center in Boston to evaluate DxO in real-world hospital settings. Patients will receive standard care while consenting patients also interact with the AI. The results will be compared to human decisions. Dominic King, vice president of Health at Microsoft AI, called this the biggest study of its kind, noting that the technology is ready for real-world evaluations. This effort mirrors a similar study by Google and Beth Israel published in The Lancet this month.

Microsoft views this as a differentiator in the competitive frontier lab landscape. The company already sells dictation and radiology tools and sees DxO as an expansion of those efforts. Suleyman stated that the goal is to have the number one state-of-the-art medical clinical decision support product in the coming years, aiming to integrate with clinicians for better outcomes rather than just benchmarking individual tasks.