Microsoft AI diagnostic tool outperforms doctors in clinical tests. Microsoft researchers have developed an AI system that achieved higher d...
 |
| Microsoft AI diagnostic tool outperforms doctors in clinical tests. |
Microsoft researchers have developed an AI system that achieved higher diagnostic accuracy than generalist physicians in tests involving complex medical cases and emergency department records. The findings showed that the system correctly diagnosed 83.2% of complex cases, compared with 36.1% for physicians using reference materials.
The researchers also found that the system's estimated diagnostic testing costs varied depending on whether spending limits were applied. The system, called Microsoft AI Diagnostic Orchestrator (MAI-DxO), works through medical cases in stages rather than relying on a complete set of patient information from the beginning. It determines which questions to ask, what tests to request, and when sufficient evidence is available to make a diagnosis.
AI Approaches Diagnosis
MAI-DxO uses a large language model to simulate a panel of medical specialists with different approaches to diagnosing a patient. The system maintains possible diagnoses, selects investigations, reviews its reasoning, and considers testing costs throughout the process. The research used GPT-5 as the underlying model for its main evaluations. Instead of receiving all medical information at once, the system begins with a limited clinical description and requests additional details as the investigation progresses. As new information becomes available, the orchestrator reviews its working diagnoses and determines whether further tests are necessary. It also considers the cost of additional investigations before deciding how to proceed.
The researchers developed a testing framework called the Sequential Diagnosis Benchmark (SDBench) to assess this process. It uses an automated mechanism to release patient information only when requested, requiring both AI systems and participating physicians to make decisions based on the evidence available at each stage. The benchmark includes 304 clinicopathological conference cases published in The New England Journal of Medicine and 325 emergency department encounters from the MIMIC-IV-ED database.
Each requested test receives an estimated cost, allowing researchers to compare diagnostic accuracy and resource use. The researchers found that MAI-DxO asked an average of 25.5 questions before ordering its first diagnostic test, compared with 5.6 questions among participating physicians. This allowed the system to gather more patient-history information before selecting investigations. Unlike static medical AI benchmarks that present case information upfront, SDBench evaluates diagnostic decisions as additional information becomes available.
AI Outperforms Physicians
On the complex medical cases, MAI-DxO achieved 83.2% accuracy, while generalist physicians recorded 22.3% without reference tools and 36.1% when allowed to consult external resources. The highest-performing physician reached 51.8% accuracy, below the AI system's reported results. The comparison involved individual physicians rather than specialist teams working together on the same cases. The researchers noted that the performance gap might be smaller in comparisons with specialists experienced in the conditions being evaluated. The complex cases were also selected for their diagnostic difficulty and educational value, rather than representing the full range of conditions encountered in routine clinical practice.
The system was also tested on historical emergency department records, where it correctly diagnosed 80.3% of cases, compared with 46.8% for participating physicians. The highest-performing physician in this evaluation achieved 62.5% accuracy. Researchers found that MAI-DxO's incorrect answers generally involved conditions related to the correct diagnosis but with the wrong level of specificity. The physicians' incorrect answers tended to be further from the correct diagnosis. Both evaluations used previously documented medical cases rather than live patient consultations. The researchers recreated the diagnostic process under controlled conditions to assess how the system and physicians selected tests and reached their conclusions.
The study also examined how much each diagnostic investigation would cost based on the tests requested. Researchers assigned estimated prices using standardised US hospital billing data, allowing them to assess whether the system could reach a diagnosis while limiting unnecessary expenditure. Without budget restrictions, MAI-DxO incurred higher estimated testing costs than physicians. On complex cases, its average cost was $5,431 per case, compared with $4,725 for physicians.
In the emergency department evaluation, the corresponding figures were $4,979 and $3,400. Researchers then tested configurations with spending limits. On complex cases, one configuration achieved 80.9% accuracy at an estimated cost of $2,241 per case, while a lower-budget version recorded 79.6% accuracy at $1,303. On emergency department cases, a budget-constrained configuration achieved 73.2% accuracy at an estimated cost of $2,634 per case. Another recorded 69.5% accuracy at $2,037, with both exceeding the average diagnostic accuracy of participating physicians.
The researchers examined both diagnostic accuracy and the cost of reaching a diagnosis, rather than measuring accuracy alone. "We're also measuring how judiciously and carefully it goes through a diagnostic work-up," Harsha Nori, head of AI at Microsoft AI Health, said in an interview with Reuters. The cost figures represent simulated diagnostic expenditure rather than savings recorded in hospitals. They were based on standardised US pricing and did not account for factors such as patient discomfort, the risks associated with invasive procedures, or the time required to obtain test results.
Clinical Testing
Despite the reported results, the study has several limitations. The benchmark relied on text-based medical records and did not assess the system's ability to interpret medical images, pathology slides, or other clinical data directly. The evaluations also excluded challenges associated with real patient consultations, including incomplete or inaccurate medical histories. Although the emergency department dataset contained records from routine hospital admissions, all cases had established diagnoses, and the benchmark did not include healthy patients or benign consultations.
The research measured diagnostic accuracy rather than treatment decisions, patient safety, or clinical outcomes. Its findings therefore do not establish whether the system would improve patient care or outperform physicians in routine clinical practice. The research was conducted by Microsoft employees who hold equity in the company, with eight authors also named on a patent application related to the evaluation framework.
