A Closer Look At Microsoft’s Claim That AI Beats Doctors

🔍 Read the full analysis: A Closer Look At Microsoft’s Claim That AI Beats Doctors on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic system scored 83.2% accuracy on 304 complex case records, compared with 36.1% for generalist physicians using textbooks and the internet, according to a peer-reviewed paper published October 22 in Science. The result concerns a text-based benchmark built from past cases, not care delivered to real patients, and the comparison did not include specialist doctors or medical teams.

The paper, “Sequential diagnosis with language models,” evaluates Microsoft’s Sequential Diagnosis Benchmark, which presents a short case summary and requires the system to request additional information, such as test results, as it works toward a diagnosis. The benchmark scores both the final diagnosis and the estimated cost of the tests requested. On a separate set of 325 emergency department cases, the paper reports scores of 80.3% for MAI-DxO and 46.8% for physicians.

Microsoft’s system uses a language model in a multi-step process: virtual physician agents suggest possible diagnoses, choose questions or tests, and challenge one another’s conclusions. Microsoft’s published results also show GPT-5 alone scored 78.9% on the complex-case benchmark, close to the full system’s 83.2%. The source account says the orchestration layer added about four percentage points and lowered estimated testing costs; the figures are estimates, not measured spending in clinical care.

The study appeared in Science, volume 394, pages 423–428. Its authors are affiliated with Microsoft, including Microsoft AI chief executive Mustafa Suleyman. An earlier version appeared on arXiv in June 2025 as a preprint, before peer review, and reported a different comparison using an earlier model. The newer journal publication is peer-reviewed, but that process does not itself establish that independent researchers have reproduced the findings.

What the Scores Could Mean for Diagnosis

The results matter because diagnosis often requires deciding which question to ask next and whether a test is worth its cost, rather than selecting an answer after all the facts are supplied. The benchmark attempts to represent that sequence, so it tests a different task from a medical exam in which every case detail is already available. It could offer researchers a way to compare how AI systems gather information as well as how often they reach a recorded diagnosis.

But the scores do not show that MAI-DxO is ready to replace doctors, or that it improves patient outcomes. The cases were historical and presented as text; the test did not involve a physical examination, real-time imaging review, or the consequences of making a wrong diagnosis for a patient. The benchmark’s cost figures also rely on estimates. For hospitals and insurers, the relevant questions include whether the system is safe and useful in routine care, how it affects testing and clinician workload, and whether any benefit justifies its costs.

The comparison is also narrower than the headline can suggest. Generalists worked alone, while MAI-DxO uses a panel-like process. The cases were unusually difficult and selected from published reports; the source account says the NEJM set contained no healthy patients or harmless explanations. That limits what the accuracy figures can say about everyday patients, including the risk of unnecessary alarm or testing.

How the Benchmark Was Set Up

In the Sequential Diagnosis Benchmark, a participant begins with a brief summary and requests further case details from a gatekeeper model holding the full record. The benchmark assigns costs to requested tests and adds an estimated physician-visit charge, allowing the researchers to compare diagnostic scores alongside estimated resource use. This design aims to capture the process of gathering information, not just the final answer.

Some methodological details described in the source account come from the 2025 preprint because the full Science article is behind a paywall; those details may not match the final paper in every respect. In that version, a model judge graded diagnoses against the published answer, and physicians reviewing a sample showed good but imperfect agreement with the judge. When a requested test result was absent from the record, the gatekeeper generated a synthetic result. Test prices were mapped to billing codes and a 2023 U.S. hospital price list, with a fixed $300 physician-visit estimate. The authors characterized these costs as approximations.

The preprint described 21 U.S. and UK generalists: 17 in primary care and four hospital generalists, with a median of 12 years of experience. The source account says those physicians were assessed on 56 held-out cases, while the AI was assessed on all 304. It also reports that doctors could not use search engines or language models in the original comparison. The Science paper adds a reference-material arm, reporting 36.1% accuracy for physicians using textbooks and the internet and 22.3% without them.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch

What the Patient Test Has Not Shown

The published benchmark does not establish how MAI-DxO would perform with real patients, incomplete or changing information, physical examinations, or clinicians responsible for care. It also does not establish whether its suggestions reduce diagnostic errors, improve health outcomes, or avoid unnecessary tests. The source account says false alarms could not be measured on the NEJM cases because the set contained no healthy patients or benign explanations.

It remains unclear how well the results would hold against specialists or multidisciplinary teams, rather than generalists working alone, and how the system would perform across a broader range of ordinary cases. The figures are Microsoft’s results for its own system; the source material reports no independent replication. The full Science article is paywalled, so some procedural details available in the source account come from the earlier preprint and may differ from the journal version.

A Prospective Study in Boston

Microsoft’s next stated step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients will interact with the system alongside their normal care. That study could provide evidence about how the tool works in a clinical setting, but its design, timetable, number of participants, and measures of safety or patient outcomes were not specified in the material provided.

Until those details and results are available, the Science paper should be read as a benchmark comparison, not a clinical validation. Further evidence will be needed to assess performance with real patients, compare the system with specialist-led care, and determine whether its diagnostic scores translate into better decisions or outcomes.

Key Questions

What did Microsoft’s MAI-DxO score on the complex cases?

The Science paper reports 83.2% accuracy on 304 complex case records. Generalist physicians scored 36.1% with textbooks and the internet, and 22.3% without those resources.

Was the system tested on real patients?

No. The reported benchmark used historical text-based case records. The source account says Microsoft plans a prospective study at Beth Israel Deaconess Medical Center involving consenting patients alongside their usual care.

Did MAI-DxO outperform specialist doctors?

The comparison described in the source material involved generalist physicians working alone. It did not compare MAI-DxO with specialist doctors or medical teams, so the reported results do not answer that question.

Does peer review mean the results have been independently confirmed?

No. Peer review means the paper was assessed for publication in Science; the source material does not report an independent replication of Microsoft’s results.

Source: ThorstenMeyerAI.com