AI Beats ER Physicians in Diagnostic Accuracy: Harvard Study Sparks Debate

Harvard Medical School and Beth Israel Deaconess Medical Center evaluated OpenAI’s large-language models o1 and 4o on 76 emergency-room records. At triage, o1 gave an exact or very close diagnosis in 67 % of cases, versus 55 % and 50 % for two physicians. Accuracy rose to 72 % after more data and 81 % at admission.

Harvard Study Finds OpenAI o1 Ahead Of Doctor Baselines

• Sample: 76 ER patients.

Harvard Study Finds OpenAI o1 Ahead Of Doctor Baselines

• Triage: o1 67 % vs physicians 55 %/50 %; post-triage 72 % vs 68 %; admission 81 % vs 73 %.

Harvard Study Finds OpenAI o1 Ahead Of Doctor Baselines

• Management-reasoning: AI 89 % median vs 34 % for physicians on five complex cases.

YnetNews

• Model comparison: o1 outperformed 4o and GPT-4 (88.6 % vs 72.9 % on 70 cases).

YnetNews

Higher early-stage accuracy could cut diagnostic errors and aid clinicians during rapid triage. The study suggests AI could serve as a “second opinion” tool, but AI recommendations may also prompt unnecessary testing, raising safety and cost concerns.

AI Diagnosis Accuracy Surpasses Emergency Room Doctors in Groundbreaking Harvard Study

Lead author Arjun Manrai said the model “eclipsed both prior models and our physician baselines” and urged urgent prospective trials before clinical use. Co-author Adam Rodman warned that “no formal framework exists for accountability” and stressed patients still want human guidance for life-and-death decisions. Peter G. Brodeur highlighted the unprecedented benchmark scores while noting AI may suggest superfluous tests. David Reich emphasized the need to integrate AI in ways that improve care.

Critics note the study’s reliance on text-only records, modest sample size, and lack of real-world outcome data. Wei Xing warned the research does not identify patient subgroups where performance may lag and cautioned against substituting AI for professional advice. Concerns about overreliance, bias, and lack of regulatory accountability were voiced.

AI Diagnosis Accuracy Surpasses Emergency Room Doctors in Groundbreaking Harvard Study

All sources report identical accuracy figures, but none provide data on non-text inputs like imaging or vitals. The study does not assess AI’s impact on outcomes or examine disparities across age, language, or socioeconomic status.

AI Diagnosis Accuracy Surpasses Emergency Room Doctors in Groundbreaking Harvard Study

• “We tested the AI model against virtually every benchmark, and it eclipsed both prior models and our physician baselines.” — Arjun Manrai, Harvard Medical School

AI Diagnosis Accuracy Surpasses Emergency Room Doctors in Groundbreaking Harvard Study

• “Currently, no formal framework exists for accountability around AI diagnoses.” — Adam Rodman, Beth Israel Deaconess Medical Center

AI Diagnosis Accuracy Surpasses Emergency Room Doctors in Groundbreaking Harvard Study

• “Management reasoning is likely a more complex task than diagnostic reasoning,” — Peter G. Brodeur, Beth Israel Deaconess Medical Center

YnetNews

• “It does not demonstrate that AI is safe for routine clinical use, nor that the public should turn to freely available AI tools as a substitute for medical advice.” — Wei Xing, University of Sheffield

YnetNews

The team will launch prospective, multi-site trials with larger cohorts and non-text data such as imaging and labs. Parallel work aims to create accountability frameworks and assess AI’s effect on outcomes, cost, and workflow integration.

AI Diagnosis Accuracy Surpasses Emergency Room Doctors in Groundbreaking Harvard Study