News
Stanford-Harvard NOHARM study: physicians chose OpenEvidence over all other clinical AI chatbots combined
The NOHARM study, run by ARISE (a clinical AI research network led by Stanford and Harvard physicians), evaluated 45 large language models and 4 clinical AI systems with 12,747 expert annotations across 4,249 potential clinical actions. When physicians were free to consult any AI tool, they turned to OpenEvidence more than ChatGPT, Claude, Gemini, and all other external AI chatbots combined. The study also found that OpenEvidence produced fewer potentially harmful clinical recommendations than general-purpose LLMs.
- research
- clinical-AI
- benchmarks
- physicians
- LLM
The NOHARM study does something methodologically interesting that most AI benchmarks for clinical use avoid: it measures what physicians actually choose to use in practice, not just how well a system scores on standardized test questions. Most LLM clinical benchmarks — MedQA, USMLE step exams, PubMedQA — measure in-distribution performance on structured medical knowledge. The NOHARM study’s 12,747 expert annotations of real clinical actions, evaluated for potential harm, is a different kind of signal.
The preference gap — OpenEvidence over all other chatbots combined — is large enough to be architecturally meaningful, not just statistical. It suggests that the clinical-specific context OpenEvidence layers on top of its underlying models (the evidence citations, the dosing references, the clinical guideline grounding) translates into physician trust in ways that general-purpose LLMs with a system prompt do not replicate. This is a useful data point for anyone building clinician-facing AI tools: “fine-tuned on medical knowledge” and “trusted in practice” are not the same thing.
The harm annotation methodology deserves attention. ARISE’s annotators rated clinical recommendations for potential patient harm rather than factual accuracy alone. This distinction matters because an AI system can be factually accurate (the drug name and dose are correct) while still being harmful in context (the drug is contraindicated for this patient’s renal function). The finding that OpenEvidence produced fewer harmful recommendations than general-purpose LLMs is more clinically meaningful than USMLE pass rates.
What the study cannot tell us is whether the preference result reflects product quality, brand familiarity, or distribution — OpenEvidence has been deeply integrated into Doximity’s physician network and is embedded in workflows where physicians are already context-switching between clinical information sources. Those effects are hard to disentangle from pure model quality, and future replication with a blinded study design will be important.
Related coverage: health-system copilots topic.
Primary source: Read the full original on OpenEvidence ↗