Clinical LLMs

Clinical LLMs — large language models used in a clinical context — are the fastest-moving corner of healthcare AI. The category barely existed in a serious product form before 2023 and by 2026 covers everything from ambient scribes and inbox-message drafting to patient-facing symptom-checker chatbots to physician-facing evidence-synthesis copilots.

The taxonomy that actually matters

Regulators and health-system CIOs increasingly split clinical LLMs along two axes: who is the user and what is the intended use. In practice you can group most deployments into four buckets:

1. Clinician-facing, non-diagnostic. Ambient scribes, inbox-message drafting, chart summarization, prior-authorization letter drafting, evidence retrieval. The clinician remains in the loop and generally reviews and signs whatever the LLM produces. Most enterprise deployments live here.
2. Clinician-facing, decision support. LLM-assisted differential diagnosis, treatment recommendation, drug interaction reasoning. The 21st Century Cures Act CDS exclusion is doing heavy lifting for many of these — the physician must be able to independently review the basis for the recommendation.
3. Patient-facing, informational. Chatbots that answer general health questions, triage-style questionnaires, patient-portal message drafting. Growing quickly; the FDA’s stance on patient-facing clinical LLMs has been sharpening.
4. Patient-facing, diagnostic or therapeutic. The high-risk bucket — LLMs that claim to diagnose or make a treatment recommendation directly to a patient. These are the closest thing to a “device” claim and get the most FDA scrutiny. UpDoc’s patient-facing LLM clearance is an early data point in what a cleared product looks like here.

Guardrails: what “safe” actually means

The safety conversation for clinical LLMs is not really about model IQ; it is about guardrails — the layers of retrieval, prompt engineering, output filtering, and human-in-the-loop checkpoints that constrain what the model can say and do. In practice, safety is decided by:

• Retrieval grounding. Whether the model can answer from a curated, versioned knowledge base and cite what it used.
• Refusal behavior. How the system handles out-of-scope questions, especially in patient-facing settings.
• Hallucination detection. Whether the vendor has a real post-hoc check on factual claims, or is relying on model self-consistency.
• PHI handling and BAAs. Where prompts and completions are logged, for how long, who at the vendor can access them, and whether the deployment is under a Business Associate Agreement.
• Model provenance. Frontier foundation models are typically served via an API; some deployments swap in an on-premises open-weight model for PHI-sensitive workflows.

The 2026 regulatory picture

The FDA’s stance on clinical LLMs is best described as increasingly explicit. Draft guidance released in 2025 clarified when an LLM-based feature counts as a device, when the Cures Act CDS exclusion applies, and how PCCP-style change control can accommodate LLM updates. The Joint Commission’s RUAIH certification program has become a de-facto governance standard: even health systems that don’t formally certify are using the framework internally.

What we cover

Our reporting in this topic focuses on: FDA clearances of LLM-based products, the shifting line between “device” and “CDS exclusion,” patient-facing clinical LLM deployments, HIPAA/BAA landscape and vendor comparison, guardrail architectures in real deployments, and safety incidents that surface in MAUDE, RUAIH, or the Joint Commission Sentinel Event Alert stream. Explore related articles and news below.