Clinical LLMs

Clinical LLMs — large language models used in a clinical context — are the fastest-moving corner of healthcare AI. The category barely existed in a serious product form before 2023 and by 2026 covers everything from ambient scribes and inbox-message drafting to patient-facing symptom-checker chatbots to physician-facing evidence-synthesis copilots.

The pace of change makes this a hard topic to track rigorously. New products ship weekly; existing products silently swap out their underlying foundation model every few months; evidence lags; regulators are chasing a moving target. The way to stay oriented is to focus less on the model and more on the deployment: who the user is, what the intended use is, what the guardrails are, and what happens when the system is wrong.

The taxonomy that actually matters

Regulators and health-system CIOs increasingly split clinical LLMs along two axes: who is the user and what is the intended use. In practice you can group most deployments into four buckets:

1. Clinician-facing, non-diagnostic. Ambient scribes, inbox-message drafting, chart summarization, prior-authorization letter drafting, evidence retrieval. The clinician remains in the loop and generally reviews and signs whatever the LLM produces. Most enterprise deployments live here.
2. Clinician-facing, decision support. LLM-assisted differential diagnosis, treatment recommendation, drug interaction reasoning. The 21st Century Cures Act CDS exclusion is doing heavy lifting for many of these — the physician must be able to independently review the basis for the recommendation.
3. Patient-facing, informational. Chatbots that answer general health questions, triage-style questionnaires, patient-portal message drafting. Growing quickly; the FDA’s stance on patient-facing clinical LLMs has been sharpening. The Hippocratic AI coalition rollout is the clearest signal that incumbents are organizing around a shared risk framework.
4. Patient-facing, diagnostic or therapeutic. The high-risk bucket — LLMs that claim to diagnose or make a treatment recommendation directly to a patient. These are the closest thing to a “device” claim and get the most FDA scrutiny. UpDoc’s patient-facing LLM clearance is an early data point in what a cleared product looks like here.

Guardrails: what “safe” actually means

The safety conversation for clinical LLMs is not really about model IQ; it is about guardrails — the layers of retrieval, prompt engineering, output filtering, and human-in-the-loop checkpoints that constrain what the model can say and do. In practice, safety is decided by:

• Retrieval grounding. Whether the model can answer from a curated, versioned knowledge base and cite what it used.
• Refusal behavior. How the system handles out-of-scope questions, especially in patient-facing settings. “I can’t help with that, please contact your clinician” is a safety feature, not a bug.
• Hallucination detection. Whether the vendor has a real post-hoc check on factual claims, or is relying on model self-consistency. The latter is cheaper and less effective.
• PHI handling and BAAs. Where prompts and completions are logged, for how long, who at the vendor can access them, and whether the deployment is under a Business Associate Agreement.
• Model provenance. Frontier foundation models are typically served via an API; some deployments swap in an on-premises open-weight model for PHI-sensitive workflows. Each posture has different availability, update-cadence, and liability consequences.
• Red-teaming cadence. Mature deployments run adversarial testing on new model versions before they reach production; immature ones trust the vendor’s benchmarks.

The 2026 regulatory picture

The FDA’s stance on clinical LLMs is best described as increasingly explicit. Draft guidance released in 2025 clarified when an LLM-based feature counts as a device, when the Cures Act CDS exclusion applies, and how PCCP-style change control can accommodate LLM updates. The Joint Commission’s RUAIH certification program has become a de-facto governance standard: even health systems that don’t formally certify are using the framework internally.

ONC’s HTI-2 rule adds a parallel transparency surface — covered in the HTI-2 enforcement playbook — that applies to predictive DSI regardless of FDA status. Taken together, the 2026 compliance envelope for a clinician-facing LLM deployment is: BAA in place, documented intended use, documented risk management, a clear accounting of model provenance and change control, a surveillance plan, and a path to display the required HTI-2 attributes inside the EHR.

From chatbots to agents

A category-defining shift in 2026 is the move from “LLM responds to a single prompt” to “LLM takes multi-step action on behalf of a clinician.” Our deep-dive on autonomous AI agents on the regulatory frontier and HHS’s RFI on autonomous AI agents make the stakes clear: a chatbot that gives bad advice is a safety event; an agent that places an order, schedules a referral, or sends a patient-portal message is a different kind of thing entirely. The regulatory, liability, and audit frameworks for agentic clinical AI are being written in real time, and most well-governed health systems are not yet comfortable with full-autonomy deployments.

What we cover

Our reporting in this topic focuses on: FDA clearances of LLM-based products, the shifting line between “device” and “CDS exclusion,” patient-facing clinical LLM deployments, HIPAA/BAA landscape and vendor comparison, guardrail architectures in real deployments, agentic AI and multi-step automation, HTI-2 transparency enforcement, and safety incidents that surface in MAUDE, RUAIH, or the Joint Commission Sentinel Event Alert stream. Explore related articles and news below.