Skip to content
AI in Healthcare

Article

HIPAA + LLMs in 2026: BAAs, PHI handling, and the model-vendor landscape

Which frontier-model vendors will sign a Business Associate Agreement, what their PHI-handling terms actually say, and what deployment architectures keep clinical LLMs on the right side of HIPAA.

By AI in Healthcare Editorial Updated
  • HIPAA
  • BAA
  • LLM
  • clinical-LLM
  • PHI
  • compliance
  • OpenAI
  • Anthropic
  • Google
  • Azure
  • vendor-landscape

Ask a hospital CISO what keeps them up at night about clinical LLMs and you will very rarely hear “hallucinations.” The answer is almost always PHI — where the prompts and completions go, who touches them at the vendor, whether there is a signed Business Associate Agreement, and whether the vendor’s terms of service actually describe a HIPAA-compliant service or just a “we won’t look at your data” promise wrapped in marketing language.

This is a mostly-solved problem now for the largest frontier-model vendors. But the details still matter — and they still trip up procurement and privacy officers who have not looked at the specific product tier, endpoint, or region their engineering team is planning to use. This piece walks through the 2026 HIPAA-and-LLMs landscape: what a BAA does, which vendors offer one and on which product, how PHI is (and isn’t) handled in practice, and what deployment architectures make the resulting stack defensible.

What a HIPAA BAA actually does — and doesn’t do

A Business Associate Agreement is the contract a HIPAA covered entity signs with a business associate that will create, receive, maintain, or transmit Protected Health Information on the covered entity’s behalf. It is a required document under the HIPAA Privacy Rule and the HITECH-era Omnibus Rule. It is not, however, a technical certification. A signed BAA obliges the vendor to specific PHI-handling obligations — subject to the same breach-notification and enforcement rules the covered entity is subject to — but it does not, on its own, tell you what happens to a prompt after it leaves your firewall.

For clinical LLM deployments, the BAA is a necessary but not sufficient condition. You also need:

  • Product-tier alignment. Not every product from a BAA-eligible vendor is covered under the BAA. The consumer chat product is almost never in scope; the enterprise API endpoint usually is.
  • Region alignment. The BAA typically only covers usage in specific data-center regions. Falling out of region — even accidentally, via a routed request — takes you outside the BAA.
  • Zero-retention or bounded-retention configuration. Most vendors require you to explicitly enable a zero-retention or short-retention mode; the default retention window is not always compatible with a covered entity’s own retention policies.
  • Sub-processor discipline. The BAA covers the vendor; sub-processors need their own downstream flow-through. Vendors that use their own hosted infrastructure are simpler than vendors that layer on top of hyperscaler infrastructure they do not fully control.

Sources on the underlying regulatory frame:

The vendor landscape in 2026

Every serious frontier-model vendor and every hyperscaler that resells or hosts one has published a HIPAA/BAA posture by 2026. The details differ in ways that matter. The four postures below cover most of the market.

OpenAI

OpenAI signs BAAs for enterprise API and ChatGPT Enterprise / ChatGPT Business customers with an explicit BAA amendment. The BAA covers the API endpoints used by the customer’s account and, for the chat surfaces, the enterprise workspace. Consumer ChatGPT (free and Plus tiers) is explicitly not covered. The default enterprise API posture is zero-data-retention on request, meaning inputs and outputs are not persisted after the response is returned. Fine-tuning and evaluation workflows require additional review because the training-data lifecycle is longer than the request lifecycle.

Where healthcare deployments most commonly go wrong with OpenAI: someone on the team spins up a personal ChatGPT account for exploratory PHI-adjacent work; a developer forgets that the Assistants API’s default behavior retains conversation state; or logging middleware written for observability ends up persisting request bodies containing PHI in a non-BAA-covered store.

Anthropic

Anthropic signs BAAs for its direct API and Claude for Enterprise offerings. The BAA covers the Claude models served from Anthropic-owned infrastructure; usage through cloud-marketplace resale on AWS Bedrock or Google Vertex is covered under those hyperscalers’ BAAs instead (see below). Anthropic’s default enterprise API posture is zero-retention outside a short abuse-monitoring window; the Enterprise product surface layers on additional controls (workspace-scoped keys, audit logs, and admin visibility).

The most common Anthropic HIPAA pitfall in 2026: teams route through a marketplace partner without realizing the BAA question flows to that partner. The technical behavior is nearly identical; the contract lineage is not.

Google (Gemini + Vertex AI)

Google offers a BAA covering Google Cloud, which includes Vertex AI’s generative-AI endpoints (Gemini models). The consumer Gemini product and Bard-lineage endpoints are not covered. The Vertex AI BAA has been in place since the mid-2010s (originating with the broader GCP BAA) and Gemini-family endpoints were added to the covered-services list as they became generally available.

Vertex AI’s PHI handling defaults are more granular than OpenAI’s or Anthropic’s: retention is configurable per project, logging can be routed through Cloud Logging with its own PHI controls, and the encryption / KMS / VPC-service-control layers give a covered entity more surface to configure. That granularity is a strength and a weakness — teams that don’t configure it get default behavior that may not match their internal PHI policies.

Microsoft (Azure OpenAI Service + Azure AI Foundry)

Microsoft signs BAAs covering Azure, which includes the Azure OpenAI Service and the newer Azure AI Foundry offerings. This is the highest-volume path to enterprise LLMs in healthcare in 2026, because Microsoft already has a signed BAA with essentially every large U.S. health system as part of the Microsoft 365 / Azure enterprise agreements those systems already have.

Azure OpenAI PHI handling: content filtering runs by default and can log flagged content for abuse review; healthcare customers routinely file for the abuse-monitoring-opt-out (Microsoft’s process for turning off the default logging), which is required for a fully PHI-safe posture. The BAA covers the service; the abuse-monitoring opt-out is separate paperwork.

Hyperscaler bedrock / model-garden pattern

AWS Bedrock and Google Vertex Model Garden host Anthropic, Meta, and other third-party models. The BAA in these cases is between the covered entity and the hyperscaler (AWS or Google), not the model vendor directly. This is fine — actually simpler in some ways, because the hyperscaler is already a signed business associate of most large health systems — but the contract path is worth understanding when you are drawing your data-flow diagram.

Deployment architectures that make this defensible

The most-defensible clinical LLM deployment patterns in 2026 look similar across vendors:

Pattern 1 — Enterprise API with zero-retention and application-level PHI logging

The vendor’s default enterprise API endpoint is used with zero-data-retention enabled. Prompts and completions are logged inside your own environment — in a system already covered by your existing PHI logging and retention policies — rather than at the vendor. Fine-tuning, evaluation, and prompt-development workflows use de-identified data.

This is the default for most large health-system deployments and is the pattern the HIPAA + BAA glossary entry implicitly describes. It works well for ambient scribing, clinical documentation, chart summarization, and inbox-message drafting.

Pattern 2 — Hyperscaler-hosted with private networking

The model runs on a hyperscaler you already have a BAA with, inside a private network segment with no public egress. PHI never leaves your VPC / VNet. Retention is controlled at the hyperscaler-project level. This is the pattern favored by health systems that already have a mature cloud security posture — Azure private endpoints for Azure OpenAI, or GCP private service connect for Vertex AI, are the two most common instances.

Pattern 3 — On-premises / dedicated instance

For the most PHI-sensitive workflows — psychiatric notes, substance-use records covered by 42 CFR Part 2, certain research-adjacent workflows — some health systems run open-weight models on their own infrastructure or a dedicated single-tenant vendor deployment. Llama-family, Mistral-family, and specialized medical open-weight models see this pattern most often. The technical bar is higher (you own the GPU capacity planning, model updates, and safety-evaluation lifecycle), and the model capability ceiling is lower than the frontier models — but no data ever leaves your environment.

For a picture of how ambient scribes fit this pattern, see the ambient AI scribes adoption article and the VA nationwide ambient scribe rollout.

Pattern 4 — Retrieval-augmented with tokenization at the boundary

Increasingly common for patient-facing chatbots and inbox-triage systems: PHI is tokenized (or replaced with reversible pseudonyms) at the application boundary before hitting the model, and re-hydrated in the response before it reaches the clinician or patient. This adds engineering complexity but pushes the sensitivity boundary further from the vendor endpoint. Well-architected versions of this pattern reduce the effective PHI surface at the vendor to metadata that would not be identifiable on its own.

Common HIPAA-and-LLM mistakes

The pattern of what goes wrong is consistent across health systems that have engaged us with post-mortems on near-misses:

  1. Shadow-IT LLM adoption by clinical or operations staff. A well-meaning nurse manager builds a prompt library in a personal ChatGPT account and starts pasting in real PHI. This is the single most common HIPAA-LLM incident category and it is almost entirely a governance and enablement problem, not a technology problem.
  2. Observability middleware persisting request bodies. Application-performance-monitoring tooling captures a sample of API request bodies for latency debugging. Those samples include prompts. Those samples land in a system without a BAA.
  3. Region routing. A load balancer routes traffic to an out-of-region endpoint under high load. The endpoint may be technically identical, but it is outside the BAA’s covered geography.
  4. Sub-processor drift. A vendor adds a new sub-processor (a caching layer, a safety-classifier vendor, a new logging destination). The BAA does not automatically flow through; the covered entity has an obligation to review.
  5. Fine-tuning without a data-lifecycle review. Fine-tuning with PHI has different retention semantics than inference. Vendors handle this differently; some treat fine-tuning data as covered under the same BAA, some require additional agreement, some don’t support it under BAA at all.

What to ask your vendor

If you are a health-system CIO, CISO, or privacy officer sitting down with an LLM vendor in 2026, five questions surface most of what matters:

  1. Which product tier of yours is covered under our BAA — and can you show me the exact endpoint, region, and product SKU?
  2. What is the default retention behavior, and how do we set it to zero (or the shortest window your abuse-monitoring policies allow)?
  3. Who at your company can access prompts and completions for support, safety, or research purposes — and what does the audit trail look like?
  4. What is your sub-processor list, how are we notified of changes, and are your sub-processors flow-through under our BAA?
  5. If we fine-tune or use your evaluation tooling, what changes about the data lifecycle?

Vendors who cannot answer these crisply in 2026 are not ready for enterprise healthcare deployment. Vendors who can are the ones you will see mentioned in the mature health-system governance programs profiled in the health-system AI governance article.

Where this goes next

Three trends worth watching through 2027:

  • BAA coverage will expand to more product surfaces. Voice-mode APIs, real-time streaming endpoints, and agentic-tool-use endpoints are the current growth edge; each has its own BAA question that vendors are working through.
  • On-premises open-weight models will get more clinically capable. The gap to frontier models is narrowing on healthcare-specific evaluations, and the PHI-never-leaves-the-building deployment posture will become the norm for the most sensitive workflows.
  • Audit expectations will tighten. Health-system procurement RFPs are increasingly asking for SOC 2 Type II + HITRUST + BAA + a specific PHI-flow diagram. Vendors without all four will be filtered before they even see the pilot.

HIPAA-and-LLMs is not the mystery it was in 2023. It is a solved set of contract and architecture patterns. The mistake most teams still make is treating “we have a BAA” as the end of the conversation rather than the beginning.