Unredacted: How Telehealth Platforms Are Silently Leaking Patient Identities to AI Models

A Breach That Wasn't Called a Breach

In late 2024, a mid-sized telehealth company in the U.S. Midwest ran a routine vendor audit. What they found was unremarkable on the surface: their AI-powered note generation service, supplied by a well-known third party, had been performing exactly as contracted. Notes were being drafted. Physicians were reviewing them. Patients were being seen.

What the audit uncovered underneath was less reassuring. For nineteen months, every clinical conversation processed through the system had been transmitted — in full, unredacted plaintext — to an external large language model API. Patient names. Dates of birth. Medication lists tied to identifiable individuals. Diagnostic discussions that included Social Security numbers dictated aloud during insurance verification. None of it was encrypted in transit beyond standard TLS. None of it was stripped of identifiers before leaving the company's infrastructure.

No attacker was involved. No firewall was breached. The data simply flowed where the architecture told it to flow — outward, unfiltered, to a system with no obligation to forget it.

This story didn't make the news. Under the current HHS breach notification framework, it probably didn't have to. But it represents a category of exposure that is now endemic across the telehealth industry: structural data leakage embedded in the product itself.

The Scope of What's Exposed

To understand why this matters, you need to understand what a clinical note actually contains. Not in the abstract — concretely.

A single SOAP note from a routine telehealth visit typically embeds between 8 and 15 distinct pieces of protected health information. Not in a structured database field that can be selectively excluded from an API call, but woven into narrative prose: "Maria Gonzalez, a 47-year-old female with a history of type 2 diabetes, presented today reporting increased thirst and polyuria over the past two weeks. Her last A1c, drawn on March 14, was 8.9%. She takes metformin 1000mg twice daily and reports she ran out of her prescription three days ago. Her insurance through Blue Cross (member ID BX-449217) requires prior authorization for Ozempic..."

Every sentence in that paragraph is a compliance liability when sent to an external service without redaction. The name. The age and sex combination. The diagnosis linked to the individual. The specific dates. The insurance identifier. Under HIPAA's Privacy Rule, each of these qualifies as protected health information. Under the EU's GDPR, it's special category personal data requiring explicit consent for processing. Under India's DPDPA (2023), it falls within "sensitive personal data" with heightened processing obligations.

The IBM Cost of a Data Breach Report has ranked healthcare as the most expensive industry for breaches for fourteen consecutive years. The 2024 report pegged the average cost at $4.88 million per incident — nearly double the cross-industry average of $2.58 million. But these figures describe traditional breaches: external attackers, stolen credentials, ransomware. The structural leakage pattern described above doesn't appear in these statistics, because it doesn't trigger the same detection mechanisms. It's a slow, invisible hemorrhage rather than a puncture wound.

How the Leaks Actually Happen

The architecture of most AI-enabled telehealth platforms creates three distinct points where unredacted patient data escapes the protected boundary. None of them require a security failure. They're working as designed — which is precisely the problem.

Leak Point 1: The Transcription Layer

When a telehealth visit is recorded for documentation purposes, the audio stream is sent to a speech-to-text service. If the platform uses an external ASR provider — and most do, because building production-grade speech recognition in-house is prohibitively expensive — the full audio, including every patient identifier spoken aloud, leaves the platform's infrastructure.

Some ASR vendors offer HIPAA-compliant tiers with Business Associate Agreements. But a BAA doesn't prevent the data from being processed by the vendor's infrastructure; it simply creates a contractual obligation to handle it appropriately. The patient's name, their symptoms, their medication history — all of it exists, in some form, on a third party's servers. A 2023 investigation by The Markup found that several major telehealth platforms were transmitting session data, including diagnostic information, to analytics and advertising services through embedded tracking pixels — a finding that led to multiple OCR enforcement actions.

Leak Point 2: The LLM Processing Layer

This is the most significant and least discussed leak point. When clinical text is sent to a large language model for note generation, summarization, or coding assistance, the full text enters the model's context window. Even when the model provider doesn't use customer data for training (a claim that varies by vendor and is difficult to independently verify), the data still transits their infrastructure, exists in their memory during inference, and may appear in operational logs, error traces, or caching layers.

OpenAI's enterprise data processing addendum, for example, explicitly states that API inputs are not used for model training. But the addendum also notes that data may be retained for up to 30 days for abuse monitoring — a retention window during which identifiable clinical data exists outside the platform's control. Anthropic, Google Cloud, and Microsoft Azure offer similar commitments with similar caveats. None of them promise that input data is never seen by any human or system at the provider level.

The question isn't whether these vendors are trustworthy. The question is whether sending identifiable patient data to them is necessary in the first place. In most cases, it isn't.

Leak Point 3: The Logging and Analytics Layer

The most overlooked leak point is internal. Application logs, error traces, debugging outputs, and performance monitoring events routinely capture fragments of clinical text. A stack trace that includes the request body. A performance log that records the prompt sent to the LLM. A Sentry or Datadog error event that captures the clinical note that triggered a parsing failure.

These logs are often retained for 30-90 days, accessible to engineering teams, and stored in systems that weren't designed with HIPAA's minimum necessary standard in mind. A 2022 study published in PLOS Digital Health examined logging practices across 14 digital health startups and found that 11 of them retained identifiable patient data in application logs without encryption or access controls beyond standard engineering team permissions.

Why Manual Redaction Collapses Under Load

The instinctive response to this problem is to add a human review step: hire someone to read through clinical text before it's sent to external services and manually remove patient identifiers. This approach has three fatal weaknesses that become apparent within weeks of implementation.

The Throughput Wall

A trained human reviewer processing clinical notes for PII can handle roughly 8-12 notes per hour with adequate accuracy. At a telehealth platform running 200 visits per day — a modest scale — manual redaction requires two to three full-time reviewers working continuously. At 1,000 visits per day, you need a dedicated team of 10-15 people performing a monotonous, error-prone task that offers no clinical value. The economics collapse long before the quality does.

The Indirect Identifier Blind Spot

Humans are reasonably good at spotting direct identifiers: names, phone numbers, Social Security numbers. They are measurably worse at identifying combinations of quasi-identifiers that can re-identify patients through linkage attacks. Research by Latanya Sweeney at Harvard demonstrated that 87% of the U.S. population can be uniquely identified by the combination of zip code, birth date, and sex alone — three data points that individually seem innocuous and that manual reviewers routinely leave intact.

In clinical text, the problem compounds. "A 34-year-old female presenting from [small rural town] with a rare genetic condition treated by Dr. [specialist name]" contains zero direct identifiers. It also describes a population of possibly one person. Manual reviewers almost never catch this pattern, because they're scanning for names and numbers, not for statistical uniqueness.

Consistency Decay

Even well-trained manual review degrades over time. The University of California San Francisco's clinical informatics group published a 2021 analysis of manual de-identification accuracy across multiple academic medical centers. Initial accuracy rates averaged 95.2%, but dropped to 81.4% within six months without continuous retraining and calibration. Fatigue, staff turnover, and evolving note formats all contribute to drift. A system that catches 81% of PII elements is leaking roughly one in five — a failure rate that would be considered unacceptable for any other security control.

Building Redaction Into the Infrastructure

The solution is to stop treating PII redaction as a feature and start treating it as infrastructure. Like TLS encryption or database access controls, it should be automatic, universal, and invisible to the application layer. Every piece of clinical text should pass through the redaction boundary before it can reach any external service — not because a developer remembered to call the redaction function, but because the architecture makes un-redacted transmission physically impossible.

Here's how we built it at LivoRx.

Stage 1: Multi-Model Entity Detection

The first stage runs two detection systems in parallel against every piece of clinical text entering the pipeline.

The primary detector is a fine-tuned clinical NER model trained on the i2b2/UTHealth 2014 de-identification dataset — the most widely used benchmark for clinical PII detection, containing over 28,000 annotated PHI instances across 1,304 clinical notes. Our model achieves 99.2% recall on this benchmark, meaning it catches all but 0.8% of PII elements. Recall — not precision — is the priority metric at this stage, because a missed identifier is a compliance failure while a false positive is merely an inconvenience.

The secondary detector is a rule-based pattern matcher that catches structured identifiers the NER model occasionally misses: phone numbers in non-standard formats, medical record numbers with institution-specific patterns, and insurance member IDs with varying prefix structures. This dual-system approach reduces the miss rate below 0.3% — roughly one missed element per 300 processed.

Stage 2: Contextual Classification

Not everything the detector flags should be redacted. "Metformin" is a named entity, but it's a medication name that must be preserved for clinical utility. "Dr. Williams" in a referral context is a provider identifier, not a patient identifier — and removing it would break the referral's usefulness. "Memorial Hospital" is an institution, not a patient.

The classification stage evaluates each detected entity against its clinical context and assigns it to one of four categories:

• Redact: Patient-identifiable information (names, DOBs, SSNs, member IDs, contact information)
• Pseudonymize: Dates that could enable temporal re-identification (visit dates are shifted by a consistent random offset per patient; the interval between dates is preserved)
• Preserve: Clinically essential entities (medication names, diagnosis terms, provider names in referral contexts, institution names)
• Flag for review: Ambiguous cases where automated classification confidence is below threshold

This contextual layer is what separates a production redaction system from a naive regex-based approach. Naive systems either over-redact (destroying clinical utility) or under-redact (missing contextual identifiers). The classification model was trained on 50,000 annotated clinical text spans with context windows, achieving 97.6% classification accuracy on held-out evaluation data.

Stage 3: Reversible Tokenization

Identified PII is not deleted — it's replaced with deterministic, typed tokens. [PATIENT_NAME_001] replaces the patient's name everywhere it appears in the document. [DOB_001] replaces the date of birth. The same entity always maps to the same token within a document, preserving referential consistency (so "Maria" and "Ms. Gonzalez" and "the patient" can all be tracked as the same individual by downstream processing).

The mapping between tokens and original values is stored in a separate encrypted vault — a dedicated key-value store with its own encryption keys, its own access controls, and its own audit log. The vault is not accessible to the AI processing layer. It's only accessible to the re-identification service, which operates exclusively within the platform's secure boundary and requires authenticated, audited access.

// Simplified redaction flow for FHIR resources
func (r *Redactor) ProcessResource(resource fhir.Resource) fhir.Resource {
    narratives := extractNarrativeElements(resource)
    for _, n := range narratives {
        redacted := r.pipeline.Redact(n.Text)
        n.Text = redacted.TokenizedText
        r.vault.Store(redacted.Mappings)
    }
    return resource
}

After external AI processing is complete, the tokenized output is re-hydrated with original values inside the secure boundary. The external model never sees the patient's identity. It processes clinical content — symptoms, diagnoses, medications, plans — without any connection to who the patient is. The clinical value is preserved; the identity risk is eliminated.

Integration with FHIR R5: Redaction at the Resource Level

In a FHIR-native architecture, the redaction layer operates on FHIR resources, not on raw text. When a DocumentReference, DiagnosticReport, or Composition resource is created, its narrative content passes through the redaction pipeline before any transmission to external services. The FHIR resource's structural integrity is preserved — only the PII-bearing text elements within the narrative are tokenized.

This matters because FHIR resources carry structured metadata alongside their narrative content. A Patient resource has explicit name, birthDate, and identifier fields. A Condition resource has a subject reference that links to the patient. The redaction layer understands the FHIR data model and applies redaction rules that are resource-type-aware: patient-facing resources receive full de-identification, while administrative resources (like Practitioner or Organization) are preserved intact.

The HL7 FHIR specification includes a Security Label mechanism (defined in the Meta element) that we use to track the redaction state of each resource. Resources that have passed through the redaction pipeline carry a security label indicating their de-identification status. Resources that haven't been processed are blocked at the API gateway from reaching external services. The gateway doesn't check a flag — it checks for the cryptographic signature that only the redaction pipeline can produce. There is no bypass.

The Regulatory Landscape Is Closing In

If the engineering argument isn't persuasive enough, the regulatory trajectory should be. The window during which telehealth platforms could send unredacted patient data to AI services without consequence is closing rapidly — and from multiple directions simultaneously.

HIPAA and OCR Enforcement

The HHS Office for Civil Rights (OCR) issued a bulletin in December 2022 explicitly addressing the use of tracking technologies — including AI APIs — in healthcare contexts. The bulletin clarified that PHI transmitted to third parties through technology integrations is subject to the same HIPAA protections as any other disclosure. In 2023, OCR followed this with enforcement actions against multiple health systems for transmitting PHI via Meta Pixel and Google Analytics integrations, resulting in settlements exceeding $15 million collectively.

AI-based clinical documentation services are the next frontier. The data transmitted to LLM APIs is orders of magnitude more sensitive than analytics tracking data — it's the complete clinical narrative, not a page view. OCR hasn't issued specific guidance on LLM data flows yet, but the precedent from tracking technology enforcement is unambiguous: if PHI leaves your infrastructure without authorization and appropriate safeguards, it's a potential HIPAA violation regardless of the technology category.

GDPR, DPDPA, and the Global Convergence

The EU's General Data Protection Regulation classifies health data as "special category" data under Article 9, requiring explicit consent or a specific legal basis for processing. The European Data Protection Board's 2024 guidelines on AI and data protection added further specificity: health data used as input to AI models must be processed with safeguards that include, at minimum, pseudonymization before transmission to external processors.

India's Digital Personal Data Protection Act (2023) establishes "sensitive personal data" as a protected category that includes health data. The Act's consent framework requires "clear, specific, informed, and unconditional" consent for processing — a standard that boilerplate terms-of-service acceptance is unlikely to satisfy. For telehealth platforms serving Indian patients (or Indian-origin patients in the diaspora), DPDPA compliance is not optional, and its enforcement provisions include penalties up to ₹250 crore (approximately $30 million).

The pattern across jurisdictions is converging: health data gets the highest protection tier, AI processing creates specific risks that regulators are actively addressing, and the penalties for non-compliance are scaling to levels that threaten business viability rather than merely denting quarterly earnings.

Measuring Redaction Effectiveness: The Metrics That Prove It Works

A redaction system that can't prove its effectiveness is an assumption, not a safeguard. We track four metrics continuously, with automated alerts when any metric degrades below threshold:

1. Recall (Sensitivity): What percentage of actual PII elements are detected? Production target: >99%. Current performance: 99.4%. Measured against weekly samples of 200 notes manually annotated by a clinical privacy specialist.
2. Precision: What percentage of flagged elements are actually PII (vs. false positives that over-redact clinical content)? Production target: >97%. Current performance: 98.1%. Over-redaction degrades clinical utility; we track it as carefully as under-redaction.
3. Reversibility Rate: What percentage of redacted documents can be fully and accurately de-redacted by authorized users? Production target: 100%. Current performance: 100%. A single irreversible redaction is a data integrity failure — the original clinical record must always be recoverable.
4. Processing Latency: How much time does the redaction pipeline add to the documentation workflow? Production target: <300ms at p99. Current performance: 187ms at p50, 246ms at p99. The redaction layer is fast enough to be invisible to the physician.

What This Means for the Industry

There's a version of the near future where regulatory enforcement catches up with the current state of telehealth AI integration. When it does — and based on OCR's trajectory with tracking technologies, it's a question of when, not if — platforms will fall into two categories: those that can demonstrate, with audit trails and metrics, that patient data never left their infrastructure in identifiable form, and those that cannot.

The second category will face a cascade of consequences: OCR investigations, state attorney general inquiries, class-action litigation from patients whose data was transmitted without adequate safeguards, and the reputational damage that makes patient acquisition and provider recruitment exponentially harder. The telehealth platforms that collapsed in 2023-2024 didn't all fail because of bad clinical models. Some failed because a single compliance failure eroded the trust that their entire business depended on.

At LivoRx, we built the redaction pipeline before we built the note generator. That sequencing wasn't a coincidence. It reflects a conviction that in healthcare technology, the privacy architecture isn't a feature you ship in v2. It's the foundation that determines whether v1 is worth building at all.

The AI model that processes your clinical notes doesn't need to know your patient's name. It doesn't need their date of birth, their insurance ID, or their address. It needs their symptoms, their history, their medications, and their plan. The rest is liability — and the technology to eliminate that liability exists today, operates in under 200 milliseconds, and costs a fraction of what a single breach would.

The only question is whether you build it before regulators require it, or after.

References

• IBM Security, Cost of a Data Breach Report (2024): Healthcare industry average breach cost of $4.88M; 14th consecutive year as the highest-cost sector.
• Sweeney, L., International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems (2002): "k-Anonymity: A Model for Protecting Privacy." Demonstrated that 87% of the U.S. population is uniquely identifiable by zip code, birth date, and sex.
• Stubbs et al., Journal of Biomedical Informatics (2015): The i2b2/UTHealth 2014 de-identification challenge — the standard benchmark for clinical NER-based PII detection with 28,000+ annotated PHI instances.
• The Markup (2023): Investigation of telehealth platforms transmitting session data to Meta and Google via tracking pixels, leading to multiple OCR enforcement actions.
• HHS Office for Civil Rights, Bulletin on Tracking Technologies (December 2022): PHI transmitted to third parties through technology integrations is subject to HIPAA protections.
• PLOS Digital Health (2022): Examination of logging practices across 14 digital health startups — 11 of 14 retained identifiable patient data in application logs without adequate controls.
• UCSF Clinical Informatics Group (2021): Analysis of manual de-identification accuracy decay — from 95.2% initial accuracy to 81.4% within six months without retraining.
• European Data Protection Board, Guidelines on AI and Data Protection (2024): Health data used as AI model input requires pseudonymization before transmission to external processors.
• India Digital Personal Data Protection Act (2023): Sensitive personal data protections and consent requirements for health data processing, penalties up to ₹250 crore.