Designing a Doctor-in-the-Loop Review System for AI-Generated Notes

The Trust Problem with Fully Automated Clinical Notes

In 2023, a study published in JAMA Internal Medicine by Ayers et al. made headlines across healthcare media. The researchers compared responses to patient questions from licensed physicians and from ChatGPT. The AI-generated responses were rated significantly higher in both quality and empathy by a panel of healthcare professionals — 78.6% of evaluators preferred the chatbot's answers. The finding was striking, and it fueled a wave of enthusiasm about AI replacing clinical documentation entirely.

But here's what the headlines missed: answering a patient's general health question and generating a clinical note that becomes part of a legal medical record are fundamentally different tasks. One is advisory. The other is evidentiary. A clinical note isn't just a summary of what happened during a visit — it's the document that drives billing, informs downstream clinical decisions, triggers referrals, supports malpractice defense, and follows the patient across every future encounter. The stakes are not comparable.

The core problem with fully automated clinical notes is not accuracy in the average case. Modern large language models, fine-tuned on medical corpora, can produce impressively structured SOAP notes that look clinically sound at first glance. The problem is what happens in the non-average cases — the edge cases, the ambiguities, the hallucinations.

The FDA's 2024 discussion paper on AI and machine learning in clinical decision support drew a critical distinction between "locked" algorithms (which produce the same output for the same input every time) and "continuously learning" algorithms (which adapt over time). Most LLM-based documentation tools fall into the latter category, which the FDA flagged as requiring more rigorous oversight frameworks. The reason is straightforward: if the model's behavior changes between version deployments, yesterday's validation data doesn't guarantee today's output quality.

And then there's the hallucination problem. A 2023 analysis published in Nature explored the tendency of large language models to generate plausible-sounding but factually incorrect medical information. In clinical documentation, this manifests in specific and dangerous ways: fabricated medication dosages, invented allergy histories, hallucinated lab values, or clinical assessments that sound authoritative but have no basis in the actual patient encounter.

"A note that sounds clinically perfect but contains a fabricated allergy or an invented lab value is more dangerous than a note with a typo — because no one questions it."

This is the fundamental asymmetry. A bad note that looks bad gets caught. A bad note that looks good propagates through the medical record indefinitely, compounding its potential for harm with every downstream decision that relies on it.

Why Human-in-the-Loop Isn't Optional — It's a Regulatory Requirement

The argument for physician oversight of AI-generated documentation isn't just a clinical best practice — it's embedded in the regulatory infrastructure of healthcare in the United States and, increasingly, globally.

Start with billing. The Centers for Medicare & Medicaid Services (CMS) requires that clinical documentation used for evaluation and management (E/M) billing reflect the medical decision-making or time-based criteria of the treating physician. Under the 2024 E/M documentation guidelines, the physician must attest to the accuracy and completeness of the clinical note. An AI-drafted note that a physician hasn't meaningfully reviewed doesn't satisfy this requirement — and submitting such notes for reimbursement creates false claims risk under the False Claims Act.

The 21st Century Cures Act added another dimension. The ONC's information blocking rules require that clinical notes be made available to patients through their portals — the Open Notes mandate. This means AI-generated errors don't stay buried in the EHR. They appear in the patient's portal, where the patient reads them, shares them with family members, and may act on them. A hallucinated diagnosis or an incorrect medication listed in an Open Note isn't just a documentation error — it's a patient communication failure with real clinical consequences.

The American Medical Association's 2024 policy on augmented intelligence (Policy H-480.940) is explicit: "Augmented intelligence should be designed to enhance physician decision-making, not replace it." The AMA further states that AI tools in clinical settings should preserve the physician's authority over clinical decisions and maintain the integrity of the patient-physician relationship. Automated documentation that bypasses physician review violates the spirit and increasingly the letter of these guidelines.

And then there's malpractice liability — the dimension that should keep every health system CTO awake at night. Under current tort law in every U.S. jurisdiction, the physician who signs a clinical note is liable for its contents. It doesn't matter if an AI wrote it, if a scribe drafted it, or if a template auto-populated it. The signing physician is the legally responsible author. AI vendor terms of service universally include disclaimers that explicitly reject clinical liability for model outputs. The physician is the last line of defense — and the only one the legal system holds accountable.

Anatomy of a Well-Designed Review System

Acknowledging the need for physician oversight is the easy part. The hard part — and the part that most platforms get wrong — is designing a review system that physicians will actually use effectively. A poorly designed review interface is worse than no AI at all, because it creates the illusion of oversight without the substance.

At LivoRx, we've spent over a year iterating on the review layer of our automated SOAP pipeline. What we've learned is that the review system is not a feature — it's the product. Everything else (the NLP, the draft generation, the coding suggestions) is infrastructure. The review experience is where clinical trust is either built or destroyed.

Here's how a well-designed 5-stage pipeline works:

Stage 1: Structured Clinical Capture

During the telehealth visit, the conversation is processed through an ambient clinical documentation pipeline in real-time. But unlike simple transcription services that produce a verbatim text dump, a structured capture system extracts clinical entities — discrete, categorized pieces of clinical information that map directly to the components of a clinical note.

This means identifying and extracting: the chief complaint, the elements of the history of present illness (onset, location, duration, character, severity, modifying factors, associated symptoms), review of systems responses, relevant past medical history mentioned during the conversation, the physician's assessment, and the plan including medications, labs, referrals, and follow-up instructions.

The distinction between transcription and structured extraction is critical. A transcript gives you a wall of text that requires a physician to read, parse, and reorganize. Structured extraction gives you categorized clinical data that can be assembled into a note format automatically. Products like Nuance DAX and Abridge have demonstrated the feasibility of ambient clinical capture — but the gap between "captured audio" and "structured clinical entities ready for note assembly" is where most systems underperform. Raw transcripts don't make good notes. Structured entity extraction does.

Stage 2: AI Draft Generation with Confidence Scoring

Once structured clinical entities are extracted, they're assembled into a complete SOAP note draft. But the assembly isn't a simple template fill — it involves clinical language normalization (converting conversational phrasing like "my stomach's been killing me for three days" into clinical terminology like "Patient reports severe epigastric pain, onset 3 days prior to presentation"), section organization, and appropriate level of detail for the encounter type.

The critical innovation in this stage is confidence scoring. Each section of the generated note carries a confidence score between 0 and 1, derived from the clarity, completeness, and unambiguity of the source conversation for that section. A chief complaint captured clearly ("I'm here because of chest pain that started yesterday") scores high. A plan section reconstructed from a conversation where the physician's instructions were partially inaudible or ambiguous scores lower.

Low-confidence sections are flagged visually in the review interface — typically with amber highlighting — so the physician knows exactly where to focus attention. This transforms the review from "read everything carefully" to "verify the flagged sections and spot-check the rest" — a fundamentally more efficient cognitive task.

Medications and dosages mentioned in the note are cross-referenced against the patient's existing medication list and known drug formularies. ICD-10 diagnostic codes are suggested with individual confidence levels, allowing the physician to accept, modify, or reject each suggestion independently.

Stage 3: PII Redaction Before External Processing

Before any clinical text reaches an external AI model for processing — whether for language refinement, coding assistance, or quality analysis — automated PII redaction removes all patient-identifiable information. Patient names, dates of birth, Social Security numbers, medical record numbers, addresses, and phone numbers are tokenized and replaced with anonymous placeholders. Only de-identified clinical content enters the processing pipeline.

This architectural decision is non-negotiable. HIPAA's minimum necessary standard requires that only the minimum amount of protected health information needed for a specific purpose be disclosed. Sending identifiable patient data to external AI services — even with a Business Associate Agreement in place — expands the attack surface and regulatory exposure unnecessarily. De-identification at the pipeline level eliminates this risk category entirely.

The redaction layer operates before the AI processing step, not after it. This is a subtle but important architectural distinction: the AI model never sees the patient's identity, which means there's no possibility of PII appearing in model logs, training data, or cached outputs. (For a deeper exploration of this architecture, see our companion piece: The Urgent Case for Automated PII Redaction in Clinical Records.)

Stage 4: The Physician Review Interface

This is the stage that separates systems that physicians trust from systems they abandon. The review interface is where the human-in-the-loop promise is either fulfilled or broken. Get it wrong, and physicians will rubber-stamp notes without reading them (creating liability) or rewrite them from scratch (negating the automation benefit). Get it right, and you transform a 12-minute writing task into a 90-second editing task with higher output quality.

A well-designed review interface includes five key elements:

Side-by-side verification. The AI-drafted note appears on one side of the screen. Key conversation excerpts — the actual words the patient and physician spoke — appear on the other side, aligned to the corresponding note section. This allows the physician to verify that the note accurately reflects the encounter without replaying the entire visit in their memory.

Section-by-section approval. Rather than approving or rejecting the entire note as a unit, physicians review and approve each SOAP section independently. The subjective section might be approved with no changes while the assessment section requires a modification. This granularity makes the review process faster and the audit trail more precise.

Diff highlighting. Any changes the physician makes to the AI draft are tracked and displayed as diffs — additions in green, deletions in red. This creates an automatic audit trail showing exactly what the AI generated and exactly what the physician changed. For compliance and malpractice defense, this record is invaluable.

Smart defaults and personalization. Over time, the system learns each physician's documentation preferences — their preferred phrasing patterns, their typical level of detail for different encounter types, their common corrections. This isn't model retraining; it's preference learning that adjusts formatting and language to match the physician's style, reducing the number of edits needed per note.

Time-budgeted design. The entire review interface is designed around a 1-3 minute interaction. Every element that doesn't directly contribute to the review decision is removed. No unnecessary navigation, no extra clicks, no modal dialogs that interrupt the flow. The physician opens the note, scans the flagged sections, makes any needed corrections, and signs — all without leaving the review screen.

"The goal isn't to make physicians rubber-stamp AI output. It's to transform their role from 'author writing under time pressure' to 'editor reviewing a structured draft' — a cognitively easier task that produces better outcomes."

Stage 5: Attestation, Audit Trail, and EHR Integration

When the physician completes their review and signs the note, the attestation carries the same legal and compliance weight as a manually authored note. The physician's electronic signature confirms that they have reviewed the content, verified its accuracy, and accept clinical responsibility for the documented encounter.

Behind the signature, the system maintains a comprehensive audit trail. Every action is logged with timestamps: when the draft was generated, when the physician opened the review, which sections were modified, what specific text was changed, and when the final signature was applied. This audit trail serves three critical purposes: CMS documentation compliance, malpractice defense (demonstrating that the physician actively reviewed and verified the note), and system quality monitoring (identifying patterns of physician corrections that indicate areas where the AI model needs improvement).

The signed, finalized note is then transmitted to the EHR via FHIR R4/R5 APIs — specifically using the DocumentReference and DiagnosticReport resources for structured clinical notes, and the Encounter resource for visit metadata. This integration ensures the note appears in the patient's chart alongside all other clinical documentation, without requiring manual re-entry or copy-paste workflows that introduce their own error risks.

Three Failure Modes That Break Doctor-in-the-Loop Systems

Designing a review system is necessary but not sufficient. The harder challenge is designing against the failure modes that erode the system's effectiveness over time. In our experience — and consistent with decades of human-factors research in aviation, nuclear power, and other safety-critical domains — there are three primary failure modes that can undermine even a well-designed human-in-the-loop system.

Failure Mode 1: Automation Bias

Automation bias is the well-documented tendency of humans to over-rely on automated system outputs, even when those outputs conflict with their own observations or training. Parasuraman and Manzey's foundational 2010 paper in Human Factors demonstrated that automation complacency increases as system reliability increases — paradoxically, the better the AI performs on average, the more likely users are to miss its errors.

In clinical documentation, automation bias manifests as "checkbox attestation" — physicians clicking "approve" on each section without genuinely reading the content, because the AI has been right so many times before that active verification feels unnecessary. This is the most insidious failure mode because it's invisible: the audit trail shows a review occurred, but the review was not cognitively meaningful.

Mitigation strategies must be structural, not motivational. Telling physicians to "pay attention" doesn't work — the same human-factors literature is clear on this. Effective countermeasures include: randomized verification prompts that ask the physician to confirm a specific clinical detail ("Is the dosage of metformin 500mg or 1000mg?"), mandatory edit requirements for sections flagged below the confidence threshold, and periodic calibration audits where a clinical quality team reviews a random sample of approved notes for accuracy.

Failure Mode 2: Review Fatigue

If the AI draft quality is poor — if every note requires substantial editing — then the "review" model quietly degrades into a "rewriting" model. The physician spends just as long fixing the AI's output as they would have spent writing the note from scratch, but with the added cognitive overhead of first reading and understanding someone else's draft before correcting it. This is worse than no automation at all, and it's the fastest way to destroy physician trust in the system.

The threshold is narrower than most teams expect. Internal user research across multiple health systems has consistently shown that when physicians need to edit more than 20-25% of a generated note, they perceive the tool as "creating more work, not less" — regardless of whether the total time is actually shorter. Perception drives adoption, and perceived inefficiency drives abandonment.

Mitigation requires continuous accuracy monitoring with per-physician granularity. Different physicians have different documentation styles, specialty-specific terminology, and levels of detail preference. A system that performs well for a family medicine provider may underperform for a psychiatrist whose notes require more nuanced language. Per-physician accuracy dashboards, automated regression detection (alerting when a physician's edit rate increases significantly), and specialty-specific model tuning are all essential.

Failure Mode 3: Workflow Friction

The third failure mode is the most straightforward but the most commonly overlooked: if the review interface itself is slow, clunky, or difficult to access, physicians will find workarounds. They'll write notes manually and skip the AI tool entirely, or they'll approve without reviewing just to clear the queue.

This is particularly acute in telehealth, where physicians often review notes between back-to-back virtual visits — sometimes from a tablet, sometimes from a phone during a break. The review interface must be fully responsive and mobile-optimized, with touch-friendly approval controls and text editing that works on small screens.

Effective friction reduction includes: progressive disclosure UI (showing the summary view first, with section details available on demand), keyboard shortcuts for power users on desktop, voice-activated approval for low-risk notes reviewed in mobile contexts, and sub-2-second page load times regardless of note complexity. Every additional second of latency in the review interface increases the likelihood of abandonment or rubber-stamping.

Measuring Success: The Metrics That Matter

A doctor-in-the-loop system that can't prove its value quantitatively will eventually lose executive sponsorship, physician buy-in, or both. But not all metrics are equally informative. Vanity metrics — like "number of notes generated" or "total hours of audio processed" — measure activity, not impact. The metrics that matter are the ones that demonstrate clinical safety, physician efficiency, and documentation quality simultaneously.

We track five primary metrics:

1. Review Time: Median time from draft presentation to physician sign-off. This measures the efficiency of the review interface and the quality of the AI draft. Target: under 90 seconds. If median review time exceeds 3 minutes, the system is adding friction, not removing it.
2. Edit Rate: Percentage of AI-generated content modified by the physician during review. This is the most direct measure of AI accuracy from the physician's perspective. Target: under 15%. Rates above 25% indicate the AI is generating work, not saving it.
3. Override Rate: Percentage of AI-suggested ICD-10 diagnostic codes changed by the physician. Coding accuracy directly impacts revenue cycle performance and compliance risk. Target: under 10%.
4. Pajama Time Reduction: After-hours EHR documentation time compared to the pre-automation baseline. This is the metric physicians care about most viscerally. Any amount of pajama time that persists after automation deployment represents a failure to fully solve the problem.
5. Note Quality Score: Clinical audit scores for completeness, accuracy, and compliance as measured by independent chart review. This metric guards against the risk that faster documentation comes at the cost of clinical quality. Target: equal to or higher than pre-automation baselines.

The Road Ahead: From Review to Collaboration

The doctor-in-the-loop model as described above is fundamentally review-centric: the visit happens, the AI drafts, the physician reviews. It's a sequential workflow — the AI's contribution comes after the clinical encounter is complete. This model represents a massive improvement over manual documentation, but it's not the end state.

The next evolution is collaborative: AI and physician working simultaneously during the visit itself. Instead of waiting until after the encounter to generate a note, the system provides real-time clinical intelligence during the conversation — surfacing relevant patient history when a new symptom is mentioned, suggesting differential diagnoses based on the symptom pattern described, flagging potential drug interactions when a medication change is discussed, or recommending evidence-based screening tests ("Based on the symptoms described, consider ordering TSH and free T4 levels").

In this model, the physician isn't reviewing AI output after the fact — they're interacting with AI suggestions in real time, accepting or dismissing them as part of the clinical reasoning process. The documentation becomes a byproduct of this collaborative interaction, not a separate post-visit task.

This shift requires significant advances in several areas: real-time natural language understanding with sub-second latency, deeper clinical knowledge graphs that can reason about symptom-disease-treatment relationships, and user interfaces that surface suggestions without interrupting the patient conversation. The physician must remain the decision-maker — the AI is a real-time clinical assistant, not an autonomous agent.

We're already building toward this at LivoRx. Our current pipeline captures structured clinical data in real time; the next step is feeding that structured data into a clinical decision support layer that operates during the visit rather than after it. The regulatory framework for this evolution is still developing — the FDA's evolving guidance on clinical decision support will shape what's permissible — but the direction is clear.

The platforms that get the human-AI collaboration model right will define the next era of clinical care. They'll attract the best physicians — the ones who want technology that amplifies their clinical capabilities rather than burying them in documentation. And the platforms that remove the physician from the loop, prioritizing speed over safety, will face the inevitable consequences: regulatory action, malpractice exposure, and the erosion of patient trust that no technology can rebuild.

The physician must remain in the loop — not as a bottleneck, but as the guarantor of clinical quality. Our job as technologists is to make that loop as efficient, as intuitive, and as clinically valuable as possible.

References

• Ayers et al., JAMA Internal Medicine (2023): "Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Social Media Forum." ChatGPT responses rated higher quality and more empathetic than physician responses in 78.6% of evaluations.
• Parasuraman & Manzey, Human Factors (2010): "Complacency and Bias in Human Use of Automation: An Attentional Integration." Foundational study on automation complacency in safety-critical systems.
• American Medical Association, Policy H-480.940 on Augmented Intelligence (2024): AI should enhance physician decision-making, not replace it; physician authority over clinical decisions must be preserved.
• FDA Discussion Paper: "Artificial Intelligence and Machine Learning in Software as a Medical Device" (2024 update): Distinguishes locked vs. continuously learning algorithms with different oversight requirements.
• 21st Century Cures Act, Section 4003 — Conditions of Certification: Establishes requirements for health IT certification including information blocking provisions and Open Notes mandate.
• CMS Documentation Requirements for E/M Services (2024 update): Physician attestation requirements for clinical documentation used in evaluation and management billing.
• Rotenstein et al., JAMA Network Open (2023): Median 36.2 minutes total EHR time and 6.2 minutes of pajama time per primary care visit across 307 physicians.