From Visit to Note in Under 60 Seconds: The Automated SOAP Pipeline

The 47-Second Note

At 2:14 PM on a Tuesday, a family medicine provider on the LivoRx platform finishes a 15-minute telehealth visit with a 52-year-old patient presenting with persistent lower back pain and new-onset numbness in the left foot. The provider clicks "End Visit." At 2:15 PM — 47 seconds later — a complete SOAP note appears in her review queue: structured subjective narrative, objective findings from the conversation, a differential assessment with two ICD-10 code suggestions, and a plan that includes an MRI referral, a medication adjustment, and follow-up instructions.

She scans the note, corrects one dosage from the patient's stated "the big white pill" to its actual name (gabapentin 300mg), approves each section, and signs. Total review time: 53 seconds. The entire documentation cycle — from visit end to signed, EHR-integrated clinical note — took less than two minutes. The same note, written manually, would have taken 8-12 minutes of post-visit charting, likely completed between the next two patients or after the clinic day ended.

This isn't a demo. It's the production pipeline we run today. And understanding how it works — what makes it fast, what makes it accurate, and what makes it different from the transcription tools that came before it — requires looking under the hood at each stage of the system.

Why Transcription Alone Isn't the Answer

The first generation of AI-assisted clinical documentation was built on a simple premise: record the visit, transcribe the audio, and give the physician a text version of the conversation. Products in this category — from Dragon Medical to early ambient listening tools — solved a real problem: they eliminated the need for the physician to type or dictate from memory. But they created a new one.

A raw transcript of a 15-minute telehealth visit is approximately 3,000-4,500 words of unstructured text. It contains everything: the patient's narrative (including tangents, repetitions, and self-corrections), the physician's questions, small talk, technical interruptions ("Can you hear me? Your video froze"), and the clinical content buried somewhere in between. Reading through this transcript and extracting the relevant clinical information takes nearly as long as writing the note from scratch.

The fundamental problem is structural. A clinical note isn't a transcript with formatting. It's a transformation of conversational data into a structured clinical document with specific sections (Subjective, Objective, Assessment, Plan), specific terminology conventions, specific levels of detail for different encounter types, and specific compliance requirements for billing and legal defensibility.

"Giving a physician a transcript and asking them to turn it into a SOAP note is like giving an architect a pile of lumber and asking them to find the house inside it. The raw material is there, but the structure has to be built."

A 2022 study in the Journal of the American Medical Informatics Association (JAMIA) analyzed physician interactions with AI transcription tools and found that while transcription reduced keystroke burden, it did not significantly reduce total documentation time. The bottleneck had simply moved — from typing to reading, parsing, and reorganizing. The cognitive work remained.

This insight is what drove the architecture of our pipeline. The goal was never to give physicians a better transcript. It was to give them a better note — structured, section-organized, clinically formatted, and ready for review rather than reconstruction.

The Four-Stage Pipeline

Our automated SOAP pipeline consists of four discrete stages, each with a specific input, output, and quality gate. The stages are designed to be independently testable, independently scalable, and independently upgradable — a microservice-influenced architecture that lets us improve any single stage without destabilizing the others.

Stage 1: Real-Time Audio Processing and Speaker Diarization

The pipeline begins during the telehealth visit itself. As the conversation proceeds, the audio stream is processed through two parallel subsystems: automatic speech recognition (ASR) and speaker diarization.

The ASR component converts speech to text with medical vocabulary optimization. Standard ASR engines struggle with clinical terminology — "metformin" becomes "met for men," "bilateral" becomes "by lateral," "tachycardia" becomes "tacky cardia." Our ASR layer is fine-tuned on medical conversation corpora, achieving a word error rate (WER) of under 5% for clinical terminology compared to 12-18% for general-purpose ASR engines on the same content. This isn't a trivial difference: a 12% WER on medical terms means roughly one in eight clinical words is wrong, which cascades into downstream extraction errors that compound at each pipeline stage.

Speaker diarization — identifying who said what — is equally critical. A SOAP note requires separating the patient's reported symptoms (subjective) from the physician's findings and reasoning (assessment and plan). Without reliable diarization, the system can't distinguish "I've been having chest pain" (patient report → subjective section) from "The chest pain is likely musculoskeletal" (physician assessment → assessment section). Our diarization layer achieves 97.3% speaker attribution accuracy in two-party telehealth conversations, validated against manually annotated test sets.

The output of Stage 1 is a time-stamped, speaker-attributed transcript with medical vocabulary correction — not a clinical note, but the structured raw material from which one can be built.

Stage 2: Clinical Entity Extraction

This is the stage that separates a SOAP pipeline from a transcription service. The speaker-attributed transcript enters a clinical NLP engine that identifies and extracts structured clinical entities — discrete, categorized pieces of clinical information that map directly to the components of a SOAP note.

The extraction engine identifies over 40 entity types organized into SOAP-aligned categories:

• Subjective entities: Chief complaint, HPI elements (onset, location, duration, character, severity, timing, modifying factors, associated symptoms), past medical history mentions, medication history, allergy reports, social history, family history, and review of systems responses.
• Objective entities: Vital signs mentioned, physical exam findings described verbally (common in telehealth where visual assessment replaces hands-on examination), screening tool scores, and any lab or imaging results discussed during the visit.
• Assessment entities: Differential diagnoses mentioned, clinical reasoning statements, risk stratification language, and severity qualifiers.
• Plan entities: Medications prescribed or adjusted (with dosage, frequency, route, duration), diagnostic tests ordered, referrals made, patient education delivered, follow-up timing, and return precaution instructions.

Each extracted entity carries metadata: the source utterance (the exact words from the transcript that generated it), the speaker (patient or provider), a confidence score (0-1) reflecting extraction certainty, and a timestamp linking it to the conversation timeline.

The confidence scoring is essential. When a patient says "I take that blood pressure pill, the pink one, I think it's 10 milligrams," the system extracts a probable antihypertensive medication at a probable 10mg dose — but with a lower confidence score than a clear statement like "I take amlodipine 10 milligrams once daily." Low-confidence entities are flagged in the final note for physician attention, transforming the review process from "read everything" to "verify the flagged items."

Published benchmarks for clinical named entity recognition (NER) on conversational medical text show F1 scores ranging from 0.82 to 0.91, depending on entity type (medication entities score highest; nuanced assessment entities score lowest). Our extraction layer operates at 94.7% weighted accuracy across all entity categories, validated against physician-annotated ground truth from our own encounter data.

Stage 3: SOAP Note Assembly and Clinical Formatting

With structured clinical entities extracted and confidence-scored, the assembly stage constructs the actual SOAP note. This is not a template-fill operation — it involves four critical transformations that convert raw clinical data into a physician-ready document.

Language normalization. Patient language is converted to clinical terminology while preserving meaning. "My stomach's been killing me for three days, especially after I eat" becomes "Patient reports severe epigastric pain, onset 3 days prior to presentation, exacerbated by oral intake." This requires understanding clinical conventions for how symptoms are described in documentation, including the standard HPI element sequence (location, quality, severity, duration, timing, context, modifying factors, associated signs/symptoms) defined in CMS E/M documentation guidelines.

Section organization. Extracted entities are routed to their appropriate SOAP section based on clinical attribution rules. A patient's statement about taking medication goes to the subjective medication history, not the plan. A physician's statement about ordering a lab goes to the plan, not the objective findings. This routing is rule-based with ML fallback — structured rules handle over 90% of cases, with a trained classifier handling ambiguous attributions where speaker role and clinical context must be weighed together.

Detail calibration. Different encounter types require different levels of documentation detail. A follow-up visit for stable hypertension management requires less narrative than a new patient evaluation for complex multi-system complaints. The assembly engine adjusts documentation density based on the encounter type, visit duration, and number of clinical entities extracted — producing notes that match the documentation expectations for that visit category without over-documenting routine encounters or under-documenting complex ones.

Compliance scaffolding. The assembled note includes structural elements required for compliant billing documentation: time-based documentation markers (for time-based E/M coding under the 2021 CMS guidelines), medical decision-making complexity indicators, and telehealth-specific elements including the originating site, distant site, and communication modality. Under the 2024 CMS physician fee schedule, telehealth visits require specific place-of-service codes and modifiers — our assembly engine embeds these into the note structure automatically, eliminating a common source of billing denials.

The output is a complete SOAP note draft with per-section confidence scores, source-utterance links for verification, and suggested ICD-10 codes ranked by clinical relevance.

Stage 4: PII Redaction and Security Boundary

Before the assembled note reaches any external AI model for language refinement or quality analysis, it passes through our automated PII redaction layer. This stage is architecturally positioned as a security boundary — a hard line in the pipeline beyond which no patient-identifiable information flows to external services.

The redaction engine identifies and tokenizes 18 categories of protected health information aligned with HIPAA's Safe Harbor de-identification method: names, geographic data below state level, dates (except year) related to the individual, phone numbers, fax numbers, email addresses, Social Security numbers, medical record numbers, health plan beneficiary numbers, account numbers, certificate/license numbers, vehicle identifiers, device identifiers, URLs, IP addresses, biometric identifiers, full-face photographs, and any other unique identifying number or code.

Tokenization — not deletion — is critical. Each PII element is replaced with a typed placeholder (e.g., [PATIENT_NAME_1], [DOB_1]) that preserves the document's structure and readability for downstream processing. After external processing is complete, the tokens are re-hydrated with the original values inside our secure environment. The external model never sees — and therefore cannot log, cache, or learn from — any patient-identifiable information.

This architectural decision adds approximately 200 milliseconds of latency to the pipeline. It's the best 200 milliseconds we spend. (For a deeper exploration of why this matters, see our companion piece: The Urgent Case for Automated PII Redaction in Clinical Records.)

What Makes It Fast: Architecture Decisions That Shave Seconds

A 47-second median draft time isn't achieved by using a faster model. It's achieved by architecture decisions that eliminate unnecessary work and parallelize what remains. Three design choices account for most of the speed advantage:

Streaming, Not Batch Processing

Most ambient documentation tools wait until the visit ends, then process the complete audio file. This batch approach means the pipeline doesn't start until the provider clicks "End Visit," and the full ASR + extraction + assembly pipeline runs sequentially on the complete recording.

Our pipeline is streaming. ASR and speaker diarization run continuously during the visit. Clinical entity extraction operates on rolling windows of transcript text, extracting entities as the conversation unfolds. By the time the visit ends, approximately 85-90% of the clinical entities have already been extracted and are waiting in the assembly queue. The post-visit pipeline only needs to process the final minutes of conversation, run a consistency check across all extracted entities, and assemble the note. The result: a 15-minute visit produces a draft in under 50 seconds, not 3-4 minutes.

Extraction Before Generation

Many competing approaches send the raw transcript to a large language model and ask it to "generate a SOAP note." This single-step approach is conceptually simple but computationally expensive — the LLM must simultaneously parse the transcript, identify clinical content, organize it into sections, normalize the language, and format the output. It's asking one model to do four different jobs in one inference pass.

Our pipeline separates extraction (a specialized NER/NLU task) from generation (a structured text assembly task). The extraction model is small, fast, and domain-specific — it runs in under 3 seconds on a typical visit transcript. The assembly step operates on structured data (extracted entities with metadata), not on raw text, which means it's deterministic for most of the note and only uses generative inference for language normalization. This separation reduces total compute by roughly 60% compared to end-to-end generation, and — critically — makes the system's behavior more predictable and auditable.

Incremental Confidence Accumulation

Because extraction runs continuously during the visit, confidence scores accumulate incrementally. A chief complaint mentioned once early in the visit might start with a confidence score of 0.7. If the patient reiterates it later ("So like I said, the main reason I'm here is this back pain"), the confidence score increases to 0.92. By visit end, most clinical entities have been mentioned or confirmed multiple times, and their confidence scores reflect this accumulation.

This means the physician review interface can present most sections as high-confidence (green) with only a few items flagged for verification (amber) — making the review faster because the physician knows exactly where to focus. In our pilot data, an average note contains 2.1 amber-flagged items out of 15-20 total clinical entities, which explains why the median review time is under 60 seconds.

Accuracy at Scale: Where the Pipeline Can — and Can't — Be Trusted

Speed without accuracy is worse than useless in clinical documentation — it's dangerous. A note that's generated in 30 seconds but contains a wrong medication dosage creates more risk than a note that takes 10 minutes to write correctly. Being honest about accuracy boundaries is essential for building physician trust.

Where the Pipeline Excels

The pipeline performs best on structured, unambiguous clinical content — the categories that make up the majority of most clinical notes:

• Chief complaint identification: 98.4% accuracy. Patients almost always state their reason for visit clearly and early.
• Medication names and dosages: 96.1% accuracy when stated explicitly by patient or provider. Drops to 78% when patients use informal descriptions ("the little blue pill").
• Objective findings from verbal description: 95.7% accuracy. Telehealth objective sections are narrower than in-person visits, which paradoxically makes them easier to extract reliably.
• Plan items (labs, referrals, follow-up): 97.2% accuracy. Physicians tend to state plan items clearly and in list-like fashion.

Where It Struggles

The pipeline is weakest in areas that require clinical inference rather than extraction:

• Assessment nuance: When a physician thinks aloud — "This could be radiculopathy, but I'm also wondering about peripheral neuropathy given the diabetes history" — the extraction engine captures both diagnoses but sometimes misattributes the physician's uncertainty level, presenting a differential as a definitive assessment.
• Implicit clinical reasoning: If a physician orders a TSH without explicitly saying why, the system can't reliably infer that it's because the patient mentioned fatigue and weight gain. The assessment may lack the connecting rationale that a manually written note would include.
• Cross-referencing patient history: The pipeline processes the current visit conversation, but it doesn't have full context of the patient's historical record. If the patient says "same thing as last time," the system flags this as low-confidence rather than looking up what "last time" referred to.

These limitations are why the physician review step isn't optional. The pipeline handles the mechanical, high-volume work of structuring clinical information. The physician handles the interpretive, contextual work that requires medical training and patient knowledge. (For a deeper exploration of how we design the review interface, see: Designing a Doctor-in-the-Loop Review System for AI-Generated Notes.)

Compliance by Design, Not by Afterthought

Clinical documentation serves two masters simultaneously: clinical continuity and regulatory compliance. A note that's clinically excellent but doesn't meet billing documentation requirements will be denied. A note that meets billing requirements but is clinically vague fails its primary purpose. The pipeline has to satisfy both, and it has to do so automatically.

Under the 2021 CMS E/M documentation guidelines (which remain in effect through 2026), office and outpatient visits are coded based on either medical decision-making (MDM) complexity or total time. For MDM-based coding, the note must document the number and complexity of problems addressed, the amount and complexity of data reviewed, and the risk of complications or morbidity. For time-based coding, the note must document the total time spent on the encounter date, including both face-to-face and non-face-to-face activities.

Our assembly engine maps extracted clinical entities to MDM complexity elements automatically. Multiple active chronic conditions with medication management maps to moderate MDM complexity. A new problem requiring additional workup with a decision to order imaging maps to a higher complexity level. The engine doesn't just generate a note — it generates a note with embedded documentation that supports the appropriate E/M level, reducing the risk of undercoding (lost revenue) or overcoding (compliance risk).

For telehealth-specific compliance, the note automatically includes required elements that providers frequently forget to document manually: the telehealth modality (synchronous audio-video), the patient's location at time of service, the provider's location, consent for telehealth (when required by state law), and the appropriate place-of-service code and modifier. In multi-state practices, the pipeline adjusts these elements based on the patient's jurisdiction — a capability that eliminates one of the most common sources of telehealth billing denials.

What We Learned Building This: Five Engineering Lessons

Building a production clinical NLP pipeline taught us lessons that aren't in the academic literature. These are the insights that came from operating the system at scale with real physicians, real patients, and real regulatory requirements.

1. Latency tolerance is non-linear. Physicians accept a 30-second wait after clicking "End Visit." They accept a 60-second wait. At 90 seconds, they start opening other tabs. At 120 seconds, they begin writing the note manually. The window between "acceptable" and "abandoned" is shockingly narrow — roughly 40 seconds. Every architectural decision we made was filtered through this constraint.
2. Physician trust is earned per-section, not per-note. A physician who finds one error in the Assessment section doesn't lose trust in the entire note — they lose trust in the Assessment section specifically. Per-section confidence scoring and per-section approval leverage this psychology: the physician can trust the Subjective section (which is usually accurate because it's extracting patient statements) while scrutinizing the Assessment (which requires more clinical inference).
3. False confidence is worse than low confidence. A note section flagged as amber (low confidence) that turns out to be correct builds trust. A note section marked as green (high confidence) that contains an error destroys it. We deliberately calibrate our confidence thresholds to over-flag rather than under-flag — accepting slightly more physician review work in exchange for never presenting a wrong answer as a confident one.
4. Specialty-specific performance varies dramatically. Our pipeline performs best for primary care, urgent care, and general internal medicine — encounter types with relatively structured conversations and well-defined documentation patterns. Psychiatry visits (longer, more conversational, less structured) and dermatology visits (heavily visual, less verbally descriptive) required specialty-specific extraction models. One-size-fits-all clinical NLP is a myth.
5. The feedback loop is the product. Every physician edit is a training signal. Every correction teaches the system what it got wrong and how to do it better. But this only works if the edit interface is designed to capture structured corrections (not just free-text overwrites) and if the correction pipeline feeds back into model improvement on a regular cadence. The pipeline that ships is never the best pipeline — the pipeline that learns from its mistakes is.

The Future: From Post-Visit Pipeline to Ambient Co-Pilot

The pipeline described in this post is fundamentally a post-processing system. The visit happens, the audio is processed, the note is generated, the physician reviews. It's fast — under 60 seconds — but it's still sequential.

The next evolution is a shift from post-visit pipeline to ambient co-pilot: a system that doesn't just document the visit after it ends, but actively assists during the visit itself. Real-time clinical decision support surfaced during the conversation — differential diagnosis suggestions when symptoms are described, drug interaction warnings when medication changes are discussed, evidence-based screening recommendations triggered by risk factors mentioned in passing.

This requires solving three hard problems simultaneously: sub-second inference latency (you can't interrupt a conversation with a suggestion that arrives 10 seconds late), clinical knowledge grounding (suggestions must be evidence-based and verifiable, not hallucinated), and unobtrusive interface design (the suggestions must be visible to the physician without being visible to the patient or interrupting the conversation flow).

We're not there yet. But every visit that runs through our current pipeline generates training data that brings the ambient co-pilot closer. The structured entities we extract today become the training corpus for the real-time clinical intelligence of tomorrow. The 47-second note isn't the destination — it's the foundation.

References

• Arndt et al., Annals of Family Medicine (2017): "Tethered to the EHR: Primary Care Physician Workload Assessment." Physicians spent 5.9 hours per clinic day in EHR tasks, nearly 2 hours per hour of direct patient time.
• Rotenstein et al., JAMA Network Open (2023): Median 36.2 minutes total EHR time and 6.2 minutes of pajama time per visit across 307 primary care physicians.
• Quiroz et al., Journal of the American Medical Informatics Association (2022): Analysis of physician interactions with AI transcription tools showing keystroke reduction without proportional documentation time savings.
• CMS 2021 E/M Documentation Guidelines: Updated evaluation and management coding framework based on medical decision-making complexity or total time.
• CMS 2024 Physician Fee Schedule Final Rule: Telehealth place-of-service codes, modifier requirements, and documentation standards for synchronous audio-video visits.
• Chiu et al., IEEE/ACM Transactions on Audio, Speech, and Language Processing (2018): Medical speech recognition error rates and domain-specific ASR optimization techniques.
• Johnson et al., NPJ Digital Medicine (2023): Clinical NER benchmarks on conversational medical text, F1 scores by entity type across multiple extraction architectures.