A clean five-minute recording can make almost any speech system look ready. Production clinical audio is less polite. A patient talks over the clinician. A caregiver answers from across the room. The otoscope wrapper crackles beside the microphone. Wi-Fi drops after the plan begins. Then the transcript assigns a denial to the wrong speaker.
That is the test set.
Takeaways:
- Score clinical errors by consequence and location, not only by total word count.
- Treat diarization as a fallible label stream, not proof of identity.
- Put capture, transient processing, storage, logs, support access, and deletion on one PHI map.
- Test recovery after interruption before tuning punctuation.
Define the output before choosing the recognizer
“Transcribe this visit” is too loose for an engineering requirement. State which artifact the system must produce and what happens next.
A live caption stream, a dictated paragraph, and an ambient encounter transcript have different timing and speaker needs. A note-drafting workflow may not need a verbatim transcript to be the final record at all. It needs enough faithful source material for a bounded draft that an authorized clinician reviews.
Write a contract with six fields:
| Field | Decision |
|---|---|
| Capture | Device, room, channel count, sample format, and start/stop control |
| Speakers | Expected roles, unknown speakers, overlap, interpreter, and caregiver cases |
| Timing | Streaming or batch, partial-result policy, and acceptable delay for the next step |
| Language | Supported languages, code-mixing, specialty terms, acronyms, numbers, and units |
| Output | Transcript, speaker turns, timestamps, confidence metadata, or a downstream draft |
| Failure | User-visible state, retry rule, local fallback, and what cannot be reconstructed |
This contract prevents a fast demo from winning the wrong contest. A vendor can return partial words in a few hundred milliseconds and still fail the visit because speaker turns drift or the recovered stream drops the assessment.
Build the test set from rooms, not podcasts
Clinical audio varies by specialty, room, device, speaker, and task. Assemble consented or synthetic fixtures that cover the intended setting. Do not copy real patient recordings into an engineering folder because the benchmark feels internal.
Include at least these scenes:
- a calm two-person visit with ordinary turn-taking;
- overlapping speech during history-taking;
- a caregiver or interpreter joining midway;
- distant speech while the clinician examines the patient;
- medication names, allergies, abbreviations, dosages, dates, and units;
- explicit negation and later correction;
- a quiet stretch followed by a resumed conversation;
- a device switch or network interruption; and
- a specialty case with locally common terms.
Use a written fixture manifest. Record the audio conditions, intended speakers, required phrases, known ambiguities, and the workflow consequence of each mistake. A benchmark without that context becomes a leaderboard nobody can apply.
Separate four kinds of failure
One overall word-error number compresses unlike problems. NIST’s Speech Recognition Scoring Toolkit includes word-error scoring plus speaker-level and speaker-error reports, which is a useful reminder that the transcript and the speaker map are different evaluation surfaces (NIST SCTK).
For a clinical workflow, keep four ledgers.
Words and clinical meaning
Track omissions, substitutions, insertions, negation, drug names, measurements, laterality, dates, and units. Mark the location in the encounter. “No chest pain” losing its first word deserves a different review than a missing filler word.
Do not invent a universal weighting formula. Have clinical reviewers define categories before they see vendor results, then apply the same rubric to every candidate.
Speakers and provenance
Google’s Speech-to-Text documentation describes diarization as detecting distinct speakers and assigning speaker labels to words in a result (Google Cloud diarization documentation). A label such as speaker_2 is not a verified patient identity. The application still needs a way to handle unknown speakers, overlap, label swaps, and a third person entering the conversation.
Review whether a statement remains attached to the correct role after interruptions. A clinically correct sentence under the wrong speaker can be worse than a visible gap.
Time and interaction
Measure time to first useful partial result, time to a stable segment, and time to the artifact the next step consumes. Record how often prior text changes. A user who edits a partial transcript while it is still being rewritten can create a race between the person and the model.
Latency budgets should follow the workflow. Live captions need a different budget from a note delivered after the encounter. Publish the chosen budget with the result.
Recovery and operations
Pull the cable. Revoke a token. Send an expired credential. Fill a queue. Restart a worker while audio is arriving. Repeat a request after the client times out.
Then answer the questions that matter: what state is durable, what is lost, what can be retried, how a duplicate is detected, which alert fires, and what the user sees. A transcript that works until the first reconnect is a prototype.
Put PHI on a data-flow map
Audio, partial transcripts, final transcripts, derived notes, error payloads, and support artifacts can all contain PHI. Draw them as separate objects. For each object, record the processor, storage location, encryption, access role, logging behavior, retention, deletion, backup, and incident route.
HHS says a cloud provider that creates, receives, maintains, or transmits ePHI for a covered entity or business associate is generally a business associate, even if the provider only stores encrypted data and does not hold the key. An applicable BAA and the regulated organization’s own risk analysis are still required (HHS cloud guidance).
NIST SP 800-66 Revision 2 maps HIPAA Security Rule obligations to cybersecurity resources and risk-management activities (NIST SP 800-66r2). Use it to pressure-test the whole path, not just the recognizer endpoint.
Logs deserve their own row. Developers often remove audio retention and then place raw transcript fragments inside request logs, exception trackers, tracing spans, or support tickets. The shortest retention policy in the main database does not repair those copies.
Use an evidence packet, not a winner slide
The final comparison should let another team repeat the decision. Keep the following together:
- fixture manifest and consent or synthetic-data basis;
- audio capture settings and devices;
- configuration and model identifiers available from the provider;
- reference-transcript rules and adjudication notes;
- error ledgers for words, speakers, timing, and recovery;
- PHI data-flow and contract review;
- runbooks for interruption, retry, and incident response; and
- the clinical owner’s sign-off on the intended use.
Do not crown a winner from one mean. Show distributions and bad cases. Preserve disputed examples with reviewer notes. If the team changes punctuation cleanup, vocabulary hints, or speaker settings, rerun the same fixture set and record the change.
Keep recognition and note review separate
Speech recognition converts audio into words and timing information. A downstream system may structure those words into a note draft. The draft then needs clinical review. Mixing all three steps into one accuracy claim makes it impossible to locate a failure.
Patient Square is an AI clinical platform. Practice Copilot brings the whole practice under one AI copilot: an ambient AI Medical Scribe that hands back a structured SOAP note, ICD-10 suggestions, and a prescription draft minutes after the visit, plus a bundled AI EHR, scheduling, and messaging as you move up the plan. Hospitals get Hospital Copilot.
Patient Square does not publish an ASR accuracy percentage or disclose a speech-recognition vendor in this guide. Its stated product boundary is that audio is processed in memory and discarded once the note is drafted; the clinician reviews the note. That boundary is described on the security page. It does not turn a draft into a diagnostic decision or a signed record.
The useful engineering question is not “Which API transcribes best?” It is narrower: which tested pipeline produces reviewable evidence for this room, recovers from its failures, and keeps its PHI promises? Save the worst fixture. It will teach the team more than the clean demo ever did.