An AI-scribe audit should compare the evidence from the encounter with the signed note, classify each meaningful difference and preserve reviewer disagreement. It should also inspect workflow failures such as missing drafts or wrong destinations. One score hides too much. A useful audit tells clinical leadership what failed, how often and who decided severity.
Key takeaways
- Audit signed notes and abandoned drafts, not vendor-selected examples.
- Stratify by clinician, visit type and operating condition.
- Separate clinically meaningful errors from style preferences.
- Preserve two first-pass reviews before adjudication.
Pull 20 notes, but call it a screen
Start with 20 consecutive eligible notes across the clinicians and visit types in scope. That number is deliberately practical. It is not a statistical threshold and it cannot establish a universal accuracy rate.
Do not take the 20 easiest notes. Build a small stratification grid before sampling:
| Dimension | Example strata | Why keep it visible? |
|---|---|---|
| Clinician | Each pilot clinician | Editing and dictation habits differ |
| Visit type | New, follow-up, procedure or local categories | Content and structure differ |
| Operating condition | Quiet, caregiver present, remote or other approved context | The source encounter differs |
| Draft outcome | Signed, heavily edited, abandoned, missing | Success-only samples hide workflow failure |
| Timing | Morning, late clinic, after-hours completion | Process conditions can change |
Use categories that fit the approved use case. Remove a stratum when it does not apply. Add one when the pilot’s risk assessment names it.
The sample record should show the eligible population, selection method, exclusions and denominator. “We reviewed 20 notes” is not interpretable without those four items.
Keep prospective review separate from the audit
Every AI-generated clinical note is a draft. The APA’s AI-scribe evaluation guidance tells clinicians to review and edit notes before signing and states that clinicians remain responsible for clinical decisions and documentation. A retrospective audit checks how the workflow performed. It does not replace the signer.
Patient Square is an AI clinical platform. Practice Copilot brings the whole practice under one AI copilot — an ambient AI Medical Scribe that hands back a structured SOAP note, ICD-10 suggestions, and a prescription draft minutes after the visit, plus a bundled AI EHR, scheduling, and messaging as you move up the plan. Hospitals get Hospital Copilot.
The current US plan scope does not make those outputs final. ICD-10 items are suggestions, and a prescription output is a draft. The clinician decides what belongs in the signed record.
Use an error taxonomy reviewers can apply
Give reviewers definitions and examples before they see the sample. Keep clinical meaning separate from prose polish.
| Class | Audit question | Record in the log |
|---|---|---|
| Omission | Is material supported information absent? | Missing concept, note section and potential consequence |
| Unsupported addition | Does the note state something not supported by the approved source? | Exact statement and conflicting evidence |
| Attribution | Is a statement assigned to the wrong person, time or condition? | Incorrect and correct attribution |
| Contradiction | Does one part conflict with another or with the source? | Both statements and reviewer interpretation |
| Specificity loss | Was a documented detail generalized in a way that changes meaning? | Source detail and signed wording |
| Structure or style | Is content hard to find without changing clinical meaning? | Local preference or usability issue |
| Workflow failure | Was the draft missing, duplicated, delayed or sent to the wrong destination? | Event, owner, response and outcome |
Severity belongs to the organization. A clinical quality lead should define the levels and examples for the use case. Do not import a vendor’s labels without testing agreement among local reviewers.
Preserve the edit trail
Use four artifacts when the approved workflow makes them available: the authorized encounter source, original AI draft, clinician edits and signed note. Follow local privacy, retention and access policy. Do not create an extra audio archive merely to run this audit.
The edit trail answers different questions. A corrected draft shows that prospective review worked, while the original still reveals a pattern the vendor or training plan may need to address. An unchanged signed note does not prove the source was captured correctly. Reviewers still compare it with the authorized evidence.
The CMS Evaluation and Management compliance guidance says medical records should support the services reported and be complete and legible. Use applicable payer and organizational rules for the actual audit. This article does not decide coding or billing compliance.
Two reviews, then adjudication
Reviewer A and Reviewer B should complete their first pass independently. Hide the other’s ratings until both are locked. Then compare results.
| Field | Reviewer A | Reviewer B | Adjudicated result |
|---|---|---|---|
| Error class | Entry | Entry | Final class |
| Evidence span | Entry | Entry | Cited source and note text |
| Severity | Entry | Entry | Final local severity |
| Rationale | Entry | Entry | Decision and policy reference |
The adjudicator can be one of the reviewers if local policy allows, but the original ratings should remain visible. A large disagreement rate may point to a vague rubric, missing clinical context or insufficient reviewer training. It is a reason to inspect the method before advertising a score.
A peer-reviewed note-assessment study used a modified PDQI-9 framework and two blinded specialty reviewers for each note. That research can inform local rubric design. It is not a regulator’s approval of a product, and the instrument should not be presented as the only acceptable audit method.
Report counts before percentages
For each class, report the number of notes reviewed, number with the finding and number of findings. One note can contain several findings, so those denominators answer different questions.
Also report:
- signed, abandoned and missing drafts;
- notes with a material clinician correction;
- cases sent to adjudication;
- unresolved cases;
- workflow incidents by type;
- excluded notes and reasons.
Do not compress everything into “accuracy.” An omission, a wrong attribution and a comma change should not receive identical weight. We think a one-number vendor leaderboard is the wrong output for a local safety review.
Write the stop rule before looking at results
Clinical leadership, privacy, security and operations should name the events that pause use. A wrong-patient association, a serious unsupported clinical statement, repeated material omission, loss of review or failure of the fallback may be candidate triggers. The organization sets the actual threshold.
The audit plan should state who receives the report, who can stop the pilot, which workflow resumes and what evidence is required to restart. Without that route, reviewers can discover a serious pattern and still wait for the next monthly meeting.
The healthcare AI governance charter provides a decision record for those owners. The AI-scribe implementation guide covers rollout mechanics outside the audit.
What a passing audit means
A pass means the sampled use case met the organization’s stated criteria for that period. It does not prove perfection, validate every specialty or authorize a new workflow. Record the exact scope and the next review trigger.
An organization may decide that a simpler documentation tool is a better fit when it cannot staff independent review and adjudication. That is an honest operating choice. For a team ready to test its own rubric, compare the options.
Frequently asked questions
Should the vendor choose the audit sample?
No. The organization should define the eligible population and selection method. A vendor may help export permitted artifacts, but it should not remove hard cases or substitute hand-picked notes. Include abandoned and missing drafts so the audit measures the workflow rather than only the outputs that reached signature.
Can one clinician review their own notes?
Self-review can provide operational feedback, but it does not give an independent quality check. For a formal audit, add a second qualified reviewer and preserve both initial ratings. The organization should define reviewer qualifications, access, conflicts and adjudication authority for the clinical scope being examined.
Is edit distance a clinical quality measure?
Not by itself. A long stylistic rewrite may leave clinical meaning unchanged, while one short insertion can correct a material omission. Track edit burden as an operational measure and classify clinical meaning separately. Reviewers need the source evidence and signed note, not only a character-difference score.
Can audit data be reused for model training?
Do not assume so. Training is a separate purpose and data flow that needs its own contractual, privacy, security and governance review. The audit protocol should state who can access each artifact, how long it is kept and what happens at deletion. Keep the audit approval bounded to the stated purpose.
How often should the audit repeat?
Set a scheduled cadence and event-based triggers. Repeat after a material model, workflow, population, data-use or destination change and after serious findings. Stable lower-risk use may justify less frequent sampling. The governance record should name the next trigger instead of relying on memory.