How to Evaluate an AI Medical Scribe: 9 Questions

Most AI scribe demos look identical. Clean note, two minutes, scripted visit. The differences that decide it sit underneath: what the tool does with your audio, whether it drafts prescriptions and how it checks them, whether it suggests codes or pretends to bill them, which languages it actually handles, whether you can take your notes and walk, and what the price is after year one. Nine questions sort the field. Here’s the scorecard.

Take it into every demo. A vendor who answers all nine cleanly, in plain sentences, is rare. That’s the point.

Key takeaways

• Nine questions separate AI scribes faster than any feature grid: audio handling, Rx drafting, Rx safety, coding honesty, languages, export rights, BAA posture, pricing transparency, and trial quality.
• The fastest disqualifier is the audio answer. A vendor who can’t say where the recording lives and for how long, in one sentence, has answered the question.
• Primary-care physicians log a median of 36 minutes of EHR time per 30-minute visit, so the tool you pick runs your evenings, not just your notes.
• Make the scorecard your demo agenda. Ask the same nine of every vendor and compare the answers side by side.

The 9-question scorecard, at a glance

Score each vendor 0 to 2 per row: 0 if the answer is evasive or wrong, 1 if partial, 2 if it’s a clean, specific, in-writing yes. A tool worth trialing clears most of these without a follow-up email.

#	Question	What a strong answer sounds like
1	What happens to the visit audio, and when is it deleted?	”Processed in memory, discarded when the note drafts. No archive.”
2	Does it draft prescriptions, and what checks them?	”Rx draft, run through a safety screener; you review and sign.”
3	Does it claim to code or bill?	”ICD-10 suggestions you confirm. Not a coding or billing engine.”
4	Which languages does it actually capture?	A named list that matches your patients, with notes in clean English.
5	Can I export and delete any visit, anytime?	”Yes and yes, self-serve, no fee, no support ticket.”
6	Will you sign a BAA, and are you “HIPAA certified”?	”BAA for any practice size. And no one is HIPAA certified, no such certificate exists.”
7	What’s the price after year one?	The full ladder in writing: launch, annual, month-to-month.
8	What’s the real failure mode on a noisy, multi-speaker visit?	An honest answer, not “it’s perfect.”
9	Can I trial it on my own visits before I commit?	”Yes, 7 days, your real clinic, no card.”

The rest of this guide is one section per question, so you know what a good answer looks like and where vendors quietly fail.

Question 1: what happens to the visit audio?

Ask this before you ask about price. A visit recording is more revealing than the note, because it holds everything that didn’t make the note. Vendors handle it very differently. Some retain audio for days or weeks. Some let you opt out. Some are vague, which is its own answer.

This isn’t theoretical. A proposed class action filed against Sharp HealthCare in late 2025 alleges patients were recorded by an ambient AI tool without consent, with more than 100,000 patients potentially affected. The cleanest defense against that whole category of risk is not retaining the audio at all.

Our position: audio is processed in memory and discarded the moment the note is drafted. No archive, not for us, not for the practice, not for anyone. What survives is the note you reviewed and signed. If you want the privacy and consent angle in depth, our psychiatry privacy guide walks the behavioral-health version of this question, where the stakes are highest.

Question 2: does it draft prescriptions, and what checks them?

A lot of scribes stop at the note. Some draft a prescription too, which saves real time, but a prescription draft an LLM wrote is still a draft an LLM wrote. The question isn’t only “does it draft Rx,” it’s “what catches a bad one.”

AI Medical Scribe by Patient Square is an ambient AI medical scribe that listens during the visit and hands back a structured SOAP note, ICD-10 suggestions, and a prescription draft, ready to review and sign about two minutes after the visit. The prescription draft passes a deterministic safety screener that checks drug interactions, renal dosing, and pregnancy flags. It re-runs at sign time and hard-blocks unsafe combinations unless you override with an attestation. It’s a draft, not e-prescribing, and you stay the prescriber. The mechanics are in our prescription draft safety explainer.

Question 3: does it claim to code or bill your notes?

Here’s a line that separates honest vendors from the rest. A scribe can suggest ICD-10 codes from the conversation. It should not claim to be a coding engine, an E&M leveler, or a billing system. Coding is a human-confirmed clinical-and-financial decision, and a tool that pretends otherwise is selling risk.

We offer ICD-10 suggestions to speed your coding. A coder or clinician confirms them. That’s the whole claim. We don’t ship CPT, E&M, or HCC automation, and we won’t pretend to. The honest version of this distinction is the entire point of our ICD-10 suggestions explainer. If a vendor blurs “suggests codes” into “codes your notes,” ask exactly what gets submitted and by whom.

Question 4: which languages does it actually capture?

Read the language list literally, then test it. “Multilingual” without a named list is marketing, and a list that looks long on a slide can still miss the accents and dialects your patients actually speak. The only proof is your own clinic.

We capture English and more languages, with notes always returned in clean clinical English. Input can be multilingual; the output note is English. In low-signal settings, capture works offline with on-device encryption and syncs later. If your patients code-switch or carry accents a model rarely hears, that’s the test case to run on day one of a trial, not after procurement.

Question 5: can you export and delete any visit, anytime?

This is the lock-in question, and it’s easy to skip until you want to leave. Ask three things. Can I export any visit? Can I delete any visit? Do you sell or share clinical data? The answers you want are yes, yes, and no, and you want them in the contract, not the sales deck.

Our answer: notes belong to your practice, you can export or delete any visit at any time, and we never sell or share clinical data. If export needs a support ticket or carries a per-record fee, that’s friction designed to keep you. For the buyer who wants the full ownership and exit checklist, run the same questions a switching practice runs.

Question 6: will the vendor sign a BAA, and what’s the compliance posture?

The compliance floor isn’t optional, and a vendor who can’t clear it is out, however good the note looks.

The floor is a signed BAA, available to any practice size, plus encryption in transit and at rest and access you can audit. Software is never “HIPAA certified,” because no such certificate exists. Treat that phrase as a red flag. We map our safeguards to the HIPAA Security Rule and offer a BAA to every customer.

One level up, ask where the vendor sits on an independent audit. Ours is straight: our SOC 2 Type II audit is underway, and we won’t call ourselves “certified” until it’s finished. “Audit in progress” and “certified” are different things, and a vendor who runs the two together has told you something about how carefully they’ll describe everything else.

Question 7: what’s the price after year one?

Get the full ladder, per clinician, in writing: the launch rate, the annual-commitment rate, and the month-to-month rate. Three numbers, no asterisks. Vendors that route a per-seat subscription to “contact sales” are telling you the renewal will be a negotiation.

We publish the whole ladder: $89 per clinician per month on annual billing, no feature gating between tiers. The full table is on our pricing page. Compare cost per visit, not the sticker: $89 across 400 visits a month is about $0.22 a visit. For the head-to-head across the self-serve field, our best AI medical scribes roundup lays out the honest comparison.

Question 8: what’s the honest failure mode?

Every scribe has one. The vendor who admits it is more trustworthy than the one who claims perfection. Two failure modes are worth probing.

First, noisy, multi-speaker rooms. A quiet consult room is the easy case. A crowded clinic with a relative answering half the questions, or a pediatric visit with a caregiver in the room, is the real test, and quality varies sharply across products. If your setting is loud or multi-party, make the demo run on a recording that sounds like your room.

Second, the drafts themselves. Models mishear drug names. They compress two complaints into one. Now and then they write something plausible that didn’t happen. That’s why review-and-sign is load-bearing, why our Rx drafts pass a safety screen, and why you should distrust any pitch that implies you can skip reading the note.

Question 9: can you trial it on your own visits?

The trial is the decision. A scripted demo flatters every scribe equally; your Tuesday clinic doesn’t. Run a 7-day trial on a normal day, with your patient mix, your languages, your interruptions. Grade three real notes against what you’d have written. If the tool survives your worst visit of the week, it’ll survive the rest.

We offer a 7-day free trial, no card to start. The structured way to run one, with a baseline day and a note-grading rubric, is in our how to run a trial guide. And if you’re about to roll a tool out across a small clinic, the 2-week implementation plan turns the trial into a rollout.

Turn the scorecard into a demo agenda

Don’t watch nine demos and try to remember the differences. Bring these nine questions to each one, write the answers in a row, and compare. The vendor with the most 2s, especially on audio, Rx safety, coding honesty, and export, is the one to trial.

When you’re ready to see how AI Medical Scribe by Patient Square answers all nine against your own visit type, book a demo. Then run the 7-day trial on a real clinic week. The scorecard tells you what to ask; the trial tells you what’s true.