How to Evaluate AI Scribes: Vendor Checklist | Scriber Wiki
Last updated
If you're evaluating multiple AI scribe vendors, you need to assess three categories: clinical capability, compliance, and integration. This checklist helps you compare apples to apples.
Clinical Capability
Transcription accuracy. Ask vendors for their word error rate (WER). Expect 3–8% WER for clean audio; above 12% is concerning. Compare against your actual patient audio if possible. Some vendors will run a free trial on a sample recording.
Note format support. You need the formats you actually use. Common formats: SOAP, DAP, BIRP, GIRP, SIRP, biopsychosocial, intake, treatment plan, discharge summary. Does the vendor support your specialty (therapy, psychiatry, counseling, primary care)? Can you create custom templates?
Multi-language support. How many languages? At minimum: English, Spanish, French, Mandarin. If you work with non-English speakers, verify the vendor's specific language performance (some languages have higher error rates).
Clinical accuracy beyond transcription. Transcription is only half the problem. Does the AI correctly map transcribed content to the note template? Read some sample outputs, not just transcripts. A perfect transcript mapped to the wrong assessment section is useless.
Speaker diarization. Does it distinguish clinician vs. patient speech? This matters for therapy notes where capturing the patient's own words is important.
Compliance
HIPAA compliance. All vendors should claim this, but ask:
- Is there a signed Business Associate Agreement (BAA) included, or is it enterprise-only?
- Where is data stored (US data center, specific region)?
- What's the data retention policy — do they store audio, transcripts, or both?
- Is data encrypted in transit and at rest?
Audio retention policy. This is critical. Some vendors keep audio "indefinitely for quality improvement." Others delete it immediately after transcription. For behavioral health, you should strongly prefer vendors who delete audio immediately. Ask explicitly: "Do you retain audio after transcription?" (The answer should be "no" or "only for X days.")
Audit logs. Can you access audit logs showing who accessed which patient records? This matters if your EHR compliance team audits you.
Compliance certifications. Beyond HIPAA, do they have SOC 2, FedRAMP, HITRUST, or other third-party certifications?
Data location and residency. Some vendors use cloud providers (AWS, Azure, Google Cloud). Ask which region. If you have state-level data residency requirements, confirm the vendor meets them.
Liability and insurance. If something goes wrong (data breach, incorrect note), what's the vendor's liability? Is it covered by cyber insurance? Ask about their E&O coverage.
Integration
EHR compatibility. Can the note export cleanly to your EHR? Best case: direct API integration (Scriber has direct integration with some platforms). Second-best: copy-paste compatibility (note is formatted so it pastes cleanly). Worst case: you have to manually reformat the note into your EHR.
Telehealth platform integration. If you use Zoom, Google Meet, Teams, or Doxy.me, does the vendor integrate natively? Direct integration means the session audio flows directly to the scribe without a separate recording device. This is faster and more reliable than asking you to record separately.
Calendar integration. Does the vendor pull your schedule from your calendar or practice management system? This reduces manual setup (you don't have to manually label each patient).
Pricing model. Monthly subscription ($50–150/month is typical)? Per-note fees? Per-minute fees? If you're seeing 20+ patients per week, unlimited monthly is usually cheaper than per-note.
Free tier or trial. Can you test it before committing? Ideally 7–14 day free trial, or a free tier with limited sessions.
Vendor Comparison Template
Print this and fill in for each vendor:
| Criterion | Vendor A | Vendor B | Vendor C |
|---|---|---|---|
| WER on clean audio | ___ | ___ | ___ |
| Formats supported | SOAP, DAP, custom | SOAP only | SOAP, DAP, BIRP |
| Languages | 30+ | 5 | 8 |
| BAA at signup? | Yes | Enterprise only | Yes |
| Audio deleted after transcription? | Yes, 30 min | No, indefinite | Yes, 1 min |
| Doxy.me integration | Yes, direct | No | Yes, via API |
| SimplePractice integration | Copy-paste | Copy-paste | Direct API |
| Monthly cost | $99 | $149 | $79 |
| Free trial? | 14 days | 7 days | 7 days |
Fill this in and rank by your priorities. If HIPAA compliance and audio deletion are non-negotiable, that filters the list quickly. If EHR integration is critical, that filters further.
Red Flags
Unclear on data retention. If a vendor can't clearly answer "do you delete audio after transcription," that's a red flag. Ask again. If they still can't answer, move on.
No BAA or enterprise-only BAA. For healthcare, a BAA at signup is standard. Requiring enterprise sales conversations just to sign a BAA is a sign of higher friction (and usually higher cost).
No pricing transparency. If the vendor won't publish prices or forces you to "request a demo," that usually means enterprise pricing ($1,000+/month). Make sure you know the cost before investing time in evaluation.
Overstated accuracy claims. Vendors claiming 99%+ accuracy are either cherry-picking test datasets or measuring something different than real-world performance. Real-world WER is 3–12% depending on audio quality.
Limited format support. If you use DAP notes and the vendor only supports SOAP, the tool won't work for you. Match the vendor's formats to your actual needs.
No integration support. If the vendor only exports PDFs or emails notes to you, every note requires manual copy-paste. This eliminates most of the time savings.
Final Step: Free Trial Checklist
Before committing, run a 1-week free trial and verify:
- I can record or integrate my actual patient sessions
- Transcription accuracy is acceptable for my audio environment
- Generated notes match my usual format and clinical style
- I can review and finalize a note in <5 minutes
- Export works cleanly into my EHR or note system
- I feel confident signing the notes without extensive editing
- The vendor's compliance claims match our requirements
If you check all these boxes, the vendor is worth the cost. If you're hesitating on 3+ items, keep evaluating.
Related questions
See it in action with Scriber
Start with a 7-day trial — HIPAA-compliant, no audio retention, BAA included at signup.