The 12-Item AI Scribe Vendor Evaluation Checklist
The 12-Item AI Scribe Vendor Evaluation Checklist
Evaluating an AI scribe is not the same as evaluating software. The evaluation involves HIPAA compliance, clinical vocabulary accuracy, workflow fit, data handling, and billing support — across a market where vendors range from genuinely purpose-built clinical tools to consumer transcription tools with a "healthcare" label added.
Most clinicians start with a demo. The demo is always impressive. The right approach is to start with a structured vendor inquiry before the demo — because several of the most important evaluation criteria cannot be observed in a demo environment that uses fake patient data and is configured for optimal performance.
This checklist covers 12 questions across four domains: compliance, clinical accuracy, workflow integration, and economics. Ask each question in writing, before you enter any real patient data, and before you commit time to a full trial.
A printable PDF version of this checklist is available at /resources/ai-scribe-vendor-checklist.pdf.
Domain 1: Compliance (Items 1–4)
Item 1: Is a Business Associate Agreement available on the plan I am evaluating, before I enter any patient data?
This is the highest-priority filter and should be the first question you ask. If the BAA is gated behind an enterprise plan, a sales call, or a minimum seat count, the product is not available to you in a compliant form at the tier you are evaluating. Do not proceed under the assumption that you will upgrade later.
What good looks like: BAA available on every paid plan, readable before signup, no minimum commitment required.
Red flag: "Our BAA is available on Business and Enterprise plans." On an individual or starter plan, this means you are not covered.
Follow-up: Can I receive a copy of the BAA to review before creating an account?
Item 2: Where is patient audio processed and stored, and for how long?
Audio of a clinical session is PHI the moment it is recorded. How the vendor handles that audio — where it goes, how long it is retained, who can access it — is a core HIPAA question and a major risk differentiator.
What good looks like: Audio processed in memory through a HIPAA-eligible pipeline and deleted immediately after the transcript and note are generated.
Red flag: Audio retained for "quality improvement" or "model training" without explicit patient consent. This creates a larger breach surface and typically is not properly disclosed in the BAA.
Follow-up: Is audio retained after the note is generated? For how long and for what purpose?
Item 3: Does your BAA address sub-processors, including your transcription engine?
Most AI scribe vendors call APIs from Deepgram, AssemblyAI, Azure Speech, Speechmatics, or other engines. Each engine receives patient audio, making it a sub-processor touching PHI that requires its own BAA with the AI scribe vendor.
What good looks like: The vendor can name their transcription engine(s) and confirm a BAA is in place with each. Some vendors publish their sub-processor list.
Red flag: "We use a HIPAA-compliant transcription engine" without naming the engine or confirming the sub-BAA.
Follow-up: Can you provide your list of sub-processors who receive patient audio and confirm each has signed a BAA with you?
Item 4: Do you have a SOC 2 Type II report available?
A BAA tells you what the vendor promises. A SOC 2 Type II report tells you whether an independent auditor verified those controls were actually operating over 6–12 months. These are different things.
What good looks like: SOC 2 Type II report available for review (typically under NDA), issued within the past 12 months.
Red flag: "We are working toward SOC 2 compliance." Controls are unaudited in the interim.
Follow-up: When was your most recent SOC 2 Type II audit completed? Who conducted it?
Domain 2: Clinical Accuracy (Items 5–7)
Item 5: How does your transcription engine handle psychiatric medication names and DSM-5 diagnostic terminology?
General-purpose transcription engines are not trained on clinical vocabulary. Psychiatric medication names — bupropion vs. buspirone, quetiapine vs. quetiapine XR, lamotrigine vs. lamotrigine ER — are phonetically similar in ways that create clinically meaningful errors. DSM-5 diagnostic language and symptom descriptors require specific training to capture accurately.
What good looks like: The vendor uses an engine specifically trained on medical vocabulary, can provide word error rates on clinical samples, and has documented which medication categories their engine handles accurately.
Red flag: "We use a state-of-the-art AI engine" without any specifics. Ask which engine and request accuracy data.
Follow-up: What is your word error rate on psychiatric medication names? Do you have accuracy benchmarks for clinical vs. general speech?
Item 6: Can I test transcription with my own voice and recording conditions before trial?
Demo recordings are optimized. Your sessions happen in real offices, with your accent, your audio equipment, and sometimes background noise. A vendor who will not let you test with your own voice before trial is preventing you from evaluating the thing that matters most.
What good looks like: Free or low-cost test recordings with your own audio before committing patient sessions.
Red flag: Trial requires creating a full account with billing information before you can test transcription quality.
Follow-up: Can I submit a sample recording for accuracy testing before starting a trial?
Item 7: How does note generation handle my specific note format and specialty?
A scribe optimized for primary care SOAP notes will not produce adequate DAP, BIRP, or psychiatric E&M documentation without customization. If the vendor has no specialty-specific examples, production quality may be lower than the demo.
What good looks like: Templates specific to behavioral health or your specialty, reviewable before trial. Custom template support without a professional services engagement.
Red flag: "Our AI adapts to any specialty" without examples. This typically means mediocre quality in specialized contexts.
Follow-up: Do you have templates for [BIRP/DAP/psychiatric E&M/your format]? Can I see a sample generated note?
Domain 3: Workflow Integration (Items 8–10)
Item 8: What is your EHR integration story — direct integration, copy-paste, or API?
Your note has to get into your EHR. The method matters for workflow efficiency and, in some cases, for documentation integrity. The integration landscape in 2026 is mixed: a small number of EHRs have direct API integrations with AI scribes; most workflows still require copy-paste or manual import.
What good looks like: Direct integration with your EHR, or a clear roadmap. If copy-paste is required, the export format should match your EHR's note entry structure.
Red flag: "We integrate with all major EHRs" followed by a list that does not include yours, or an "integration" that is a CSV export.
Follow-up: Do you have a direct integration with [your EHR]? If not, what is the recommended copy-paste workflow?
Item 9: What is the end-to-end latency from session end to a review-ready note?
If the note is not ready before your next patient, the between-session documentation workflow breaks.
What good looks like: Transcript and note complete within 2 minutes of session end. Faster engines achieve under 30 seconds.
Red flag: "Notes are ready in minutes" without a specific benchmark. Two minutes and twenty minutes are both "minutes."
Follow-up: What is your average time from recording end to review-ready note for a 50-minute therapy session?
Item 10: What happens when transcription quality is poor?
Audio quality varies, recordings fail, and transcription engines produce low-quality output in some conditions. How the vendor handles these cases determines real-world reliability.
What good looks like: Quality indicators on generated notes, ability to re-process a recording, support response time under 4 hours for production issues.
Red flag: No quality indicators, no re-processing capability, support response times measured in business days.
Follow-up: How am I alerted to low-confidence transcriptions? What is your SLA for production recording failures?
Domain 4: Economics (Items 11–12)
Item 11: What is the total cost of ownership, including any per-session, per-recording, or add-on fees?
Published pricing often covers only the base subscription. The actual cost of running the tool in production depends on per-session fees (some vendors charge per recording above a monthly threshold), storage fees, EHR integration fees, custom template fees, and support tier pricing.
What good looks like: Flat monthly pricing covering your expected volume, no per-recording overage fees, BAA and standard templates included at every tier.
Red flag: Low base price followed by per-session charges, compliance features gated behind add-ons, custom templates requiring professional services.
Follow-up: At [X] sessions per month, what is my all-in monthly cost? Are there per-session charges above a threshold?
Item 12: What is your data portability and exit policy?
Clinical records are your practice's legal and professional responsibility. If a vendor fails, gets acquired, or becomes unusable, you need to export your documentation history in a usable format.
What good looks like: Export in standard formats (PDF, structured text) available at any time, not just on cancellation. Data returned or confirmed deleted within 30 days of cancellation.
Red flag: Export only on the highest-tier plan, proprietary formats only, or no documented data return process on cancellation.
Follow-up: How do I export my note history? Is PDF or plain text available? What is your data return or deletion process on cancellation?
Scoring Your Vendor Evaluation
Run each candidate vendor through all 12 items before starting a trial. Score each item:
| Score | Meaning |
|---|---|
| 2 | Strong answer — matches "what good looks like" |
| 1 | Acceptable answer with caveats — needs follow-up |
| 0 | Red flag present — significant concern |
| N/A | Not applicable to your practice type |
Scoring guide:
- 20–24: Strong compliance and capability posture — proceed to trial with standard diligence
- 14–19: Acceptable with known gaps — proceed to trial only if gaps are in non-critical domains
- Below 14: Significant concerns — do not proceed to trial until concerns are resolved in writing
Items 1–4 (compliance domain) should all score 2. Any score of 0 in the compliance domain is a disqualifying finding regardless of overall score.
Before You Start the Trial
Once a vendor clears this checklist, the trial evaluation focuses on:
- Note quality for your specific session types
- Review time and workflow efficiency in actual clinical use
- EHR paste-in reliability
- Support responsiveness when things go wrong
Most clinicians find that 10–15 sessions with a consistent patient population is enough to assess note quality and workflow fit. Do not evaluate during your most complex cases or your most straightforward ones — evaluate in the middle of your typical caseload.
For additional context on compliance evaluation, see HIPAA primer for AI scribe evaluation and audit readiness for AI-generated notes. For specialty-specific evaluation criteria, see AI scribe for psychiatry and AI scribe for behavioral health.
Scriber's compliance documentation is available before you create an account. Review the BAA and security details, download the vendor checklist PDF, or start a free trial with no commitment required.