Benchmark
AI scribe benchmark: Hanah and seven other AI scribes
Run on 6 October 2026, Arlo added and every note re-graded on 20 October 2026 · 13 recorded consultations · 8 AI scribes
We played the same 13 recorded consultations into Hanah, Heidi, Lyrebird, Freed, PatientNotes, CliniScripts, Preve and Arlo, then graded every note against what was actually said. Every transcript, note and grade is published on this page, with the reason for each one.
92%
of Hanah's notes free of major hallucination flags
The average across the other seven scribes was 42%.
61% fewer
clinically significant transcript errors
0.85 per consultation against an average of 2.17.
13% more
required clinical details captured
93.1% against an average of 82.3%.
Results
Scribes are listed by the share of required clinical details their notes captured. Each scribe's note for the first prompt of each recording is scored.
| AI scribe |
Clinical details captured |
Notes with every clinical detail |
Major hallucination flags per note |
Major hallucination flags, all notes |
Major hallucination flags per document, including summaries and letters |
Major hallucination flags, including summaries and letters |
Notes with no major hallucination flag |
Significant transcript errors per consultation |
Word Error Rate, English consultations |
| Hanah13 notes |
93.1% |
62% |
0.23 |
3 |
0.17 |
4 |
92% |
0.85 |
1.6% |
| Heidi13 notes |
91.4% |
38% |
0.31 |
4 |
0.22 |
5 |
69% |
1.08 |
3.3% |
| CliniScripts13 notes |
90.8% |
62% |
0.54 |
7 |
0.35 |
8 |
62% |
1.08 |
2.1% |
| Lyrebird13 notes |
89.7% |
54% |
0.85 |
11 |
0.74 |
17 |
38% |
2.08 |
3.4% |
| Freed†12 notes |
84.5% |
50% |
0.50 |
6 |
0.43 |
9 |
58% |
1.67 |
19.4% |
| PatientNotes13 notes |
75.3% |
15% |
2.46 |
32 |
2.43 |
56 |
15% |
2.38 |
3.9% |
| Arlo13 notes |
72.4% |
31% |
1.00 |
13 |
1.04 |
24 |
23% |
3.62 |
5.2% |
| Preve*10 notes |
72.2% |
30% |
2.40 |
24 |
2.40 |
24 |
30% |
3.31 |
6.7% |
| Average of the other seven |
82.3% |
40% |
1.15 |
13.9 |
1.09 |
20.4 |
42% |
2.17 |
6.3% |
A significant transcript error is one a blind rater judged could mislead a clinician: a wrong drug, dose, number, side or symptom, a lost negation or red flag, or clinical content dropped. Word Error Rate counts every word the same, fillers included.
The jaw surgery review was spoken mostly in Mandarin. Each scribe ran it with its input language set to Mandarin where it offers that setting. Heidi made the fewest significant transcript errors on it (2), then Hanah (5) and CliniScripts (6). Freed made 10, PatientNotes 12, Lyrebird 14, and Arlo and Preve 18 each. Arlo and Preve have no input language setting, and Freed detects the language itself.
How it was run
The recordings
The 13 recordings are Hanah's accuracy corpus, the same set behind the release results on our transparency page. They cover an ICU handover dictation with and without ward noise, physiotherapy, dietetics, occupational therapy, myotherapy and community nursing consults, a child with asthma whose parent speaks limited English, a quad bike fall recorded outdoors, a 50 minute myotherapy session with about seven minutes of speech in a noisy treatment room, and a jaw surgery review spoken mostly in Mandarin. Each has a verbatim reference transcript and a checklist of the details a note must capture.
Capture
Each scribe ran in its own web app in Chrome, signed in to a standard account. The recording was played through the browser's microphone input, the same file for every scribe. After recording, each recording's prompts were typed into the scribe's own document or chat feature word for word, for example "Generate SOAP notes please" and "Summarise this consult in two sentences for a referral letter". Each scribe was run once per recording on 6 October 2026, and Arlo on 20 October 2026, with English as the input language where a scribe asks for one. For the Mandarin review, scribes with an input language setting were set to Mandarin: Heidi to English and Chinese (Mandarin), Lyrebird and PatientNotes to Chinese (Mandarin, Simplified). CliniScripts produced no transcript with Chinese and English selected on two attempts, so its standard setting is used. Arlo has no input language setting. Arlo builds documents from saved templates rather than a free-text box, so each prompt was typed into Arlo's own template generator word for word and the template it built was used unedited.
Grading by a Claude jury
The notes were graded by Claude, Anthropic's model. For each recording, every scribe's output was shuffled behind a letter with product names removed, and graded against the reference transcript with a fixed rubric. Each checklist item is marked yes, partial or no. Every statement the transcript doesn't support is listed as a hallucination flag: major when it could change care or misstate the clinical picture (an invented symptom, finding, drug, diagnosis or plan, or a wrong side or drug name), minor otherwise. On 20 October 2026, when Arlo was added, every scribe's notes and transcripts were graded again together in the same blind packets, so all eight are scored by the same grader. The grades were unblinded only after every recording was graded. The full rubric is in our methodology.
Transcript errors
Each transcript was aligned word by word with the verbatim reference. Differences in filler words were set aside, and Claude judged each remaining difference blind, using the rule above for what makes a transcript error significant. Word Error Rate is calculated by code with the same strict normalisation as our own release reports, treating equivalent written forms as the same words: 22 mm and 22 millimetres, US and UK spellings, Chinese and Arabic numerals, and Simplified and Traditional characters. Chinese is scored per character.