AI scribe accuracy benchmark — United Kingdom
5.9% Word Error Rate, corpus v1
The LLM jury found 91.5% coverage of the 106 clinical facts pre-registered before the models ran, and flagged 30 claims as unsupported by the transcript across 19 generated documents — 3 of those flagged by both jurors.
Open the box on accuracy. Every recording in our corpus runs end-to-end through the production pipeline. We publish what was said, what we transcribed, what was generated, and the verdict each LLM juror reached on whether the document captured what mattered. Numbers and methodology, no PR gloss.
How this was measured
- Corpus
- v1, version-pinned in the repository — 10 recordings graded in this run, 2,810 reference words, spanning allied-health, primary-care and outpatient appointment types. All 10 were graded; none was skipped and none failed.
- Audio
- Synthesised speech from several text-to-speech vendors, covering NZ, AU, UK, US, Indian and Pacific-influenced English plus L2 speaker profiles, mixed with the conditions clinicians record in. 4 of the 10 recordings carry an accent or audio-condition tag.
- Reference transcripts
- The written script the audio was synthesised from is the reference transcript, so ground truth is exact rather than transcriber-approximated. No real patient audio is in the corpus.
- What counts as an error
- Word Error Rate is substitutions plus insertions plus deletions over the reference word count, computed with
jiwer. Text is lowercased, punctuation stripped, common abbreviations expanded and Unicode-normalised (NFKC) first. A clinically serious substitution (10mg for 100mg) and a cosmetic one score the same — which is what Layer B is for. - What counts as coverage
- Each clinically-important fact is written down and signed off before any model sees the audio, then graded yes / partial / no by every juror, one fact per call. Coverage is the share of those facts captured, counting a partial as half. Separately, every claim in the generated document that the transcript does not support is enumerated.
- Jury
- Two models from rival vendors, each grading every fact independently: Google
gemini-2.5-flash-liteand OpenAIgpt-5-nano, three passes each, a juror's vote being the majority of its own three. With a two-juror panel, agreement means unanimity: this page's headline hallucination count requires both jurors, and single-juror flags are reported beside it as provisional. - Excluded
- v1 claims nothing about te reo Māori, authentic code-switching, dysfluency, dysarthria or paediatric speech. Mixed-in television, room ambience and outdoor noise approximate a consult room rather than being a recording of one, and synthesised accents are not a substitute for real speakers.
- When it was run
- Graded 29 June 2026 on the United Kingdom production stack; aggregate published the same day.
Read the full methodology, including the limitations we state up front →
Cite this benchmark
Hanah (2026). AI scribe accuracy benchmark, United Kingdom — corpus v1: Word Error Rate and LLM-jury results. Hanah, published 29 June 2026; run graded 29 June 2026. https://hanah.health/uk/trust/transparency/
Every figure above is recomputed when a run is republished, so cite the run date alongside the URL. The machine-readable aggregate the page renders from is at data.js.
Headline metrics
Two layers of measurement. Layer A is mechanical — Word Error Rate against verbatim ground-truth. Layer B is qualitative — an LLM jury grades each clinically-important detail we declared up front (coverage: yes / partial / no), and separately enumerates every claim in the generated document that the transcript does not support (hallucinations: fabricated content the model invented).
Coverage breakdown
Jury composition
Each detail is graded by every juror configured in the panel. Verdicts disagree on hard cases — that's the signal. Per-juror reasoning is shown verbatim below.
Sessions
Click any recording to drill in: source audio, ground-truth transcript, what the STT pipeline heard, every generated document, and every per-LLM verdict with its reasoning.
Methodology & provenance
Word Error Rate, panel-of-LLM jury, prompt-scoped expectations, calibration sanity-check, and corpus graduation roadmap (synthetic → role-play → real-world).
Read the methodology document →The full machine-readable artefact this page renders from. Every number on this page is computable directly from this file.
Open raw data →Corpus, juror prompts, and judge model versions are all version-controlled. Anyone can re-run the harness against the same commit and reproduce these numbers.
Trust Center home →