AI scribe accuracy benchmark — Australia

How this was measured

Corpus

v1, version-pinned in the repository — 11 recordings, 2,857 reference words, spanning allied-health, primary-care and outpatient appointment types. All 11 were graded; none was skipped and none failed.

Audio

Synthesised speech from several text-to-speech vendors, covering NZ, AU, UK, US, Indian and Pacific-influenced English plus L2 speaker profiles, mixed with the conditions clinicians record in. 5 of the 11 recordings carry an accent or audio-condition tag.

Reference transcripts

The written script the audio was synthesised from is the reference transcript, so ground truth is exact rather than transcriber-approximated. No real patient audio is in the corpus.

What counts as an error

Word Error Rate is substitutions plus insertions plus deletions over the reference word count, computed with jiwer. Text is lowercased, punctuation stripped, common abbreviations expanded and Unicode-normalised (NFKC) first. A clinically serious substitution (10mg for 100mg) and a cosmetic one score the same — which is what Layer B is for.

What counts as coverage

Each clinically-important fact is written down and signed off before any model sees the audio, then graded yes / partial / no by every juror, one fact per call. Coverage is the share of those facts captured, counting a partial as half. Separately, every claim in the generated document that the transcript does not support is enumerated.

Jury

Two models from rival vendors, each grading every fact independently: Google gemini-3.6-flash and OpenAI gpt-5, one pass each. With a two-juror panel, "flagged by both jurors" means unanimous, so the headline count is every flagged claim and the drill-downs show which were corroborated.

Excluded

v1 claims nothing about te reo Māori, authentic code-switching, dysfluency, dysarthria or paediatric speech. Mixed-in television, room ambience and outdoor noise approximate a consult room rather than being a recording of one, and synthesised accents are not a substitute for real speakers.

When it was run

Graded 30 July 2026 on the Australian production stack; aggregate published 12 September 2026.

Read the full methodology, including the limitations we state up front →

Headline metrics

Two layers of measurement. Layer A is mechanical — Word Error Rate against verbatim ground-truth. Layer B is qualitative — an LLM jury grades each clinically-important detail we declared up front (coverage: yes / partial / no), and separately enumerates every claim in the generated document that the transcript does not support (hallucinations: fabricated content the model invented).

The headline counts every flagged claim, including those raised by a single juror. We reviewed all of them by hand. On the current corpus: four were the document's author block (clinician name, profession and date) written into clinical sections, a fault we traced and fixed on 1 August; three were fabricated clinical findings, where the note recorded an examination or a denial that never happened; three were over-readings, where the model inferred past what was actually said. Every claim, with the juror's reasoning and the transcript it was checked against, is in the per-recording drill-downs below.

Coverage breakdown

Jury composition

Each detail is graded by every juror configured in the panel. Verdicts disagree on hard cases — that's the signal. Per-juror reasoning is shown verbatim below.