How we measure accuracy.

Why this is one methodology, not two

It's tempting to publish a WER number and a separate "hallucination rate" on different pages. We don't, because they share everything that matters: the same audio, the same end-to-end run, the same versioned corpus, the same release cadence, and the same set of reviewers signing off the underlying ground truth. Splitting them into two pages would suggest two independent programmes — in fact a single run produces both numbers, and a regression in one usually shows up in the other. Each recording flows through the production pipeline once; the artefacts from that one run feed every layer of evaluation below.

1. 1
   Recording — audio + verbatim ground-truth + pre-registered fact list, version-pinned in the repo.
2. 2
   End-to-end run — clinician sign-up, patient creation, recording, document generation. Same paths a production user takes.
3. 3
   Layer A — WER compares the captured transcript to ground truth.
4. 4
   Layer B — LLM jury grades the generated document against the fact list, atomically, one fact per call.
5. 5
   Aggregate & publish — single aggregate.json renders the public transparency page.

The corpus

A fixed, version-pinned set of recordings the methodology runs against. Each recording is paired with a verbatim transcript and a hand-authored fact list of the clinical points that should appear in the generated note (and the points that must not).

Graduation roadmap

Layer A — Word Error Rate

The mechanical layer. Word Error Rate is a long-established speech-recognition metric: the count of substitutions, insertions, and deletions between what was transcribed and what was actually said, divided by the reference word count. Lower is better; 0% means the transcript matched ground truth word-for-word.

• Library. jiwer, version-pinned in the harness lockfile, so the exact numbers are reproducible from any commit.
• Normalisation. Lowercase, strip punctuation, expand common abbreviations (mg → milligram, etc.), Unicode NFKC. Normalisation rules are version-pinned; cosmetic differences don't get penalised, real errors do.
• Confidence intervals. Bootstrap resampling (10,000 resamples) at 95% confidence; published as WER% [lower, upper], not a bare point estimate.
• Per-cohort breakdown. Computed overall and per cohort tag (accent, age band, gender, specialty, audio condition, language). The variance across cohorts is the bias signal — even when the absolute numbers are upper bounds.
• Per-recording WER. Published so a regression can be traced back to a specific recording, not buried under an average.

Industry baseline anchor. Published WER ranges for ambient scribe products in healthcare typically sit in the 6–12% range for clean-audio English on technical content. Hanah's v1 numbers are presented alongside this range so reviewers have context, rather than evaluating a bare number against an undefined "clinically acceptable" bar.

Honest limitation. A high-impact substitution (10mg → 100mg) and a low-impact one (okay → OK) count the same in raw WER. That's why Layer B exists — clinical-content correctness cannot be inferred from token-level WER alone.

Layer B — LLM jury for coverage and hallucination

The qualitative layer. The generated clinical document is graded against the recording's pre-registered fact list by a panel of independent LLM evaluators drawn from multiple vendors. Two properties are measured separately so they can be reported separately:

The exact prompt every juror sees

System prompt and user-message template are version-pinned in the harness repo (app/next/tests/accuracy/lib/juror.ts) and synced to this page on every results refresh. Both are reproduced verbatim below — auditors see exactly what the model was asked, not a paraphrase. Reruns and panel composition vary; the prompt envelope itself does not.

Atomic single-question calls

Every judgment the harness asks of the jury is one fact, one question, one call. Asking a model to grade an entire note in a single call is well-known to produce "looks fine" averaging where individual errors get smoothed over. Atomic calls are also auditable — anyone can re-run a single judgment and see all three juror responses plus the panel verdict.

What counts as a hallucination

For each output, the jury is given the transcript and the generated document. Each juror enumerates every clinically-meaningful claim in the output that is not supported by the transcript — the model invented it. The panel result is the cross-juror union, deduped by claim text. The headline metric counts only confirmed flags — claims flagged by ≥2 jurors; single-juror flags are surfaced as "provisional" in the per-output drill-down but excluded from the headline. The two-juror agreement bar is what makes the metric robust against a single juror's strict-interpretation false positives.

We deliberately do not flag content that is grounded in the transcript but outside the expectations list. detailsThatMustBeCaptured is a must-have list, not a "this is everything the document should contain" contract — penalising every grounded claim that wasn't on it produced too many false positives, especially for SOAP-style notes whose Objective and Assessment sections legitimately go beyond the named must-haves.

Jury composition

Independent models from multiple vendors, so failure modes are uncorrelated. The harness is provider-agnostic: any model exposed through a supported adapter can sit on the panel, and panel size grows over time as adapters land. The pattern is published as Replacing Judges with Juries (Verga et al., 2024, Cohere) — we use it because triangulation across independent models lands more cleanly in clinical-safety contexts than single-model trust.

Same-vendor caveat. Gemini may participate in the jury, but the production document-generation pipeline also uses Gemini, and same-vendor judging can produce correlated blind spots — a fabrication produced by Gemini-as-generator may be more likely to be considered plausible by Gemini-as-judge. The mitigation is operational: Gemini's verdicts are reported separately from the multi-vendor panel verdict, so a reader sees what Gemini-only would have said versus what the multi-vendor panel said. When the panel is N=1 and that single juror is Gemini (e.g. early dev runs), the validation report names this explicitly so readers don't over-read those numbers.

Decision rule (scales with panel size)

Reports include the per-recording jury agreement rate: the fraction of calls where the jury was unanimous (when N ≥ 2). High unanimity signals clear-cut cases dominate; high split rate signals the methodology is operating in the hard zone. Both are useful information for evaluators, not flaws to hide.

Repeat-and-vote per juror

Each juror is called R times at temperature 0 (default R = 3, configurable per run); the juror's vote is the majority of its own R responses. This kills the 5–10% of calls where a single juror has a flaky response. Cost is N × R calls per judgment; the model classes used are cheap relative to the generation pipeline, and the variance reduction is the point. Lower R for fast development iteration; raise R for higher-confidence published numbers.

Calibration and inter-rater agreement

• Calibration sanity check (per harness version). A hand-crafted obviously-good note and an obviously-bad note are run through the jury. The jury must score these correctly — if it doesn't, the methodology is broken before any real evaluation runs and the harness is paused for investigation.
• Inter-rater agreement against humans (per report). A 20% subset of recordings is independently graded by the Clinical Lead. Cohen's κ is computed between jury verdicts and clinical-expert verdicts on that subset and published in the validation report.

Cohen's κ is not a vibe metric — it controls for chance agreement. Including it converts the jury from "trust the LLMs" to "the LLMs were measured against a clinician on every report".

Layer C — Semantic drift scheduled

A bridge metric between raw WER (token-level) and the LLM jury (clinical-content level). Both the reference transcript and the generated documentation are embedded with a shared embedding model; cosine similarity gives a drift score per recording. Captures restructuring fidelity — documentation is meant to abstract from the source, but should preserve clinical meaning. Useful as a cheap continuous metric without per-recording annotation, and trends easily over time. Outliers (drift > N σ from mean) are flagged for human review. On the roadmap; not in the v1 published numbers.

Repeatability

A pipeline can be accurate-on-average but inconsistent — the same audio producing different documentation each time is a clinical-safety concern, not a curiosity. A subset of corpus recordings is processed N times per CI run (default 3). All evaluation layers run on each rerun; variance across reruns is the repeatability score, reported alongside the headline accuracy numbers.

Reporting cadence

Reproducibility

• Versioned everything. Audio + transcripts + fact lists + jury prompts + judge model versions are all committed to the repository. The harness is re-runnable from the repository state at any commit.
• Single-script publish. A new harness run is published end-to-end with one script (./publish.sh from the transparency directory). It copies the harness's aggregate.json into the page's data file and rsyncs the audio files for in-browser playback. No build step, no framework, no editorial layer.
• Open under NDA, then open-source. The harness code is open to HNZ under NDA during technical due diligence, with a longer-term goal of open-sourcing it (target: 12 months post-deployment) so external auditors can re-run the methodology end-to-end.
• Public corpus where it can be. v1 (synthetic) and v2 (role-play, performer-released) corpora can be open-sourced. v3 (real consent-based) recordings remain restricted due to PHI sensitivity even after PHI minimisation; aggregate per-cohort numbers are published, raw audio and transcripts are not.

Statistical methodology

Boring but defensible. Specifying these explicitly converts every report from "we ran the harness" into "we ran the harness with this method, others can reproduce".

References

• Verga, P. et al. (2024). Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. Cohere. Published basis for the panel-of-LLM (PoLL) jury pattern used in Layer B.
• Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Documents single-judge limitations the jury pattern addresses.
• Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement. The κ statistic used for jury-vs-clinician agreement.
• Fleiss, J. L. (1971). Measuring Nominal Scale Agreement Among Many Raters. Psychological Bulletin. Used for per-juror consistency.
• jiwer — open-source Python library for WER and related metrics; version-pinned in the harness lockfile.