Accuracy over inference: your AI scribe should not fill in the blanks

핵심 요약

• Hanah explicitly instructs its drafting models to preserve the source's specificity, uncertainty, attribution and action status, without inventing clinical details.
• Our published testing includes examples of changed medication names, invented injury sides and treatment decisions nobody discussed.
• In a matched repeated-run analysis, Hanah had the lowest variability among all six tested scribes on note errors, hallucination flags, estimated review time and generation latency.
• These are vendor-run benchmark findings, not a guarantee that any individual note is error-free. The source outputs and calculation are linked below.

I want an AI scribe to write a clear note. I do not want it to get creative with what happened in the consultation.

That distinction sounds obvious until you read enough generated notes. A model recognises a familiar clinical pattern and finishes the story. A possible diagnosis becomes a diagnosis. A medication gets a dose nobody said. A plan acquires the counselling, investigations and referrals that would usually accompany it.

The result can read beautifully. It can also be wrong.

At Hanah, we deliberately prioritise accuracy over inference. The model can organise the information and write it clearly. It cannot treat clinical plausibility as permission to add facts.

A plausible detail is not a documented fact

A clinician may infer something while assessing a patient. That is their clinical judgment. A documentation system has a different responsibility: faithfully record what the encounter and its supplied sources establish, including the clinician's assessment and its uncertainty.

“Possible” matters. So do “reported,” “planned” and “not yet confirmed.” Removing those words can change the record.

Some models are simply bad at respecting that boundary. They are good at producing the sort of text that usually follows a particular presentation. That ability is useful for creative writing. In clinical documentation, filling a gap with the most likely detail is a failure.

Here are examples from the recorded test scenarios in our published benchmark. These are benchmark scenarios, not patient case reports. The outputs were captured on 6 October 2026; they describe those runs, rather than every output a product will generate.

“Recent apixaban” does not mean “recently commenced”

Our ICU handover recording includes the phrase “recent apixaban.” It does not establish when the medication was started.

In the captured Lyrebird assessment, that became “apixaban (recently commenced).” It is a small addition with a specific meaning: the patient recently started the medication. The recording never established that.

Hanah's assessment retained “Recent apixaban.” Keeping the original level of specificity is the correct documentation behaviour. The scribe does not need to resolve an ambiguity that the clinician has not resolved.

The same recording names empagliflozin and describes possible euglycaemic ketoacidosis. PatientNotes' captured output substituted dapagliflozin. A related medication is still the wrong medication. Hanah retained the named drug and the uncertainty.

You can inspect the reference transcript, product outputs and grading reasons.

These are different kinds of errors. The first expands an ambiguous phrase. The second substitutes a named drug. Both make the final record say something the source did not establish. An output alone does not tell us whether the cause was transcription, retrieval or the drafting model, so our tests evaluate the finished result as well as the transcript.

An injury does not acquire a side because the template wants one

In our quad-bike accident scenario, the patient describes landing on their shoulder and hip. No side is stated.

Lyrebird's captured note assigned the injuries to the right shoulder and hip. PatientNotes assigned them to the left. The source supports neither.

PatientNotes also added “No immediate imaging or testing required at this time” and “No specialist referral indicated currently.” Those decisions were not discussed in the recording. They may sound like reasonable lines in a musculoskeletal note. That does not make them part of this encounter.

The recording also contains a compound question about a head knock, loss of consciousness and feeling dazed. The patient replies that they did not black out and felt briefly rattled. That answer must not be expanded into a blanket denial of every part of the question.

Read the quad-bike transcript and captured notes.

Hanah's captured notes were not perfect either. The published grading flags a misattributed intention to document the incident and garbled wording in the advice. I want those failures visible too. Accuracy is something we test and improve, not an absolute we declare about ourselves.

What we explicitly tell Hanah's models

Our current note-generation instructions make these boundaries explicit. They include:

Completeness means retaining relevant information from the source, not filling gaps with clinically typical details.

They also instruct the model to:

• Match the source's specificity. Do not infer medication strength, dose, frequency, route, named investigations or anatomical detail.
• Preserve attribution. Patient-reported symptoms stay reported; findings belong in Objective only when explicitly observed or measured by the clinician.
• Require support for negatives. An unanswered question is not a denial, and answering one part of a compound question does not establish every negative.
• Preserve action status. Proposed, agreed, ordered and completed are different. An intention to examine is not evidence that the examination happened.
• Keep the plan faithful. Do not add routine counselling, safety-net advice, referrals, tests or treatment simply because they would be medically reasonable.
• Keep laterality unchanged. Never supply a side that was not stated, and preserve corrections the patient makes to their account.

Those are instructions, not a claim that a model will obey perfectly. We still evaluate the generated output against its sources.

We also allow supported synthesis: the model can briefly connect documented findings to the stated assessment. It must preserve tentative interpretation as tentative and must not use that reasoning to invent history, examination findings or agreed actions. Clear clinical writing requires understanding. It does not require manufacturing missing facts.

Model choice matters more than a polished demo

We evaluate models on the documentation job we actually need them to do. A model that writes impressive prose but repeatedly strengthens uncertain claims is a poor fit. So is one that sounds clinically fluent while dropping the numbers, negatives or qualifiers that make the note accurate.

A good score on a general model leaderboard does not settle that choice. We care about the actual transcript, the instructions, the resulting note and the errors. We test the complete system; we do not assume that a larger or more fashionable model will behave better on our task.

The same rule applies when a guideline is attached. A guideline can establish what is recommended. It cannot establish that an examination was performed or that the clinician agreed to a treatment. Evidence should make a note more defensible, not give the model more material from which to invent the encounter.

Consistency matters: the same recording should not be a lottery

A single good run is not enough. Clinicians need dependable output across repeated encounters, and repeated tests of the same recording help expose variability.

I checked the public repeated-run benchmark export on 18 October 2026. To compare like with like, I selected the eight consultations with five complete observations for every product on all four measures below: 40 observations per product, across six products.

For each consultation and product, I calculated the sample standard deviation across its five runs, then averaged those eight deviations. Lower values mean less run-to-run variation on that measure. We compare repetitions of the same consultation before averaging; we do not confuse differences between easy and difficult consultations with instability.

Hanah had the lowest run-to-run variability on all four measures among every scribe in this matched comparison. This supports a specific claim about note quality and generation performance in these tests. It does not mean Hanah is least variable on every possible metric: transcription word-error variability is a separate measure, and Hanah did not lead it across the full available repeated-case data.

Note errors include omitted or partly captured facts and unsupported claims under the benchmark's scoring method. Review time is a modelled estimate of reading and correction effort, not a stopwatch measurement of clinicians. Generation latency is the measured interval from stopping to the prompted note. The dates and scoring assumptions belong to the underlying benchmark; this article adds a variability analysis rather than reporting a new capture session.

Download the matched observations, per-consultation deviations and calculation method. The broader benchmark includes other consultations and incomplete observations; they are excluded from this matched calculation. See the benchmark methodology for capture and grading details. These are vendor-run comparisons, not independent clinical trials.

The standard I want us held to

Write clearly. Preserve the important detail. Keep uncertainty intact. Record what the clinician actually decided. Make the source easy to inspect.

I would rather Hanah leave an unsupported detail out than confidently complete the story. A note can be concise, well organised and clinically useful without being creative about the facts.

That is what accuracy over inference means to me. It is why we publish the recordings, outputs and grading reasons, including our own mistakes. You should be able to check whether our behaviour matches our words.

자주 묻는 질문

Does accuracy over inference mean the AI cannot summarise or explain?

No. Hanah can organise information, summarise it according to the chosen detail level and connect documented findings to the stated assessment. The boundary is adding unsupported facts or strengthening a tentative statement into a definite one.

Does an explicit instruction prevent every invented detail?

No. Instructions are one part of the system. We test the resulting transcripts and notes, publish failures and expect clinicians to review every draft before filing it.

What does the lowest-variability result cover?

It covers the four measures in this article, using eight matched consultations with five complete observations per product for Hanah, Heidi, Lyrebird, Freed, CliniScripts and PatientNotes. It does not cover every metric, every competitor or every clinical setting. The linked downloads contain the observations and calculation.