Specialist tests, 24 October 2026

Publisher: Hanah. These are vendor-run tests. The reference transcripts are benchmark scenarios, not patient records supplied for clinical care.

Cardiology: 5 Hanah regenerated notes, 5 captured Heidi notes and 1 captured i-scribe note per case. Hanah regeneration used gpt-6.1-sol, effort low, on 24 October 2026; it is not a fresh microphone capture. The packet includes the reference, captured transcript, generated note, fact inventory and AI grading verdicts. Facts captured = 100 * (yes + 0.5 * partial) / number of inventory facts. Facts missed = no verdicts, excluding partials. Average each metric across the available runs; round the final result.

Cardiology grading instructions:

# Blind grading of one AI-scribe note

You grade ONE packet: `packets/<ID>.json` in this folder. Read only that file. Do not open any
other file or folder, do not search the filesystem, and do not try to work out which product
wrote the note. Write your result to `grades/<ID>.json` and reply with just "done <ID>".

The packet holds:
- `reference`: the verbatim ground-truth transcript of the consultation (source of truth).
- `authorContext`: what the system told the scribe (session date etc.); ground truth like the transcript.
- `note`: the SOAP note the scribe produced.
- `facts`: the consultation's fact inventory (index 0..N-1).
- `transcript` + `marks`: the scribe's own transcript, and spans `[start, end, said]` where it differs
  from the reference (character offsets into `transcript`; `start == end` means reference words
  `said` were missed; empty `said` means inserted words).

Do three separate tasks.

## Task 1: coverage (one verdict per fact)

For each fact, decide whether the note captures it:
- "yes": the note clearly and correctly captures the fact.
- "partial": the note addresses the fact but with omissions, vagueness, or minor inaccuracies.
- "no": the note fails to capture the fact, or captures it incorrectly.

Be strict but fair. Consider clinical correctness: a wrong dose or wrong side (left vs right) is
"no", not "partial". Phrasing variations that preserve clinical meaning are "yes". Judge each
fact on its own, against the reference transcript. Reason: one short sentence (≤ 200 chars),
quoting a brief note span where useful.

## Task 2: hallucinations

Find INVENTED clinical content in the note that has no support in the reference transcript.

A FABRICATION is a specific clinical claim in the note that has NO support (direct, paraphrased,
synonymous, or contextually implied) anywhere in the transcript.

THE BAR IS HIGH. Most generated documents should have ZERO fabrications. A typical clinical
document, even an imperfect one, contains many paraphrases, restructurings, and synonym
substitutions, none of which are fabrications. If you find yourself flagging more than 2 items,
you are almost certainly flagging paraphrases or framing differences. Stop and reconsider.
Before flagging ANY item, search the transcript for supporting text in any form. Only flag if the
answer is a definitive no.

DO NOT FLAG: clinical-term substitutions ("tummy pain" → "abdominal pain"); direct restatement;
reorganisation into sections; certainty-framing differences; reasonable contextual implication;
content grounded in Q&A across speaker turns; headings, labels, formatting, boilerplate, or
standard safety-net phrasing that asserts no new clinical fact; anything grounded in authorContext.

DO FLAG: a specific clinical claim (symptom, finding, history, dose, medication, side,
measurement, temporal detail) absent from the transcript in any form; wrong-dose, wrong-side,
wrong-medication or wrong-measurement assertions; an invented negation when the transcript is
silent; a fact stated as definite when the transcript conveyed only uncertainty; a new diagnosis,
finding or detail entirely absent.

For each fabrication give `claim` (short quote/summary from the note), `reason` (what makes it
unsupported), and `fact`: the index of the inventory fact this claim gets wrong, if the claim is
a wrong version of one inventory fact; otherwise null.

## Task 3: significant transcript errors

For each span in `marks`, decide whether the difference could change diagnosis, treatment,
follow-up, safety or the record (wrong number, dose, side, drug name, negation, a missed or
invented symptom or finding). Fillers, hesitations, numerals vs words, punctuation, spelling of
the same word, small function words and speaker chatter are NOT significant. Read the surrounding
text of both transcripts before deciding.

## Output: `grades/<ID>.json`

```json
{
  "facts": [["yes|partial|no", "reason"], ...],          // same order and length as packet facts
  "claims": [{"claim": "...", "reason": "...", "fact": 3 | null}, ...],
  "significantMarks": [{"i": 12, "reason": "..."}, ...]   // indexes into marks; only significant ones
}
```

The file must be valid JSON (no comments).


Separate added-interpretation pass: 66 notes over 4 cases and 9 setups. hanah-old and hanah-new are separate configurations, not separate vendors. The article reports hanah-new as Hanah. The download retains both, including 6 of 8 older Hanah notes flagged. This pass tests added reasoning only, not every kind of hallucination.

# Blind grading: added interpretation only

You grade ONE packet: `packets/<ID>.json` in this folder. Read only that file. Do not open any
other file or folder, do not search the filesystem, and do not try to work out which product
wrote the note.

The packet holds:
- `reference`: the verbatim ground-truth transcript of the consultation (source of truth).
- `note`: the clinical note a scribe produced.

## Task: added interpretation

List reasoning in the note that the clinician did not state: a link the note draws between
findings and an assessment, diagnosis, cause or plan where the reference never makes that link.
Typical wording: "X supports Y", "consistent with", "may explain", "suggests", "increases concern",
"likely due to", a rationale attached to a plan item, or a working diagnosis the clinician did not
name.

DO NOT LIST: reasoning the clinician stated (in any wording); invented facts (symptoms, findings,
doses, history) that are not reasoning; headings, structure or restatement of the clinician's own
assessment. Before listing an item, search the reference for the clinician making that link in any
wording. Only list it if the answer is a definitive no.

For each item give `text` (short quote from the note) and `reason` (why it is the scribe's own
interpretation and not the clinician's). Most notes have zero or one item; an empty list is a
normal answer.