Accuracy over inference: matched repeated-run variability analysis
Analysis date: 18 October 2026 (Australia/Melbourne)
Source: https://scribestandard.com/assets/cases/all-runs.csv
Source SHA-256: 58ce61bcb0f168673fa3f9d666bf2b6036bbe6ba98602e9959f486f00dc30097

This is a secondary analysis of published vendor-run benchmark observations, not a new capture session or an independent clinical trial.

Select rows with numeric values on all four metrics: note_errors_weiner, hallucination_flags, review_time_s, stop_to_note_s. Retain only consultations with exactly five such observations for each of the six products. This gives eight matched consultations and 40 observations per product (240 total).

For each product and consultation, calculate sample standard deviation (denominator n-1) over the five runs. Report the arithmetic mean of the eight consultation-level standard deviations, weighting consultations equally. Round only the final published values to two decimals. This measures within-consultation repeat variability, not differences between consultations. It is descriptive, without confidence intervals or a significance test. The matched subset does not establish performance on every consultation, profession, product or future version.

Review time is the benchmark's modelled reading and correction estimate, not directly observed clinician time. Stop-to-note time is measured capture latency. Note errors and hallucination flags retain the original benchmark grading definitions; consult https://scribestandard.com/methodology/.

Matched case identifiers:
day1_consultation01
day1_consultation02
day2_consultation07
day3_consultation01
day4_consultation08
hpi-pain-management
indic-hindi
myotherapy-50min

Reproduce with Python's standard library:
import csv, collections, statistics
rows = list(csv.DictReader(open('accuracy-over-inference-matched-runs.csv')))
groups = collections.defaultdict(list)
for row in rows:
    groups[row['scribe'], row['case']].append(row)
for product in sorted({p for p, c in groups}):
    for metric in ['note_errors_weiner', 'hallucination_flags', 'review_time_s', 'stop_to_note_s']:
        deviations = [statistics.stdev(float(r[metric]) for r in group)
                      for (p, c), group in groups.items() if p == product]
        print(product, metric, round(statistics.mean(deviations), 2))