The AI scribe wrote better letters, saved everyone time, and made twice as many medical errors

What happened

AI scribes listen to the consultation and write the note. They have been adopted faster than almost any other clinical AI product, on the strength of vendor time-saving figures and clinician satisfaction surveys, and the adoption curve has run well ahead of the evidence.

The authors of this paper state the position plainly: to date, no randomised controlled trials had rigorously assessed the proposed benefits of AI scribes.

So they ran one. At the University of Saskatchewan, 304 patients in a single-surgeon adult reconstruction practice were allocated to conventional documentation – dictation transcribed by the office administrator, or preconstructed templates – or to an AI scribe. 140 patients in the conventional group, 139 in the AI group.

The time saving was real. So was an error count that the paper reports and the summary does not dwell on.

The test

Timing data were recorded from when the surgeon entered a consultation room to when they began with the next patient, which captures the full consultation cycle rather than just typing time. Additional data covered administrative processing – transcription and faxing of medical letters.

All documentation in both groups was reviewed for medical errors by the surgeon.

Grammar, spelling and wrong wording were assessed by unblinded researchers. Letter quality was independently assessed using ChatGPT, focusing on grammar, clarity and completeness.

The numbers

Measure	AI scribe	Conventional
Patients	139	140
Physician processing time	significantly reduced with AI, P < 0.0001
Administrative processing time	significantly reduced with AI, P < 0.0001
Spelling and grammatical errors	no significant difference
Letter completeness, professionalism, clarity	significantly higher with AI
Medical errors	19	9
Patient experience	not negatively affected; most found it helpful

The result

The efficiency case holds. Both physician and administrative processing times fell significantly, at p below 0.0001 for each. Letters produced using AI scribes were rated as significantly more complete, professional and clear. Patients were not negatively affected and the vast majority found it helpful.

Spelling and grammar were no different between the groups.

And there were 19 medical errors in the AI scribe group compared to nine in the conventional group.

The authors conclude that AI scribes can effectively enhance clinical efficiency and reduce administrative workload without compromising documentation quality or patient safety.

That conclusion and that error count are both in the paper. We would report the count and let readers weigh it.

How this compares to what came before

The existing evidence for AI scribes is observational: before-and-after studies of clinician burnout scores, time-in-notes measured from electronic health record telemetry, and satisfaction surveys. Those studies consistently report large time savings and improved clinician wellbeing, and they are the basis on which most purchases have been made.

None of them randomised. Before-and-after designs in a busy practice are vulnerable to everything that changes at the same time – a new template, a new administrator, a seasonal change in case mix.

This trial confirms the central observational finding under randomisation, which strengthens it considerably. It also does something the observational studies could not: it had the surgeon review every note in both arms for medical errors, which gives a comparison nobody had.

What it means

The time saving is now evidence rather than marketing, and that is a genuine advance. For a clinic weighing an AI scribe, the efficiency case is the best-supported claim in clinical AI right now.

The error finding needs a cooler look than it has been getting. Nineteen against nine, out of 304 patients, is a difference of ten errors in absolute terms. With numbers that small, chance can move them. But the direction is the one that matters, and the conclusion that documentation quality and patient safety were not compromised is doing a lot of work on a doubled error count.

The deeper problem is that nobody is grading severity. A wrong laterality and a wrong date are both one error. Until a trial reports what the errors were, neither reassurance nor alarm is justified, and both are currently being offered.

There is also a methodological point worth flagging for the field. Letter quality here was assessed by ChatGPT. Using a language model to grade language model output is a known bias risk: models tend to prefer text that resembles their own. The grammar and spelling comparison, done by humans, found no difference.

Business ideas from this paper

Error grading for scribe output. The missing product is a review layer that samples AI-generated notes and grades errors by severity, not count. Any practice running a scribe needs this and none have it.

Specialty-specific scribe tuning. This trial ran in adult reconstruction orthopaedics. Templates, terminology and the things that go wrong differ sharply by specialty, and tuning plus validation per specialty is a defensible service.

Verification workflows. If the time saving comes from the clinician not writing the note, the risk comes from the clinician not reading it either. A structured review step that takes thirty seconds and catches the categories of error that actually occur would protect most of the saving.

Independent benchmarking. Scribe vendors currently self-report accuracy. An independent benchmark with human-graded errors across several specialties would be bought by health systems and resisted by vendors, which is usually a sign it is needed.

What could be wrong with it

One surgeon, one subspecialty, 304 patients. The result describes a single person’s workflow, templates and dictation habits, and the whole point of randomising within one practice is that it controls for clinician variation – at the cost of telling you nothing about other clinicians.

The error counts are small enough that random variation could explain the difference. They are also the only randomised error comparison anyone has published.

Grammar, spelling and wording were assessed by unblinded researchers who knew which arm each letter came from.

Letter quality was scored by ChatGPT, which is a weak and potentially biased instrument for this purpose.

The trial does not report error severity, which is the single piece of information needed to interpret its most consequential finding.

How sure can you be?

We score this 7 out of 10.

It is a peer-reviewed randomised controlled trial in The Journal of Arthroplasty, from a named institution – the College of Medicine and Division of Orthopaedics at the University of Saskatchewan – and it is, as the authors say, the first randomised test of claims that have been driving purchases for two years.

It loses points for the single-centre single-surgeon design, the absence of open data, and the lack of any independent site.

The time saving we would report with confidence. The safety conclusion we would not, in either direction.

Do this today

If your practice uses an AI scribe, pull twenty notes at random and read them against what you remember of the consultation. That is the review this trial did systematically and almost no practice does at all.

If you are buying one, ask the vendor for a randomised comparison rather than a before-and-after study. There is now exactly one in the literature, and knowing whether they have read it tells you something.

If you are a clinician signing AI-generated notes, remember that you are signing them. The time saving in this trial is real and it comes from not writing. It does not come from not reading.

Source: Evaluating the Impact of Artificial Intelligence Scribes on Clinical Workflow and Documentation Quality: A Randomized Controlled Trial, The Journal of Arthroplasty, April 2026. PMID 41935758 · doi.org ().

Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.