The Centers for Medicare and Medicaid Services Severe Sepsis and Septic Shock Management Bundle - a set of steps hospitals are measured on, including timely antibiotics, blood cultures, lactate measurement and a 30-millilitre-per-kilogram fluid bolus.

Did patients do better?

Not measurably. There were no significant differences in intensive care unit admissions or 30-day mortality. The trial measured process compliance, with 301 patients, and was not powered for outcomes.

Is this just better documentation?

Partly, and the authors say so. The largest gap between arms was in noncompletion of the fluid bolus component, which they describe as documentation-sensitive. Whether more fluid was given or more fluid was recorded cannot be separated here.

How accurate was the language model?

Agreement between the LLM determination and expert review was 92 percent, meaning roughly one determination in twelve disagreed with a human reviewer.

Tell the doctor the same day and sepsis compliance rises 13 points

What happened

Hospitals are measured on how well they treat sepsis. In the United States the measure is the CMS Severe Sepsis and Septic Shock Management Bundle, known as SEP-1, and compliance with it is assessed by trained human abstractors reading charts, typically months after the patient has gone home.

A team at the Division of Biomedical Informatics, the Joan and Irwin Jacobs Center for Health Innovation and the School of Medicine at the University of California San Diego asked a different question. Not “can a language model do the abstraction more cheaply”, but “if the abstraction happened the same day and the physician was told, would they do anything differently”.

They randomised all 66 attending physicians across two academic emergency departments. Compliance rose from 70.1 percent to 82.9 percent.

The test

Single-blind, unstratified, cluster randomized trial, December 13, 2024 to July 8, 2025. The clusters were physicians: everyone who practised in the two UCSD emergency departments and worked more than three shifts a month before the study began.

Intervention physicians received targeted feedback from LLM-determined SEP-1 compliance at the time of patient discharge. Control physicians followed the standard process, which is to say they found out months later, in aggregate, if at all.

The primary outcome was overall SEP-1 compliance. Secondary outcomes were expert agreement with the LLM determination, 30-day mortality, and ICU admissions among patients with severe sepsis or septic shock in the emergency department. Effects were estimated with a mixed-effects logistic regression including a random intercept for physician, which is the right way to handle repeated cases by the same doctor.

The numbers

Measure	Intervention	Control
Physicians randomised	66 total
Patients meeting SEP-1 criteria	180	121
SEP-1 compliance	82.9%	70.1%
Absolute improvement	13.0 points (95% CI 2.5 to 23.4)
Odds ratio	2.10 (95% CI 1.15 to 3.81), P = .02
Fluid bolus noncompletion	3 of 180 (1.7%)	16 of 121 (13.2%)
LLM vs expert agreement	92%
ICU admission, 30-day mortality	no significant difference

The fluid bolus row carries most of the effect. Noncompletion fell from 13.2 percent to 1.7 percent, and the authors flag it as a documentation-sensitive component of the quality measure.

The result

Compliance improved by 13.0 percentage points with an odds ratio of 2.10 and a p-value of .02. Agreement between the language model and expert review was 92 percent. No significant differences existed in ICU admissions or 30-day mortality.

The authors conclude that AI-driven quality clinical integration may address limitations in existing hospital quality reporting and better support a learning health system.

How this compares to what came before

Automating quality abstraction with natural language processing is not new; the pitch has always been cost. Abstractors are expensive and slow, and several vendors sell software that reads charts instead.

What this trial changes is the purpose. If abstraction is just cheaper, it is a back-office saving. If it is fast enough to reach the clinician while the case is still fresh, it becomes a feedback loop, and feedback loops change behaviour. That is a different product with a different value proposition, and this is the first randomised evidence for it.

The comparison with audit-and-feedback literature from outside AI is also instructive. Decades of work on clinical audit show that feedback works better when it is timely, specific and individual. This trial is that literature’s recommendation implemented with a language model as the engine.

What it means

For hospitals, the implication is that quality measurement and quality improvement can be the same activity rather than two departments. The measurement that was going to happen anyway, moved forward in time and pointed at an individual, produced a 13-point change.

For anyone building clinical AI, note where the value came from. The model did not diagnose anything or recommend treatment. It read the chart and told the doctor what the chart said. That is a comparatively low-risk application with a comparatively direct effect.

The caveat is the fluid bolus. If the main thing that changed is that physicians documented the bolus they were already giving, then the measure improved and care did not. Hospitals are paid on measures, so that is still worth money, but it is not the same as better medicine and nobody should conflate the two.

Business ideas from this paper

Same-day quality feedback as a service. The core product is a pipeline that reads the chart at discharge, computes compliance against a defined measure, and emails the clinician. Every measure a hospital reports on is a candidate, and this trial gives the first randomised evidence that it changes behaviour.

Measure-agnostic feedback engines. SEP-1 is one bundle. Stroke door-to-needle, heart failure discharge, surgical prophylaxis and dozens of others are structured the same way. The engine generalises more easily than the clinical models do.

Documentation-versus-care reconciliation. Someone should build the tool that compares what the infusion pumps recorded against what the chart says. It settles the ambiguity in this trial and would be independently valuable to any hospital that is measured on documentation.

Agreement auditing. 92 percent agreement means 8 percent disagreement, and in a system paid on these measures that gap matters. A sampling and adjudication service that monitors drift between the model and human reviewers is a compliance product with a clear buyer.

What could be wrong with it

The documentation question is the big one and the authors raise it themselves. A measure that improves largely through its documentation-sensitive component has improved as a measure.

301 patients and 66 physicians at two emergency departments in a single health system. The confidence interval on the improvement runs from 2.5 to 23.4 percentage points, which is wide enough that the true effect could be modest.

No difference in ICU admission or 30-day mortality, and the trial was far too small to detect one. Readers should not take the absence of an outcome difference as evidence that care did not improve, nor as evidence that it did.

Physicians knew they were being observed and scored. Some of the effect may be the measurement rather than the feedback, and a trial of this design cannot separate those.

The LLM disagreed with expert reviewers on 8 percent of determinations, which in a payment-linked measure is not a trivial error rate.

How sure can you be?

We score this 8 out of 10.

It is a peer-reviewed single-blind cluster randomised trial in JAMA Network Open, with every eligible attending physician randomised, real CMS-defined cases, expert review of the model’s determinations, a pre-registered trial identifier, and an appropriate mixed-effects analysis.

It loses points because the work comes from a single institution, which limits how far the implementation generalises, and because the data are not publicly deposited.

The process finding is solid. Whether it represents better care or better charting is genuinely unresolved, and we would not report it as a clinical outcome.

Do this today

If you work in hospital quality, find out how long it currently takes from a patient’s discharge to the physician learning whether the bundle was completed. If the answer is months, you have the gap this trial exploited.

If you are considering an abstraction tool, ask the vendor whether it can deliver at discharge rather than at month-end. The clinical value in this trial came entirely from the timing.

If you are a clinician being scored by one of these systems, ask to see the agreement rate between the model and human reviewers for your institution. Eight percent disagreement is the number that will matter to you when a case goes the wrong way.

Source: Medical Record Abstraction for Quality Improvement in Sepsis Care Using Artificial Intelligence: A Cluster Randomized Trial, JAMA Network Open, June 2026. PMID 42348212 · doi.org ().

Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.