AI agents & MCP/Kenya; United Kingdom
9,691 patients in 16 Kenyan clinics: the language model was safe and changed nothing
Nature Medicine, 16 Kenyan facilities, 9,691 patients, 103 clinical officers. Large language model assistance was safe and did not reduce treatment failure.
A pragmatic cluster-randomised trial across 16 primary care facilities in Kenya gave 103 clinical officers an electronic medical record with or without large language model assistance. Across 9,691 patients, treatment failure within 14 days was 2.2 percent with the model and 2.0 percent without, adjusted odds ratio 0.77 with a confidence interval of 0.55 to 1.08. No safety signal was found.
What happened
Researchers from the Kenya Medical Research Institute-Wellcome Trust Research Programme in Nairobi, Keprecon, and the London School of Hygiene and Tropical Medicine ran a trial that the field has needed for three years. They took a large language model, wired it into the electronic medical record used by clinical officers in Kenyan primary care, randomised which officers got it, and counted what happened to their patients.
9,691 patients were enrolled between April 22 and July 16, 2025, overseen by 103 clinical officers across 16 facilities. The result is published in Nature Medicine.
Treatment failure within 14 days occurred in 2.2 percent of patients whose clinician had the model and 2.0 percent of patients whose clinician did not.
The test
This was a pragmatic, cluster-randomized trial. Clinical officers, not patients, were randomised: 52 to the LLM-assisted arm and 51 to control. Randomising the clinician rather than the patient is the right choice here, because a clinician who has been using an AI assistant all morning cannot un-learn it for the next patient.
The primary outcome was an expert-adjudicated composite of treatment failure events experienced within 14 days of enrollment. The adjudication is the important word. The people deciding whether a case counted as a failure were not the clinicians who treated it.
Safety was tracked separately, with independent review of adverse events.
The numbers
| Measure | LLM-assisted | Control |
|---|---|---|
| Clinical officers | 52 | 51 |
| Patients analysed | 4,693 | 4,654 |
| Treatment failures at 14 days | 102 (2.2%) | 94 (2.0%) |
| Adjusted odds ratio | 0.77, 95% CI 0.55 to 1.08, P = 0.13 | |
| Serious adverse events related to intervention | none | |
| Facilities | 16 primary care facilities in Kenya | |
Read the odds ratio carefully. The point estimate of 0.77 is in the direction of benefit. The confidence interval runs from 0.55 to 1.08. That means this trial is compatible with the model cutting treatment failure by nearly half, and equally compatible with it making things slightly worse. It does not settle the question; it narrows it.
The result
The primary outcome did not differ significantly between groups. No serious adverse events were judged related to the intervention, and independent review of the adverse events did not identify a safety signal.
The authors’ own conclusion is unusually direct: LLM assistance was safe but did not reduce treatment failure within 14 days, and any benefit, if present, is probably modest.
How this compares to what came before
Until this trial, the case that large language models help clinicians rested almost entirely on two things. First, vignette studies: give a model a written case and compare its answer to a doctor’s. Second, retrospective benchmarks on exam questions. Both consistently showed the models performing well, sometimes better than clinicians.
Neither tells you what happens when a tired clinical officer at the end of a long queue has a suggestion box on the screen. This trial does, and the answer is that the suggestions did not change what happened to patients in the following fortnight.
It is also one of a very small number of large clinical AI trials conducted outside high-income health systems. A recent review of 4,667 medical AI studies found that the top ten contributing countries accounted for 75.5 percent of contributions, with the United States and China alone making up 47.5 percent. Evidence from a Kenyan primary care system is scarce and disproportionately useful.
What it means
Two things, and they pull in different directions.
The null result should temper claims about LLMs transforming care delivery in low-resource settings. The transformation did not show up in the outcome that matters most, in the setting where it was most loudly promised.
The safety result should temper the opposite claim. The widely voiced fear was that a model confidently suggesting the wrong thing would hurt people, particularly where supervision is thin. Across 4,693 patients, independent review found no safety signal. That is not proof of safety at scale, but it is the best evidence anyone has.
For anyone deploying these tools, the practical reading is that the technology is safe enough to keep testing and not yet good enough to justify a purchase on clinical outcomes alone. If you are buying it, buy it for something else – documentation time, referral quality, training – and measure that instead.
Business ideas from this paper
Outcome-measurement infrastructure for clinical AI pilots. The expensive part of this trial was not the model. It was enrolling 9,691 patients, following them for 14 days and adjudicating outcomes with experts. A service that makes that follow-up cheap and repeatable would let far more health systems test tools before buying them.
Narrower, higher-frequency endpoints. Treatment failure at 2 percent is a statistically unforgiving target. There is a product in instrumenting the more frequent things a model might improve – prescribing appropriateness, test-ordering, documentation completeness – so a pilot can read out in weeks rather than needing thousands of patients.
African clinical evidence as a business. The scarcity of trials in low- and middle-income settings is a market gap. Research partnerships that can run pragmatic trials in these systems are commercially valuable to any vendor that wants to sell there credibly.
Safety monitoring as a separate layer. The independent adverse event review in this trial is a function most deployments do not have. Packaging it – an independent reviewer looking at flagged cases from a deployed clinical AI – is a defined service with a clear buyer.
What could be wrong with it
The trial was not powered to rule out a moderate benefit. An odds ratio of 0.77 with an upper bound of 1.08 means a 45 percent reduction in treatment failure remains inside the data. Calling this a negative trial would be wrong; it is an inconclusive trial on benefit and a positive one on safety.
Treatment failure was rare – roughly two percent in both arms – which gives any intervention very little headroom.
The window was 14 days. A tool that improves chronic disease management or catches something that matters three months later would not appear in this outcome.
The trial measured one composite outcome. Diagnosis quality, clinician workload, patient experience and cost were not the primary endpoints and cannot be inferred from this result.
Full electronic health record extracts are not publicly available, though the study protocol and statistical analysis plan are deposited on Zenodo.
How sure can you be?
We score this 9 out of 10.
It is a peer-reviewed pragmatic cluster-randomised trial in Nature Medicine, with named institutions in Kenya and the United Kingdom, run across 16 independent facilities, with expert adjudication of the primary outcome and independent adverse event review. It is registered with the Pan-African Clinical Trials Registry.
It loses a point because the trial data themselves are not openly deposited – the protocol and analysis plan are, which is better than most, but not the same thing.
The safety conclusion is strong. The efficacy conclusion is honestly reported as inconclusive by the authors themselves, and we report it the same way.
Do this today
If you are piloting an LLM in a clinical setting, write down your primary outcome before you start, and check whether it is common enough to detect a change with the number of patients you will see. If your event rate is two percent and you will see 500 patients, you will learn nothing.
If you are a health system in sub-Saharan Africa being offered an AI clinical assistant, ask the vendor to point at this trial and explain why their product would do better. The answer will tell you whether they have read the evidence in their own field.
If you are building these tools, the safety result is the one to take seriously. It is permission to keep going, not a claim of benefit, and the difference between those two is the whole argument.
Source: Generative AI-enabled clinical decision support system in primary care: a pragmatic, cluster-randomized trial, Nature Medicine, 2026. PMID 42362867 · doi.org ().
Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.
- The trial was run in Kenya by KEMRI-Wellcome and LSHTM, which makes it one of the few large clinical AI trials not conducted in a high-income setting.
- No significant benefit on the primary outcome, and no safety signal either - both findings matter.
- The confidence interval runs from 0.55 to 1.08, so a meaningful benefit is not ruled out; the trial is not powered to exclude one.
- Treatment failure was about 2 percent in both arms, which leaves very little room for any tool to improve things.
This is a pragmatic trial in real facilities with real patients and expert-adjudicated outcomes, not a vignette study. The technology is deployable; the benefit is unproven.
Questions people ask
Does this mean LLMs do not help doctors?
It means that in this setting, with these clinical officers, over this 14-day window, the model did not reduce treatment failure. The confidence interval of 0.55 to 1.08 leaves room for a real benefit the trial was too small to detect.
Was it dangerous?
No serious adverse events were judged related to the intervention, and independent review of adverse events did not identify a safety signal. That is a meaningful finding on its own, because safety was the main open question.
Why does it matter that it was in Kenya?
Most clinical AI evidence comes from the United States and China. This trial tests the tools in a system with different staffing, different disease patterns and different record infrastructure, which is where the technology was most loudly promised to help.
What was the primary outcome?
An expert-adjudicated composite of treatment failure events experienced within 14 days of enrolment. Adjudication by experts, rather than by the clinicians themselves, is what makes the outcome credible.
Mark Alex
Mark Alex is the founder and Managing Director of Real Biz Digital, a technology company operating out of Nairobi since 2018. He works in agentic AI and the Model Context Protocol, AI governance, enterprise software architecture and cybersecurity. He holds an MSc in Mechatronical Engineering from Obuda University in Budapest and a BSc in IT, Forensic Technology and Cybercrime, from USIU-Africa in Nairobi, and has published IEEE conference research on an AI-powered digital twin for greenhouse systems. He is the author of seven books. Between 2020 and 2024 he mentored more than 200 university students and interns in Nairobi. He writes every Just Out Tech article from the original research paper.