4,667 medical AI studies, 113 trials, and two countries wrote half of them

What happened

Every week brings another paper reporting that a model matched or beat clinicians at some task. If you follow the field closely it can feel settled. A team at the Chinese Academy of Medical Sciences and Peking Union Medical College, the National Cancer Center and the Xiang An Biomedicine Laboratory decided to count.

They took 218 systematic reviews published between September 2023 and September 2024, extracted the 4,667 primary studies underneath them, and classified every one by research stage, geography, specialty and study design.

88.2 percent were preclinical. 2.4 percent were randomised controlled trials.

The test

A scoping review maps a literature rather than pooling its results. The method here was to work downwards: start from published systematic reviews, which have already done the searching, and classify the primary studies they contain.

That approach has a known trade-off. It inherits whatever the 218 reviews included and excluded, and it misses primary studies that no review has yet picked up. In exchange it covers a very large base of studies quickly and consistently.

The classification covered four axes: research stage, geography, clinical specialty and study design, with an additional appraisal of methodological quality among the randomised trials.

The numbers

Measure	Finding
Systematic reviews surveyed	218
Primary studies identified	4,667
Preclinical	88.2% (4,114/4,667)
Randomised controlled trials	2.4% (113/4,667)
Top 10 countries, share of contributions	75.5%
United States and China combined	47.5%
Most common disease area	neoplasms, 32.5% (1,518/4,667)
Most common disease area among RCTs	digestive, 38.1% (43/113)
RCTs that were single-centre	67.3% (76/113)
RCTs reporting no guideline adherence	71.7% (81/113)
RCTs reporting favourable outcomes	82.3% (93/113)

The result

Medical AI research remains heavily skewed toward early-stage development, with limited high-quality clinical evidence. That is the authors’ conclusion and the numbers support it without strain.

Three patterns are worth separating.

Stage. Nearly nine studies in ten never reached a clinical comparison at all. The dominant activity in this field is model development and retrospective validation.

Geography. Three quarters of contributions came from ten countries, and nearly half from two. A technology promoted most loudly as a way to extend care where clinicians are scarce is being developed and tested almost entirely where they are not.

Quality. Among the 113 trials, two thirds were single-centre, seven in ten reported no adherence to any reporting guideline, and methodological concerns were common – particularly in allocation concealment and blinding, the two things that most protect a trial from its investigators’ hopes.

And 82.3 percent reported favourable outcomes. In a field where trials were well designed and hypotheses genuinely uncertain, you would not expect four in five to come out positive.

How this compares to what came before

Individual commentaries have made this argument for years, usually citing a handful of examples. Systematic reviews of specific specialties have reported similar imbalances within their own areas.

What this adds is scale and a single denominator. 4,667 studies classified consistently gives a number that can be quoted, compared and repeated in two years. It also lets the geographic and methodological patterns be seen together rather than one specialty at a time.

The favourable-outcome rate is the finding most likely to be cited going forward, because it points at something the other numbers cannot: not just how little trial evidence exists, but how selectively the trial evidence that exists may have been published.

What it means

For anyone evaluating a medical AI claim, this gives a prior. If a vendor cites studies, the base rate says those studies are probably retrospective and probably single-centre. Asking which of the cited work is a multicentre randomised trial is a question with a known likely answer.

For health systems outside the US, China and the other top contributors, the geographic concentration is a direct warning. A model developed and validated in two health systems carries their disease prevalence, their documentation habits and their population characteristics inside it.

For researchers, the reporting-guideline figure is the most fixable thing on the list. CONSORT-AI exists. Using it costs nothing and would make the next count look materially better.

Business ideas from this paper

Evidence appraisal as a procurement service. Health systems buying clinical AI need someone to read the cited literature and classify it the way this paper does. That is a repeatable, scoped engagement with an obvious buyer and almost no supply.

Multicentre trial coordination for small vendors. Two thirds of existing trials are single-centre because running multicentre studies is hard and expensive for a startup. A shared trial network that lets several vendors run properly designed studies across the same sites would lower that barrier.

Local validation services outside the top ten countries. If the evidence base is concentrated in two countries, every other health system needs its own validation before deployment. That is recurring, defensible work and it has to be done locally.

Reporting-guideline tooling. Seven in ten trials reported no guideline adherence. A checklist tool integrated into trial design software would raise the floor cheaply, and journals would have an interest in promoting it.

What could be wrong with it

This is a review of reviews. It inherits the inclusion and exclusion decisions of 218 systematic reviews, and anything not yet covered by a review is invisible to it.

The window is twelve months of review publication, which lags the primary literature by a year or more. The field moves quickly and the 2.4 percent figure may already be dated.

Counting randomised trials as a share of all studies penalises a young field by design. A field where every study was a trial would be a field that had stopped developing anything new. The right comparison is against other medical technologies at similar maturity, which this paper does not make.

The 82.3 percent favourable-outcome figure is consistent with publication bias but also with researchers choosing to trial only what already looked promising, which is rational. The design cannot distinguish them.

How sure can you be?

We score this 9 out of 10.

It is a peer-reviewed scoping review in npj Digital Medicine, with named institutions at the Chinese Academy of Medical Sciences and Peking Union Medical College, the National Cancer Center and the Xiang An Biomedicine Laboratory. It synthesises 218 independent systematic reviews and 4,667 primary studies, and the study-level deidentified dataset with its data dictionary is publicly deposited on Zenodo.

It is not a trial, so it scores two rather than three on evidence type under our rubric. Everything else is in place, including the open data that most papers in this field do not provide.

Do this today

If you are assessing a medical AI product, ask for the list of studies behind it and sort them into retrospective, single-centre trial, and multicentre trial. The sort takes an hour and the shape of the pile is the answer.

If you are publishing medical AI research, check your manuscript against CONSORT-AI before submission. Seven in ten trials in this survey did not, and it is the cheapest quality improvement available to the field.

If you work in a health system outside the main contributing countries, treat local validation as mandatory rather than optional. This paper quantifies how little of the evidence base was generated anywhere near you.

Source: A quantitative analysis of global AI medical studies: gaps in randomized controlled trials, npj Digital Medicine, 2026. PMID 42062529 · doi.org ().

Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.