The bot answered in 30 seconds instead of six hours, and was wrong slightly more often

What happened

After knee surgery you go home with a rehabilitation protocol, a wound to watch and a list of things that might go wrong. When you have a question – is this swelling normal, should this hurt this much, can I do this exercise yet – you message the hospital. In China that message usually goes through WeChat, and you wait.

Researchers at the First Affiliated Hospital of China Medical University, Shengjing Affiliated Hospital and Shenyang the Tenth People’s Hospital put a GPT-4 agent at the other end of that conversation for half of their patients. 261 patients were randomised, 140 to the AI agent and 121 to routine physician communication.

The agent replied in half a minute. The doctors replied in just under six hours.

The test

Randomised controlled trial, registered with the Chinese Clinical Trial Registry in April 2025. Patients in the intervention arm interacted with a GPT-4-based WeChat agent delivering real-time, context-aware support. The control arm received routine physician communication through the same platform.

The problems the trial set out to address are specific and familiar to anyone who has run a postoperative service: poor patient adherence to rehabilitation protocols, insufficient monitoring of wound healing, inadequate pain control, and limited access to timely psychological and functional support.

Outcomes were tracked at 1, 3 and 6 months, using the International Knee Documentation Committee score for knee function, a physical health component score, and patient satisfaction. Response time and answer accuracy were measured directly.

The numbers

Measure	GPT-4 agent	Doctor-led
Patients	140	121
Mean response time	0.5 +/- 0.6 min	358 +/- 47.5 min
Answer accuracy	93.9%	98.1%
Knee function (IKDC) at 1 and 3 months	significantly better with AI, P < 0.05
Physical health component at 1 and 3 months	significantly better with AI, P < 0.05
Overall satisfaction at 1 and 3 months	significantly better with AI, P < 0.05
All outcomes at 6 months	no significant difference, P > 0.05

The response time row is not a marginal improvement. It is a more than 700-fold difference, and it is the intervention. Everything else in this trial follows from the fact that somebody answers immediately.

The result

The AI responded far more rapidly and provided feedback of higher perceived quality, though with slightly reduced accuracy. At 1 and 3 months the AI group achieved significantly better outcomes in knee function, physical health and overall satisfaction. By the 6-month follow-up, group differences were no longer significant.

The authors conclude that the GPT-4-enabled WeChat agent may provide short-term benefits in postoperative functional recovery and patient experience, while long-term outcomes remain comparable to doctor-led care, supporting the value of LLM-based tools as a supplementary component of postoperative management.

How this compares to what came before

Most published work on medical chatbots measures whether the model gives good answers to written questions, usually scored by clinicians reading transcripts. Those studies have repeatedly found that models produce answers rated as accurate and, notably, as more empathetic than physicians’ replies.

This trial takes a different cut. It puts the model in the actual channel patients already use, randomises it, and measures recovery rather than answer quality. That lets it isolate something the transcript studies cannot: the value of availability as distinct from the value of correctness.

The answer is that availability was worth a measurable improvement in function and satisfaction for three months, and that correctness – where the doctors were better – did not visibly cost the AI group anything over the same period.

What it means

The deployable insight is narrower and more useful than “AI helps patients”. It is that in postoperative care, the clinically relevant variable may be latency.

Six hours is not negligence; it is a normal clinical response time for a non-urgent message. But a patient who is worried at 10pm about whether their knee should look like that will either wait, worry, or go to an emergency department. An immediate, mostly-correct answer defuses that.

That framing also tells you where the risk sits. If the value is speed, then the system can be shallow and still work, provided it reliably escalates the small number of messages where being wrong matters. The 4.2 percentage point accuracy gap is tolerable if those errors are in the reassurance category and intolerable if any of them were in the escalation category. This trial does not say which.

The disappearing benefit is worth taking at face value rather than as a failure. Postoperative support is supposed to be temporary.

Business ideas from this paper

Latency as the metric for patient messaging. Hospitals measure response time poorly or not at all. Instrumenting it, and reporting it per service, creates the baseline against which any messaging tool can be justified.

Escalation-first design. The product that matters here is not the answerer, it is the triage layer that recognises the handful of messages that must reach a human fast. That is a smaller, more testable model and a more defensible one.

Postoperative support as a bounded product. The benefit lasted three months and then stopped. A tool scoped to the first 90 days after surgery is easier to validate, easier to sell and easier to withdraw than an open-ended clinical assistant.

Error taxonomy tooling. This trial reports an accuracy percentage and no error categories. A review pipeline that classifies model errors by clinical consequence would turn a percentage into a safety case, and every deployment needs one.

What could be wrong with it

The AI was significantly less accurate than the doctors and the paper does not report what the inaccurate answers were. That is the most important missing piece in the study.

The benefit had vanished by six months, so nothing durable was demonstrated.

Patients cannot be blinded to whether they are messaging a bot, and satisfaction scores in particular are vulnerable to that. Some of the effect may be novelty or the simple pleasure of a fast reply.

261 patients across one hospital group in one city, in one surgical specialty.

The control arm’s 358-minute response time reflects that hospital’s staffing. A service that already replies within the hour would have far less room to improve.

How sure can you be?

We score this 8 out of 10.

It is a peer-reviewed randomised controlled trial in npj Digital Medicine, prospectively registered with the Chinese Clinical Trial Registry, with three named institutions and six months of follow-up on patient-reported and functional outcomes.

It loses points because there is no independent site or external adjudication, and the data are not publicly deposited – the authors cite informed consent agreements.

The response-time and outcome findings are well supported. The accuracy figure is reported without the detail needed to judge its significance, and we would not deploy on the strength of it.

Do this today

If you run a postoperative service, measure your actual median response time to patient messages. That number is the size of the opportunity this trial identified, and most services do not know it.

If you are building a patient-facing agent, build the escalation classifier before the answer generator. This trial suggests most of the clinical value comes from speed and most of the clinical risk comes from the small number of messages where speed is not the point.

If you are evaluating one, ask for the error taxonomy, not the accuracy percentage. A 93.9 percent accurate system is either safe or dangerous depending entirely on the remaining 6.1 percent.

Source: A randomized controlled trial of a WeChat-based artificial intelligence agent for postoperative care in orthopedic patients, npj Digital Medicine, January 2026. PMID 41548028 · doi.org ().

Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.