Medical AI’s accuracy gains are outrunning evidence of better patient outcomes

A plethora of studies in 2026 is showing a persistent gap in medical AI: the systems can impact doctors’ choices and achieve impressive diagnostic results without demonstrating that patients actually become healthier.

This is important for hospitals purchasing these systems, companies trying to showcase their usefulness, and authorities determining which evidence should be considered once AI is adopted in doctors’ clinics.

The proof problem regulators are starting to scrutinize

The disparity between the remarkable performance of AI and proven efficacy for patients appears to be becoming too big to overlook. The Financial Times summed up this situation in its headline:

“Medical AI has a proof problem.”

This issue has now moved from the realm of academic discussions. The FDA is asking many of the same questions as it looks at how to assess these AI-enabled medical devices before and after they are put to use. The August 18 discussion paper, open for public input until 19 October, covers such issues as risk assessment, premarket assessment, and postmarket monitoring.

The FDA is careful to make clear that this paper is purely a discussion paper and not a draft guideline, a proposed change in policy or any indication of future regulatory expectations.

A previous FDA’s inquiry regarding real-world performance has also brought into consideration the issue of model drift and the limitations of using static metrics. This raises a larger question that cannot be ignored anymore: how do we measure safety and efficiency once an AI solution has moved out of the laboratory?

In Kenya, AI support did not lower treatment failure

In a study published in the Nature Medicine journal, which was released on June 26, 103 clinical officers at 16 facilities of Penda Health in Nairobi and Kiambu counties treated patients with or without the help of a large language model (LLM) for decision-making. Out of the 9,691 patients registered, 9,347 patients were considered for the primary analysis.

The rate of treatment failures after 14 days was 2.2% in the AI group and 2% in the control group so the adjusted odds ratio was 0.77, although the difference is not statistically significant (P=0.13). No serious adverse events were associated with the intervention.

The group using AI did have better documentation and more appropriate diagnoses and treatment plans. The significance of the result derives from the fact that better process measures did not result in any improvement in the outcome for patients.

The trend is similar to the RAPIDx AI trial of 2024. In the primary analysis of 3,029 patients, the composite six-month outcome of cardiovascular death, myocardial infarction, or unplanned cardiovascular readmission was 26% using AI and 26.4% using standard care. However, within the non-type 1 MI group, invasive coronary angiography was performed with 47% lower frequency with AI-supported care.

The scores keep climbing; the real-world proof lags

A multi-country randomized trial under Nicholas Rounding’s guidance found that GPT-4o improved 249 doctors’ clinical vignette performances by 18% in Kenya, 10.7% in Indonesia, and 7.2% in the Netherlands (P<0.001).

However, these were simulated cases and not real-world situations, and participants in the control group could not use the Internet or clinical protocols, nor were harms evaluated. The research confirms that AI increases performance under controlled settings but doesn’t say anything about patient outcomes.

The findings of another study published in Nature Medicine, conducted by Li Zhang, Jakob Nikolas Kather, and their colleagues, indicated an accuracy of 90.04% for a seven-disease benchmark. In this study, by introducing a consistency threshold, the on-site agent kept a total of 49.4% of cases with an accuracy level of 98.9%, whereas the rest had to be reviewed by human professionals. However, the authors indicated a need for further confirmation through trials conducted in real clinical settings.

Why a doctor ‘in the loop’ does not settle it

According to an evidence map put together by Joy Xu, Justin Ko, and Joseph Kvedar, agentic AI is now not only found in administration but also in diagnosis, management, and other healthcare tasks, thus resulting in the increasing importance of being able to govern and audit such systems.

Another npj Digital Medicine commentary states that the mere presence of a clinician does not guarantee adequate supervision and governance. Automation bias can make people more likely to accept a flawed recommendation, while effective supervision requires enough knowledge, time and authority to challenge the system and intervene.

Without those conditions, the authors warn, human oversight can become little more than a liability backstop:

Responsibility may collapse onto clinicians expected to catch errors they are not well positioned to detect or correct … a moral crumple zone.

That shifts the debate away from whether a human is technically “in the loop” and toward whether that person is actually equipped to question, override and stop the system when necessary.

The commercial stakes behind the caution

MarketsandMarkets projects the healthcare AI market to grow from $36.67 billion in 2026 to $194.79 billion by 2031, a 39.7% compound annual growth rate. If that growth materializes, hospitals will be making more AI purchasing decisions while patient-outcome evidence remains uneven.

That could shift competitive pressure toward something harder to market but more valuable to providers: prospective validation, local performance monitoring and oversight that can actually be audited.

The smartest crypto minds already read our newsletter. Want in? Join them.