Randomized trials are exposing the gap between impressive medical AI reasoning and better care, putting workflow design and professional accountability under scrutiny.
A physician deciding what to do next must weigh symptoms, incomplete records, clinical guidelines and a patient’s circumstances. Artificial intelligence can supply another opinion in seconds. What remains uncertain is whether that opinion improves the decision, rather than simply making it easier to produce.
That gap is consequential for hospitals weighing software purchases. A tool may look impressive in a controlled demonstration while failing to change a busy clinician’s behavior at the bedside.
New clinical studies point in different directions. Some show that doctors reason more accurately with carefully structured AI assistance. Others find little difference in patient outcomes. For medical technology companies, the challenge is no longer demonstrating that a model knows medicine; it is proving that people can use its advice safely under real clinical conditions.
The Sequence of Advice Matters
In a 2026 randomized trial published in npj Digital Medicine, researchers led by Selin S. Everett evaluated 70 US physicians working through clinical case scenarios. Clinicians using conventional resources scored 75% on diagnostic reasoning assessments. Scores rose to 85% when an AI system offered its assessment first and 82% when clinicians formed an opinion before receiving AI feedback.
The experimental system did more than issue a diagnosis. It integrated clinicians’ and AI assessments, identified agreements and disagreements, and proposed next steps. Across overall scores, the two workflows were not statistically different; the first-opinion approach performed better on a narrower measure of actionable decisions.
That distinction matters for product design. Presenting AI recommendations before a physician starts reasoning may broaden possibilities, but can also influence what receives attention. Presenting them afterward creates a different opportunity: checking an existing conclusion against alternatives.
Neither approach was tested on actual patients. The results came from structured vignettes, and the authors cautioned against treating them as evidence of improved clinical outcomes.
Better Models Do Not Automatically Make Better Teams
A 2024 randomized trial led by Ethan Goh reached a less encouraging conclusion. Among 50 physicians, those given access to a large language model scored 76% on diagnostic reasoning, versus 74% for colleagues using conventional resources. The difference was not statistically significant.
Yet the model working alone outperformed the conventional-resource physicians. Its capabilities did not reliably translate into a stronger human-machine team.
The contradiction with the newer trial is instructive, not necessarily evidence that one study is wrong. Models, interfaces, instructions and measures differed. A capable algorithm must still communicate useful evidence at the moment a clinician can evaluate it.
This introduces two competing design risks: automation bias, when users defer excessively to generated suggestions, and underuse, when valuable recommendations are ignored or poorly integrated. The trials illuminate workflow sensitivity, but do not establish the prevalence of either behavior in hospitals.
Controlled Gains Meet Different Clinical Realities
A September 2026 multinational randomized trial, led by Nicholas Rounding, enrolled 249 physicians in Kenya, Indonesia and the Netherlands. Access to GPT-4o improved standardized clinical performance by 18 percentage points in Kenya, 10.7 in Indonesia and 7.2 in the Netherlands.
Those numbers suggest potential value across different health systems. But participants answered controlled cases, and the comparison group could not consult normal resources such as clinical guidelines or the internet. Investigators did not assess harms, limiting conclusions about everyday care.
A more demanding test involved real patients. In a Kenyan primary-care trial published in Nature Medicine, Ambrose Agweyu and colleagues studied 103 clinical officers across 16 facilities, with 9,691 patients enrolled.
The AI-enabled electronic medical record improved documentation measures, including appropriate diagnosis recording and treatment planning. However, treatment failure within 14 days occurred in 2.2% of patients in the AI group and 2.0% in the control group. The adjusted analysis found no statistically significant difference.
Researchers identified no intervention-related safety signal. Still, higher-quality notes cannot be treated as proof that patients recovered faster or needed fewer subsequent visits. The trial also examined a short follow-up period and one care network, limiting wider generalization.
Adoption Is Outrunning Evidence of Benefit
Clinicians are already incorporating AI into their information searches. An October 2026 JAMA study by Vishal Patel and colleagues tracked 952,008 US clinicians submitting 271 million queries to OpenEvidence between January 2024 and June 2026.
Among physician queries, 33.1% concerned treatment and management, compared with 13.9% concerning diagnosis. That difference shows how clinical AI is becoming relevant after a diagnosis is considered, when doctors must select therapies and manage uncertainty.
High usage can signal demand for accessible information, but it cannot tell buyers whether clinicians accept inaccurate suggestions, verify sources, or change treatment decisions.
The study measured platform use, not whether the advice was correct or whether patients benefited. Some researchers were affiliated with OpenEvidence, another reason to keep adoption statistics separate from clinical efficacy.
The Product Challenge Is Accountable Judgment
Medical AI companies cannot rely on benchmark accuracy alone. Developers need to test what happens when evidence is ambiguous, recommendations conflict with a clinician’s assessment, and the interface changes the order of reasoning.
Useful evaluation would include diagnostic accuracy, harmful recommendations, documentation burden, clinician confidence, patient follow-up and whether errors can be identified and corrected. Human oversight must involve genuine scrutiny, not automatic approval of polished machine-generated text.
Health systems also need clear responsibility for decisions. An algorithm can offer a differential diagnosis, but clinicians remain responsible for interpreting the individual patient’s history, explaining trade-offs and choosing care within professional standards.
These studies offer neither a case for abandoning clinical AI nor proof that it is ready to replace clinical judgment. They show that the measure of success is not how convincingly AI answers a medical question, but whether its presence helps professionals deliver safer, better care.
