Did web search make GPT-5 a better diagnostician?
Not reliably. In a peer-reviewed benchmark of 161 image-based clinical cases, the strongest reasoning configuration reached 46.0% diagnostic accuracy. Adding web search cut the simplest configuration from 38.5% to 26.7%, while agent and deep-research modes also trailed that baseline.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1Researchers tested ten GPT-5 configurations on the same 161 cases, producing 1,610 responses that three physicians adjudicated against the case evidence and reference diagnosis.
- 2Accuracy rose with reasoning strength from 32.9% for Thinking Mini to 46.0% for Pro, but web search cut Auto accuracy from 38.5% to 26.7%; Agent scored 36.6% and Deep Research 29.8%.
- 3This was a retrospective benchmark using curated cases and a live proprietary interface, not a trial of clinicians using AI or evidence that the system improves patient care.
Research topic
Whether stronger reasoning, web retrieval and agentic workflows improve the reliability and auditability of a general-purpose multimodal AI on open-ended diagnostic cases
The direct answer: more tools did not mean better diagnosis
Web search did not reliably improve GPT-5's answers in this benchmark. The clearest gains came from a stronger underlying reasoning configuration: diagnostic accuracy increased from 32.9% for Thinking Mini to 38.5% for Auto, 41.6% for Thinking and 46.0% for Pro. The best result therefore remained wrong on more than half of the 161 open-ended cases under the study's clinician-consensus scoring rule.
Adding search produced a particularly sharp reversal for Auto, whose accuracy fell from 38.5% to 26.7%. Search yielded only small gains for the three explicit reasoning variants. More elaborate workflows did not rescue the result: Agent reached 36.6% and Deep Research 29.8%, both below Auto without web search. The practical finding is narrower than a verdict on every medical agent. Within the tested GPT-5 configurations, core reasoning and input quality mattered more than extra browsing or workflow complexity.
This is consequential for teams considering general-purpose AI as a diagnostic aid. Retrieval can supply credible-looking material while reinforcing an incorrect first impression, and an agent can hide rather than clarify how it arrived at a conclusion. A system should not be trusted because it searched the web, cited sources or performed more steps. Those features need their own evidence that they correct errors under the intended clinical conditions.[1]
The Impact Brief · Free
Follow the evidence in health & life sciences.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
What the researchers tested across 1,610 responses
The researchers started with MedXpertQA, a multimodal benchmark containing patient narratives, symptoms, findings, medical images and reference labels. They removed basic-knowledge items, retained questions requiring reasoning about diagnosis, examination or treatment, and asked medical experts to keep cases tied to concrete, actionable clinical decisions. A final restriction to diagnostic questions and a minimum representation for each body system produced 161 cases: 40 cardiovascular, 34 digestive, 34 respiratory, 31 musculoskeletal and 22 nervous-system cases.
Every case was run independently through ten ChatGPT configurations, generating 1,610 responses. The set included Auto; Thinking Mini, Thinking and Pro; web-search versions of those four; Agent; and Deep Research. All received the same zero-shot prompt, case description and images. Tests ran through the live ChatGPT interface from 9 to 20 August 2025, so the experiment captured specific product configurations during that period rather than an immutable model that another hospital can retrieve today.
The cases included CT, MRI, X-ray, PET, pathology, EEG or ECG images and clinical charts. Age was available in 116 cases and sex in 108. Calling the cases real-world means the underlying narratives and images were clinically grounded; it does not mean the models were deployed prospectively in a hospital. The study did not measure effects on a clinician's diagnosis, treatment decision, workload or patient outcome.[1]
How the answers were judged
Three physicians independently evaluated the model responses using semantic equivalence, consistency with clinical evidence, specificity, anatomical correctness and relevance. They could accept a model answer that differed in wording or used a clinically equivalent parent or subtype label. Responses were rejected when they contradicted key findings, changed important anatomy or polarity, or introduced unsupported content. The final binary accuracy score counted a model answer as correct when the panel judged it equivalent to the ground truth or better supported by the evidence.
Observed agreement among raters ranged from 0.76 to 0.80, while Gwet's AC1 ranged from 0.56 to 0.64, which the authors characterised as moderate reliability. Majority vote formed the consensus label. A small number of cases with only two raters could tie, and the primary analysis coded those ties as correct. That choice and the moderate reliability are reasons to treat percentage-point differences cautiously, especially where the paper did not report a large separation.
The design is stronger than using one model to grade another, but it is not the same as comparing AI with clinicians diagnosing the same patients. The physician panel adjudicated answers; it did not establish a human-performance benchmark, and it did not test whether AI advice helps or distracts a clinician during care.[1]
Why retrieval sometimes amplified the first mistake
The search-enabled models often retrieved professional medical or educational sources, particularly at higher reasoning levels. Source quality alone did not translate into reliable correction. Across configurations, search rescued only 6.2% to 11.2% of cases that had initially been wrong. It also disrupted some correct answers: in Auto, 18.0% of initially correct cases became incorrect after search.
The authors' case analyses described a confirmation pattern. Search frequently followed the model's original trajectory, finding semantically related material that supported the initial guess instead of prompting a genuine reconsideration. Stronger variants more often selected domain-relevant sources, yet could still privilege visually similar examples over the lesion-specific evidence in the case. Retrieval therefore added information without necessarily improving the model's use of that information.
This matters for clinical product design. A search layer needs a process for testing alternative hypotheses, checking whether evidence actually matches the patient and surfacing disagreement. Merely appending sources to a draft diagnosis can make a weak conclusion look better documented. Evaluation should report how often retrieval corrects, preserves and worsens decisions, not just whether citations are present.[1]
Images and poor inputs remained the central weakness
Across most configurations, misinterpretation appeared in roughly 45% to 65% of cases under the paper's error framework. The models could produce an orderly differential and use patient history sensibly while reading the decisive visual evidence incorrectly. That combination is dangerous because a coherent explanation can conceal the fact that the underlying image was misunderstood.
A physician check of the error analysis reviewed 928 assigned error-type annotations from a random sample of 51 cases and confirmed 827, or 89.1%. But the first-pass labels were model-generated, and the human validation tested their precision rather than exhaustively searching every response for omitted errors. The percentages describe errors found by this framework; they are not a complete clinical incident count.
All configurations also performed worse on low-quality cases, with reported accuracy drops of roughly 10 to 30 percentage points. That result makes input standards a deployment requirement, not a housekeeping detail. A real system needs to detect blurred or incomplete images, ambiguous questions and missing context, then abstain or request better data. More reasoning steps cannot recover evidence the model cannot see correctly.[1]
Agent workflows added an audit problem
Agent and Deep Research modes were designed to plan, use tools and iterate, but neither beat the plain Auto baseline in this study. Agent returned only a final diagnosis without visible intermediate work. Deep Research exposed more detail, yet 29 of its 161 case outputs lacked citations and the remaining searches relied more often on community or case-sharing sites than the stronger reasoning models did.
Clinical audit does not require displaying an unrestricted private chain of thought. It does require a trace that a professional can examine: the inputs used, tools called, sources retrieved, material evidence selected, changes between drafts and final uncertainty. A long narrative explanation is not a substitute for those operational records. If an agent acts across systems, teams also need logs of permissions, failed calls and any data written or transmitted.
The authors compared separate open medical-agent frameworks as reference points and found that longer traces and more retrieval were not consistently associated with higher diagnostic accuracy. Because those external frameworks differ in design, they are not a clean head-to-head result. The cautious conclusion is about the general assumption: workflow complexity should earn trust through measured improvement, not receive it automatically.[1]
What this means for clinicians and patients
For clinicians, the paper supports treating general-purpose multimodal AI as an unproven diagnostic aid. A plausible differential may still be useful for reflection, but the system should not control a diagnosis or work-up, and the clinician needs the original image and clinical evidence rather than an AI summary alone. Search-generated citations must be checked for relevance to the individual case, not accepted as a safety signal.
For patients, none of the percentages estimates the chance that GPT-5 will diagnose a real consultation correctly. The benchmark deliberately selected difficult image-based cases, did not reproduce the full clinical encounter and did not test a clinician-AI team. It also did not measure delay, unnecessary testing, missed treatment or health outcomes. The headline numbers are useful for comparing configurations inside the experiment, not for consent forms or bedside risk prediction.
For hospitals and product teams, the most useful requirement is a local prospective evaluation of a frozen system. Testing should include representative image quality, specialties, languages and patient groups; measure correction and automation-bias effects; and compare the complete workflow with current practice. Search and agent functions should be removable so their incremental benefit can be tested rather than bundled into one opaque product claim.[1]
What would change the assessment
Confidence would increase with preregistered, multi-site studies in which clinicians are randomised to work with or without a locked multimodal assistant on incoming or carefully constructed unseen cases. The evaluation should measure top diagnosis and differential quality, clinically consequential omissions, time, additional tests, changes after incorrect advice, subgroup performance and patient outcomes. Separate arms for reasoning, retrieval and agent actions would show which component actually helps.
Independent replication is particularly important because the tested systems were proprietary and accessed through a live interface. Background updates could have introduced variability, and the results cannot reveal which internal component produced a change. Releasing prompts, case identifiers, adjudication rules and per-configuration outputs where licensing and privacy permit would make future comparisons more meaningful.
npj Digital Medicine published the peer-reviewed accepted paper online on 9 October 2026. The authors declared no competing interests; the work was partly supported by the US National Institutes of Health intramural programme through the National Library of Medicine. The evidence is strong enough to reject a simple 'more tools are better' assumption, but not to rank every current medical AI system or predict clinical benefit. For now, stronger reasoning improved a difficult benchmark, while browsing and agents failed to make it reliably safe.[1]
What this means for people
- Clinicians could receive a useful second perspective, but browsing may reinforce a wrong first guess and add verification work.
- Patients should not interpret benchmark accuracy as a personal probability of diagnosis or evidence of improved care.
- Hospitals need to test reasoning, retrieval and agent actions separately before allowing any component to influence a clinical pathway.
Global context
The cases and participating institutions were US-centred, while diagnostic pathways, image quality, disease prevalence, language and access to specialists vary internationally. A general-purpose model may retrieve sources from around the world without understanding whether guidance fits the local patient or health system. Each deployment therefore needs local validation and governance; a result from one English-language benchmark cannot be treated as a universal safety score.
What the evidence does not yet show
- The 161 curated benchmark cases were not consecutive patients, a prospective clinical workflow or a representative prevalence sample.
- Testing used proprietary configurations through the live ChatGPT interface in August 2025, so hidden updates may limit reproducibility.
- Physicians adjudicated model answers but did not form a direct clinician-performance comparison group on the same task.
- Inter-rater reliability was moderate, and ties in a small number of two-rater cases were counted as correct in the primary analysis.
- The model-generated error labels were checked for precision on 51 cases but were not exhaustively annotated for missed errors.
- No effect on diagnosis by clinicians, treatment, workload, safety events or patient outcomes was measured.
What to watch next
- Preregistered clinician-AI trials using frozen systems and genuinely unseen multimodal cases.
- Ablation studies reporting when retrieval corrects, preserves or worsens an initial diagnosis.
- Image-quality detection, abstention and visual-grounding tests across hospitals and devices.
- Auditable tool and evidence traces without exposing sensitive data or relying on persuasive prose.
- Independent replication after product updates and across languages, specialties and patient groups.
Living evidence record
Impact record IAI-0P9V3QM
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
12 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what npj Digital Medicine published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 12 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Health & Life Sciences
What did a real-clinic test of Google’s medical chatbot actually show?
In a supervised, single-centre feasibility study, 100 adults completed an AMIE text chat before an urgent primary-care visit and no conversation met the prespecified threshold for a safety stop. The Lancet paper does not show that the chatbot can diagnose, triage or improve care without clinicians.
9 min · 4 sources
Health & Life Sciences
Can an AI agent safely change a clinical record?
A 43-task benchmark found that long-term memory helped a GPT-4.1-mini agent use FHIR tools, but the strongest held-out result was only 60.6% task success. This was a resettable test server—not a live hospital record system.
8 min · 3 sources
Health & Life Sciences
Can AI flag a longer ICU stay?
A peer-reviewed model separated longer-stay risk across 5,281 retrospective sepsis admissions in one US development cohort and two Korean external cohorts. Its AUROC fell from 0.848 internally to 0.781 at the smaller Korean site, and no prospective study tested whether alerts improve care or capacity.
7 min · 2 sources
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.