Can clinicians see the evidence behind approved diagnostic AI?
A peer-reviewed audit found public, device-specific performance evidence for 30 of 77 approved pathology and haematology AI products. The result is a transparency finding—not proof that the other 47 lack regulatory evidence or that the documented products are clinically equivalent.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1The authors identified 77 regulated devices from 29 manufacturers: 68 for digital pathology and nine for haematology morphology. All had European market access, while eight also had US authorization.
- 2Public, quantitative, device-specific performance evidence was found for 30 devices, or 39%. The absence of an identified public source for the other 47 does not mean regulators or manufacturers had no evidence.
- 3Most documented evaluations were retrospective and metrics were highly heterogeneous. Even among the 30 evidence-supported devices, the audit could not support a league table or a claim that approval guarantees comparable public proof.
Research topic
How much device-specific performance evidence is publicly accessible for regulated AI diagnostic software in digital pathology and haematology morphology
The audit measured public visibility, not regulatory approval quality
The researchers asked a practical question that falls between regulation and clinical procurement: when a laboratory, pathologist or health-technology assessor considers an approved AI product, can they find quantitative, device-specific evidence in public? They assembled a landscape of diagnostic software for digital pathology and haematology morphology using the FDA's AI-enabled device list, a previous open device register, manufacturer portfolios, scholarly searches, conference proceedings and regulatory sources. The cutoff was 31 March 2026. After excluding unauthorized, research-only, hardware-embedded and radiology products, the final denominator was 77 devices made by 29 companies.
Every included product had European market access through IVDR certification or a legacy IVDD route; 45 were IVDR-certified and 32 held legacy certification. Eight also had US authorization—six through 510(k) and two through De Novo—leaving 69 products approved only in the European sample. Approval still implies that evidence was submitted and assessed under the relevant process. The study did not inspect confidential technical files or notified-body dossiers, and it did not retest the products. Its finding is about what an outside decision-maker can independently locate, interpret and compare.[1][2][3]
The Impact Brief · Free
Follow the evidence in ai risks & safety.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
Thirty of 77 devices had identifiable public performance evidence
The search found public, quantitative, device-specific performance evidence for 30 of the 77 devices, or 39%. For 47 devices, or 61%, the researchers did not identify such evidence. Marketing brochures and qualitative claims without device-level results were excluded. The search hierarchy favoured peer-reviewed articles, then conference abstracts, preprints and regulatory documents. Of the 30 devices with evidence, 25 had a peer-reviewed article as their highest-level source, three were supported by a conference abstract, one by a preprint and one by a regulatory filing.
That denominator needs careful language. “No identified public evidence” is not the same as “no evidence exists,” “the device is unsafe” or “the authorization was unsupported.” Data may sit in non-public regulatory submissions, manufacturer files or sources missed by the search. Conversely, the presence of a paper does not prove that a device will work in every laboratory. Public evidence matters because hospitals must judge whether a study matches their patients, tissue preparation, staining, scanner, workflow and clinical threshold—and because independent scrutiny is difficult when results cannot be seen.[1]
US-authorized products were more visible, but the groups were small
All eight devices authorized in both Europe and the United States had public evidence, compared with 22 of 69 devices in the Europe-only group, or 32%. Fisher's exact test returned p below .001. Haematology morphology products were also more likely to have public evidence—seven of nine, compared with 23 of 68 digital-pathology products—with p equal to .02. Those contrasts describe the audited sample; they should not be treated as causal proof that one regulator or specialty produces better technology.
The eight-product transatlantic group is small and may differ in company resources, product maturity, intended use or publication strategy. Regulatory pathways also create different public documents. The comparison between legacy IVDD and newer IVDR certification was not statistically significant: 14 of 32 legacy devices and 16 of 45 IVDR devices had public evidence, with p equal to .49. Publication lag may matter for recently certified products. The authors therefore resist using the result as a verdict on IVDR and argue instead for structured, consistent disclosure across pathways.[1]
The available studies were mostly retrospective and hard to compare
Among the 30 evidence-supported products, 25 evaluations were retrospective. Sample size could be extracted for 28: the median was 211 patients, slides or specimens, with an interquartile range from 103 to 720 and a range from 30 to 13,349. Fifteen of those 28 evaluations used fewer than 300 units. Those counts cannot be pooled naively because a patient, whole-slide image and specimen are different denominators, while disease prevalence and sampling strategy affect apparent performance. Few studies were prospective or multicentre, leaving selection bias and optimistic local validation as continuing concerns.
The audit found 15 primary metric types grouped into discrimination, agreement, correlation and prognostic performance. Seventeen devices used a discrimination metric, seven agreement, three correlation and three prognostic outcomes. Sensitivity appeared for ten devices, but six of those did not report a paired specificity; no device reported specificity without sensitivity. That asymmetry can make clinical trade-offs difficult to see. A high sensitivity may reduce missed cases while increasing false positives, workload or unnecessary review. Without paired metrics, thresholds and confidence intervals, a procurement team cannot reconstruct that balance.[1]
Assisted-reader studies suggest promise, not a product ranking
Most documented products were compared with a pathologist, expert or manual reference. Eight studies compared AI-assisted with unassisted readers, but only five supplied paired accuracy results. Those studies reported improvements on at least one outcome for products including Paige Prostate Detect, Lunit SCOPE PD-L1, Paige Lymph Node, Ibex Galen IHC HER2 and Deep Bio DeepDx Prostate CNB. Some differences were statistically significant and one study reported shorter review time. Yet endpoints, readers, specimens and designs differed, and several results were small or not statistically significant.
None of this creates a head-to-head league table. Only seven devices shared the same broad combination of agreement metric, retrospective single-centre design and AI-versus-pathologist comparator; eleven had unique design-and-metric combinations. A percentage from one product cannot safely be placed beside another when disease mix, reference standard, scanner, reader experience and operating threshold differ. The useful signal is narrower: some products have credible evidence that assistance can improve a defined task, but the public record is not standardized enough to compare the market or estimate system-wide benefit.[1]
Funding, conflicts and method limits shape the conclusion
The work was funded by the EU Horizon Europe EU4MEDTECH project; the authors state that the funder had no role in the study. Two authors disclosed share ownership in Cellbytes Ltd, and one also reported research funding or consultancy relationships with several pharmaceutical companies; the remaining authors reported no competing interests. Those disclosures do not invalidate the audit, but they should accompany interpretation because the paper concerns commercial diagnostic products and access to market evidence.
The authors describe a systematic landscape analysis rather than a registered systematic review. One author conducted device identification, extraction, verification and analysis, which increases the value of independent replication. European device discovery is also difficult because EUDAMED participation remains incomplete, and search results depend on manufacturers' naming and publication practices. The team used two consecutive search rounds that yielded no new devices as a completeness check, but cannot prove that every eligible product or obscure evidence source was captured.[1]
What would change the assessment
Confidence would rise if regulators or manufacturers published a stable, device-level evidence summary for every product: intended use, population, specimen and scanner, study design, sample denominator, comparator, thresholds, sensitivity and specificity with uncertainty, external validation and post-market updates. IVDR's summaries of safety and performance for higher-risk devices can help, but coverage and accessibility need to become consistent. The EU AI Act may add transparency and governance duties, yet it does not replace device-specific clinical evidence under IVDR or the Medical Device Regulation.
For individual products, the decisive next evidence is prospective, multicentre and independent validation in routine workflows. Studies should report reader assistance against unassisted practice, time and correction burden, false-positive and false-negative consequences, performance across laboratories and patient groups, failures after software updates, and post-market safety signals. Until that record improves, approval should be read as permission to market within a defined regulatory pathway—not as a guarantee that clinicians can readily inspect comparable public evidence or that products are interchangeable.[1][3]
What this means for people
- Pathologists and laboratories need visible evidence to judge whether software tested elsewhere fits their specimens, scanners, patients and workflow.
- Patients may benefit when independent validation reveals where assistance reduces missed disease, but opaque or mismatched evidence can hide avoidable error and workload.
- Procurement teams should ask for the underlying study and operating threshold rather than treating approval, a marketing percentage or a peer-reviewed paper as interchangeable signals.
Global context
The sample spans European approvals and a small group of products also authorized in the United States, but the transparency problem is international. Regulatory systems often assess information that is not fully public, while hospitals must make local adoption decisions across different populations, laboratories and workflows. Comparable device-level reporting would support procurement and post-market learning without collapsing distinct regulatory decisions into a single global quality score.
What the evidence does not yet show
- The analysis measured publicly identifiable evidence and did not access confidential regulatory or manufacturer files; the 47-device gap cannot be interpreted as evidence of no validation.
- The landscape cutoff was 31 March 2026, six months before publication, so later certifications, papers or product changes are outside the denominator.
- The dual-authorized group contained only eight devices, and observed jurisdiction differences may reflect product maturity, company resources or disclosure practices.
- Most documented evaluations were retrospective, with heterogeneous units, comparators and metrics that prevented pooled performance comparison.
- One author performed device identification, extraction, verification and analysis, and incomplete European databases make omissions possible.
What to watch next
- Public device-level summaries that link intended use, validation populations, metrics, thresholds and software versions.
- Independent prospective multicentre studies measuring pathologist assistance, errors, workload and patient consequences.
- Consistent IVDR disclosure and post-market performance reporting as legacy devices transition to the newer framework.
- Whether future audits can match regulatory identifiers across Europe and the United States without relying on product-name searches.
Living evidence record
Impact record IAI-02XAXTM
Evidence stage
Studied
Confidence
Corroborated
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 October 2026
Source trail
3 direct sources across 2 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 3 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 5 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
AI Risks & Safety
Can a few poisoned documents reframe medical AI answers?
Yes, in a peer-reviewed laboratory study across three biomedical corpora and five open-weight language models. Injecting five crafted documents per question often put at least three in the top five retrieved results, and one or two retrieved poisons could be enough in successful attacks.
6 min · 1 source
AI Risks & Safety
Can fine-tuning hide large changes in medical AI answers?
Yes. In a peer-reviewed accepted study, medical-model checkpoints with similar overall accuracy still changed which questions they answered correctly on 15.1%–18.2% of a 2,450-item test. The result exposes a model-upgrade risk, but it is a benchmark study—not clinical validation or evidence of patient harm.
7 min · 2 sources
AI Risks & Safety
Can connected AI agents turn a benign request into harm?
A Meta-led preprint found harmful outcomes in 192 of 1,000 simulated agent scenarios generated by its new red-teaming method. The result exposes cross-service risk, but it comes from one synthetic environment and model-based judges—not live consumer accounts.
7 min · 1 source
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.