Can an LLM spot what a clinical-trial paper leaves out?
A peer-reviewed benchmark found that a retrieval-plus-LLM pipeline reached F1 0.822 across 119 reporting questions on 40 held-out trial articles. Better evidence retrieval lifted F1 to 0.893, showing both the promise and the main bottleneck.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1Researchers converted 83 SPIRIT 2013 and CONSORT 2010 checklist items into 119 element-level questions and annotated 200 articles forming 100 protocol-results pairs.
- 2On a held-out test set of 20 pairs, or 40 articles, the GPT-5 pipeline reached F1 0.822, sensitivity 0.824, specificity 0.889 and agreement of 0.796; 95% confidence intervals were reported across the questions.
- 3Supplying expert-selected evidence snippets raised F1 to 0.893, while a Qwen-2.5 system was statistically comparable overall. The tool checks whether details are reported; it does not judge whether a trial is well designed, unbiased or clinically useful.
Research topic
Whether a retrieval-plus-LLM pipeline can identify missing element-level reporting in randomized-trial protocols and results papers
The answer: useful as a reporting screen, not as an evidence judge
A peer-reviewed paper published on 6 October reports that a two-stage AI pipeline could identify many specific details that randomized-trial papers did or did not report. On 119 element-level questions applied to a held-out set of 20 protocol-results pairs, the principal GPT-5 pipeline reached an F1 score of 0.822, sensitivity of 0.824, specificity of 0.889 and Gwet's AC1 agreement of 0.796 against expert annotations. The evaluation covered 40 articles, not 40 trials, because each trial contributed a protocol and a results paper.
Those figures support a narrow use: flagging passages and omissions for an author, reviewer or editor to inspect. They do not show that the system can decide whether a trial was well designed, whether its findings are true or whether a treatment helps patients. Reporting completeness and methodological quality overlap, but they are not interchangeable. A beautifully reported biased trial remains biased; a missed detail may reflect a retrieval failure rather than an absent method.[1]
The Impact Brief · Free
Follow the evidence in science & research.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
The benchmark breaks broad checklists into 119 concrete questions
The researchers started with SPIRIT-CONSORT-TM, an earlier corpus of 200 open-access articles representing 100 randomized trials. Each trial had a protocol and a results publication annotated against 83 items drawn from SPIRIT 2013 and CONSORT 2010. Because one checklist item can contain several obligations, the team decomposed those items into 119 questions. A single question might ask whether a paper reports a dose, frequency and duration separately rather than treating any mention of dosing as complete.
Three experts in epidemiology and meta-research developed the questions and their instructions. Eighty-five questions required one answer and 34 allowed multiple selections; 26 depended on an earlier answer. Across the 200 articles, annotators made 9,892 explicit judgments out of 19,500 potential article-question combinations. Another 9,608 combinations were coded automatically because no relevant text was annotated or a question did not apply. The data and code are publicly available through the authors' Open Science Framework repository.[1][2]
Human agreement was substantial but not uniform
Two experts independently annotated 50 articles, covering 25 protocol-results pairs, and reconciled disagreements. One expert annotated the remaining 150 articles. Mean inter-annotator agreement across the questions was Gwet's AC1 0.782, with a standard deviation of 0.219; the median was 0.821. Some questions produced perfect agreement, while questions about whether adjusted analyses were pre-specified or post hoc fell to 0.33.
That spread matters because a model cannot be more objectively correct than an uncertain reference makes possible. It also shows why element-level questions are more informative than one broad completeness score. Straightforward presence checks may be suitable for automation; interpretive questions may need clearer guidance or human review. The study's reconciled labels create a useful benchmark, but most articles were seen by only one annotator, leaving possible annotation error in three quarters of the corpus.[1]
The evaluation separates retrieval from question answering
The automated pipeline first retrieves sentences relevant to a checklist item using a PubMedBERT model previously fine-tuned on the corpus. It then gives those snippets, the element-level question, expert-written instructions and permitted answer options to a large language model. The principal system used GPT-5; the comparison used the open-weight Qwen-2.5-72B-Instruct model in 8-bit form on one Nvidia H100 GPU. Both were tested under the same question-answering setup.
All reported evaluations used the pre-existing test split of 20 protocol-results pairs, while 80 pairs supplied training information and the majority-class baseline. That patient-like unit split matters because keeping paired papers together reduces leakage between a trial's protocol and results report. Performance was aggregated across 119 questions, and 95% confidence intervals were estimated with 10,000 bootstrap resamples. For the principal GPT-5 pipeline, F1 was 0.822 with a 95% interval from 0.794 to 0.847; agreement was 0.796 with an interval from 0.760 to 0.829.[1]
Retrieving the right evidence is the larger bottleneck
When the question-answering model received expert-annotated snippets instead of automatically retrieved ones, F1 rose from 0.822 to 0.893, sensitivity from 0.824 to 0.894, specificity from 0.889 to 0.935 and Gwet's AC1 from 0.796 to 0.877. The expert-snippet condition is an upper-bound experiment because a deployed checker would not normally receive passages selected by a human who already knows where the relevant evidence is.
Adding one illustrative example produced only modest gains. The best one-shot setting reached F1 0.830, an increase of 0.8 percentage points. More examples degraded performance in unreported experiments. GPT-5's aggregate results were slightly higher than Qwen-2.5's, but their 95% intervals overlapped; Qwen did better on 23 questions and tied on 39. That suggests the pipeline's evidence selection and question design may matter at least as much as choosing the larger proprietary model.[1]
The dataset is valuable, but it is small and historically bounded
The 200 articles came from paired publications in the PubMed Central Open Access Subset. That design enables full-text research and protocol-results comparison, but it may not represent subscription journals, specialties, languages, publication formats or trial designs outside the sample. The held-out denominator—40 articles from 20 trials—is especially important when interpreting an aggregate score across 119 questions. A system can look consistent across thousands of question decisions while still having little evidence about new journals or domains.
The framework is based on SPIRIT 2013 and CONSORT 2010 because those were the standards behind the source corpus. The authors mapped their questions against the 2025 updates and found partial rather than full coverage for many items. A production checker would therefore need additional questions, renewed expert validation and version control as reporting standards change. The published study also evaluates manuscripts already in a research corpus; it does not test whether showing flags to authors improves a submission or reduces research waste.[1]
What this could change in publishing
A carefully deployed checker could give authors concrete prompts before submission, help editors prioritise manual review and point reviewers to a suspected omission with its supporting passage. Because the system is built around permitted answers and evidence snippets, it offers a more inspectable workflow than asking a general chatbot whether a manuscript follows CONSORT. The public dataset also gives independent teams a shared target for testing retrieval, smaller models and human-AI review.
The safer design is advisory and traceable: show the exact question, retrieved passage, model answer and uncertainty; let a human correct the record; and never equate an absent passage with misconduct. Journals would need to protect unpublished manuscripts, disclose any external model processing and monitor whether error rates differ by specialty, language or writing style. The paper reports an estimated API cost of about $1.45 per article for GPT-5 and $0.45 for hosted Qwen-2.5, but privacy, integration, human review and governance would add costs not captured by token pricing.[1][2]
What would change the assessment
Confidence would rise with a locked external test across journals, specialties, languages and recent papers written to SPIRIT and CONSORT 2025, reported at the article and question level. Independent teams should reproduce the benchmark from the released data and code, then test whether better retrieval closes the gap to expert-selected evidence without creating new false omissions. Calibration, abstention and the time experts spend correcting flags matter more to practice than one pooled F1 score.
The decisive evidence would be a prospective editorial trial: randomly offer the checker to some authors or reviewers, measure which reporting elements improve, record false alerts and workflow time, and follow whether readers can verify methods more easily. The assessment would weaken if performance drops outside the 40-article test set, if confidential manuscripts cannot be protected or if automated scores become a gatekeeping shortcut. For now, this is a credible benchmark and a useful research tool—not a licence to automate peer review.[1][2]
What this means for people
- More complete trial reports could help clinicians and patients understand what was done and judge whether results apply to them.
- Authors and reviewers could receive faster, more specific prompts, but false flags may add workload or disadvantage writers outside the benchmark's dominant styles.
- The system does not determine whether an intervention is safe or effective and should not replace methodological or clinical review.
Global context
The research team is based at US universities and evaluated English-language open-access trial publications, while clinical research and publishing are global. Reporting guidance is internationally used, but journal workflows, language, document structure, privacy law and editorial capacity vary. A shared public benchmark is useful, yet deployment claims need local and multilingual validation before a checker can support equitable research oversight.
What the evidence does not yet show
- The held-out evaluation contained 40 articles representing 20 randomized trials from one existing open-access corpus.
- Only 50 of 200 articles were double-annotated; one expert annotated the remaining 150.
- The framework derives from SPIRIT 2013 and CONSORT 2010 and does not fully cover their 2025 updates.
- Expert-selected snippets produced markedly better results, so automatic evidence retrieval remains a major source of error.
- The study measures reporting-element classification, not trial quality, risk of bias, truth of findings, patient benefit or editorial outcomes.
- The publisher identifies the peer-reviewed article as an early accepted version that may receive further editorial changes before the final Version of Record.
What to watch next
- Independent reproduction using the public code and data, with question-level error analysis.
- External validation across specialties, journals, languages and SPIRIT/CONSORT 2025 manuscripts.
- Retrieval models that approach the expert-snippet upper-bound without hiding supporting evidence.
- Prospective editorial trials measuring corrected omissions, false alerts, review time and manuscript privacy.
- Clear policies preventing a reporting-completeness score from being treated as a study-quality or misconduct verdict.
Living evidence record
Impact record IAI-0V3GTP8
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
6 October 2026
Source trail
2 direct sources across 2 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 6 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Science & Research
What did first-day verification find in OpenAI’s mass maths release?
OpenAI’s repository now lists 719 AI-produced manuscripts, down from 722 after a sign error invalidated three linked papers. Fourteen more received proof repairs, and 300 of 719 top-line results have formalizations—evidence that publication volume and verified mathematical knowledge are not the same thing.
7 min · 3 sources
Science & Research
Can AI reveal hidden disease patterns in tissue?
A peer-reviewed method used ensembles of neural networks and statistical testing to find disease-associated spatial structures across rheumatoid arthritis, ulcerative colitis and dementia datasets. It is a research-discovery tool, not a diagnostic test or evidence of improved patient care.
8 min · 2 sources
Science & Research
Can AI design better optimisation rules?
A Nature Machine Intelligence paper reports that a structured LLM framework outperformed prior LLM approaches across 36 combinatorial-optimisation benchmarks and produced feasible algorithms for four new port-logistics problems. It remains an offline benchmark, not a live operations result.
9 min · 2 sources
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.