Can a respiratory chatbot improve lay triage?
A preregistered Chinese trial found that 2,176 compliant participants answered simulated respiratory questions more accurately with a task-constrained GPT-4o chatbot than with ordinary web search. The largest gain was diagnosis; triage accuracy remained modest, and the trial did not measure real care-seeking or patient outcomes.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1The trial randomized 2,400 adults across 24 Chinese healthcare systems; the primary compliant-case analysis included 2,176 questionnaires, split evenly between LungDiag assistance and conventional mobile web search.
- 2Model-based overall item accuracy was 70.0% with LungDiag and 55.4% with search, but the diagnosis difference was much larger than the triage difference: 81.2% versus 54.5%, compared with 53.4% versus 47.1%.
- 3High-urgency triage sensitivity was 85.6% with LungDiag and 80.9% with search, while specificity was only 62.1% and 56.0%; the study used simulated cases and cannot establish safer real-world care-seeking.
Research topic
Whether a task-constrained respiratory chatbot helps adults without medical training answer simulated diagnosis, risk and triage questions more accurately than ordinary web search
The trial compared a constrained chatbot with ordinary search
Researchers recruited 2,400 adults aged 18 to 75 without formal medical training across 24 Chinese healthcare systems between April and June 2025. Participants were randomly assigned one-to-one, stratified by centre, to use either LungDiag or conventional mobile web search. Dedicated AI and generative-search functions were disabled in the control arm. Outcome assessors were blinded, but participants, data collectors and analysts could not be because the two tools were visibly different.
LungDiag was not an unrestricted public chatbot. It used GPT-4o through a WeChat mini-program, a respiratory knowledge layer and task-specific routing for risk or aetiology, preliminary diagnosis and four-level triage. The study bank contained 88 simulated respiratory cases: 44 risk-behaviour vignettes and 44 diagnosis-and-triage vignettes, with the triage set evenly divided among self-care, care within one week, care within one day and emergency care. Trial cases and answer keys were kept out of the retrieval index.[1][2]
The Impact Brief · Free
Follow the evidence in health & life sciences.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
The primary result was an item-level accuracy difference
The prespecified compliant-case analysis retained 2,176 questionnaires, 1,088 in each arm. Researchers analysed correct and incorrect answers with generalized linear mixed-effects models that included random intercepts for participant and item, recognizing that repeated answers from one person and differently difficult questions are not independent. LungDiag's model-based overall accuracy was 70.0%, with a 95% confidence interval from 65.2% to 74.8%; the search arm reached 55.4%, with an interval from 49.7% to 61.1%.
That 14.6-percentage-point adjusted difference was statistically clear, with a 95% confidence interval from 12.2 to 17.0 percentage points and a risk ratio of 1.26. Sensitivity analyses included all 2,400 randomized participants under different assumptions for assigned-but-unanswered items, and the direction of the result remained. The main analysis was nevertheless a modified intention-to-treat approach because non-compliant questionnaires were excluded, so the cleanest interpretation is performance among participants whose data met the protocol's usability rules.[1][3]
Diagnosis improved more than triage
The aggregate result hides an important difference among tasks. Preliminary diagnosis accuracy was 81.2% with LungDiag and 54.5% with search, a 26.7-point gap. Risk or aetiology questions also favoured the chatbot, at about 78.5% versus 67.1%. Triage was the weak point: 53.4% with LungDiag and 47.1% with search. A tool can therefore produce a plausible disease label more often while still struggling to recommend the appropriate urgency.
Each participant in the primary analysis answered four triage items, producing 4,352 triage responses per arm. When the four categories were simplified into high urgency—within one day or emergency—versus lower urgency, LungDiag reached 85.6% sensitivity and 62.1% specificity. Search reached 80.9% sensitivity and 56.0% specificity. The chatbot found more urgent cases, but its moderate specificity implies many lower-urgency scenarios would still be escalated. This experiment did not measure whether that caution reassures users, crowds services or delays a different necessary action.[1]
Better answers took longer and remained simulated
LungDiag users took a median 527 seconds to complete the questionnaire, compared with 461 seconds for search. The additional 66 seconds is not necessarily a defect: a structured conversation may collect information that a quick search misses. But time, cognitive load and abandonment matter in consumer health tools, especially when someone is breathless or anxious. The paper reports task completion, not whether users understood uncertainty, trusted the advice appropriately or could act on it under real pressure.
The intervention also answered fixed vignettes rather than observing genuine symptom evolution, examination findings or care pathways. Participants knew they were completing a study exercise, not deciding whether to call emergency services for themselves or a relative. The trial did not measure subsequent attendance, clinician diagnosis, treatment, diagnostic delay, health outcomes or service use. Its strongest conclusion is that this constrained interface helped lay participants answer simulated questions—not that it safely diagnoses or triages patients.[1]
Reproducibility is partial rather than complete
The investigators released de-identified participant-level data, the primary mixed-model code, sensitivity analyses and figure scripts on GitHub and Zenodo. That is unusually useful for checking the statistics. They did not release the exact internal prompt, retrieval settings, chunking and reranking thresholds, the complete LungDiag knowledge layer or the trial-specific mini-program source. Those proprietary boundaries mean an independent team cannot recreate the intervention exactly from the public package.
The study was supported by Chinese public research grants. The interface was hosted by Xingyun and used paid API access routed through APIYI; the authors say neither vendor influenced design, conduct, analysis or reporting and report no employment, equity, patent, royalty or revenue-sharing relationship. The LungDiag-derived knowledge resource is owned and maintained by the authors' group. None of these disclosures invalidates the result, but they matter when the evaluated product cannot be fully reproduced or transferred unchanged to another health system.[1][3]
What would change the assessment
Confidence would increase with a locked, fully specified system tested against alternative AI-enabled search tools rather than a control from which generative functions were removed. Replication should use new case banks, languages, literacy levels and countries; separately report under-triage and over-triage; and verify performance after model or knowledge-base updates. Independent security, privacy and failure-mode testing would be essential for a messaging platform that handles health descriptions.
The decisive next step is a prospective real-world study with safety oversight. It should recruit people actually deciding what to do, compare advice with final clinician diagnoses and observed care pathways, and measure emergency escalation, delay, unnecessary attendance, anxiety, adherence and clinical outcomes. Until then, the trial is strong evidence that a constrained respiratory chatbot can improve simulated diagnostic recognition over ordinary search, but only limited evidence that it can safely guide real people to the right level of care.[1][2]
What this means for people
- A well-constrained chatbot could help people interpret respiratory symptoms more accurately than fragmented search results.
- Moderate triage specificity could direct many lower-urgency users towards faster care, adding anxiety or pressure to services.
- The study does not show that a chatbot should replace emergency advice, professional assessment or local care pathways.
Global context
The randomized evidence comes from 24 Chinese healthcare systems and a WeChat-based interface. Respiratory disease prevalence, emergency access, language, digital literacy and the information available to search users differ internationally. The design is more informative than retrospective model testing, but local validation and prospective safety measurement remain necessary before a health service can treat the result as deployment evidence.
What the evidence does not yet show
- The primary analysis excluded 224 non-compliant questionnaires and is therefore a modified intention-to-treat analysis, although all-randomized sensitivity analyses were directionally consistent.
- Participants answered simulated case questions; the study did not observe real care-seeking, diagnostic delay, service use or patient outcomes.
- The control used conventional search with dedicated AI and generative-search functions disabled, so the result does not compare LungDiag with current AI-enhanced search products.
- Four-category triage accuracy remained modest, and binary high-urgency specificity was 62.1% with LungDiag.
- The exact prompt, retrieval configuration, knowledge layer and mini-program source were not publicly released, limiting full independent reproduction.
- The trial was conducted in China through WeChat and may not transfer to other languages, health systems, digital-literacy levels or emergency pathways.
What to watch next
- Independent replication against contemporary AI-enabled search and other named chatbots.
- Prospective trials measuring under-triage, over-triage, actual attendance, delay and clinical outcomes.
- Performance across languages, literacy levels, disabilities and health systems outside China.
- Whether locked prompts, retrieval resources and versioned knowledge bases can be audited after deployment updates.
Living evidence record
Impact record IAI-0ZE8C12
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 October 2026
Source trail
3 direct sources across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 3 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 5 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Health & Life Sciences
Can an AI agent take a better eye history than a resident?
A randomized trial at a specialist hospital in Guangzhou found that an LLM agent scored higher than ophthalmology residents on structured pre-consultation histories for 172 non-emergency patients. Interviews took much longer, examination findings narrowed the diagnostic gap, and the study did not test patient outcomes or emergency care.
8 min · 2 sources
Health & Life Sciences
Did a specialist health chatbot give safer workplace advice than a general AI assistant?
Nine experts rated 60 answers to sedentary-work questions, while 654 Chinese office workers tested the specialist tool. It scored higher than Doubao on several answer-quality measures, but the study did not measure behaviour change or health outcomes.
7 min · 1 source
Health & Life Sciences
Is operating-room AI ready for clinical use?
Not on the published evidence yet. A peer-reviewed scoping review screened 3,020 records but found only one completed feasibility study with five analysed patients; four larger prospective studies had no results posted.
8 min · 1 source
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.