Can an AI agent take a better eye history than a resident?
A randomized trial at a specialist hospital in Guangzhou found that an LLM agent scored higher than ophthalmology residents on structured pre-consultation histories for 172 non-emergency patients. Interviews took much longer, examination findings narrowed the diagnostic gap, and the study did not test patient outcomes or emergency care.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1The single-centre trial randomized 172 non-emergency ophthalmology patients one-to-one to an LLM agent or resident-led pre-consultation; median age was 56 and 55% were female.
- 2The agent's paper-defined MedHistory score was 17.7 points higher on a 100-point scale after adjustment, with a 95% confidence interval from 14.1 to 21.3, but the median interaction lasted 11.2 minutes rather than 3.1 minutes.
- 3Test-recommendation performance was not significantly different. An exploratory history-only diagnostic advantage narrowed after ocular examination findings were available, underscoring that a better interview is not the same as better clinical care.
Research topic
Whether a structured large-language-model agent can collect ophthalmology histories and recommend examinations as well as or better than resident-led pre-consultation
The trial tested pre-consultation, not autonomous eye care
Researchers at Zhongshan Ophthalmic Center, Sun Yat-sen University, recruited patients attending a tertiary ophthalmic hospital in Guangzhou between 10 May and 10 June 2025. Of 175 people approached, 172 were randomized one-to-one to a large-language-model agent or an ophthalmology resident for the history-taking and examination-recommendation stage before a specialist consultation. The median participant was 56 years old and 55% were female. The registered design was parallel and single-masked: higher-level specialists scored the medical histories without knowing the assigned group.
This was a narrow workflow comparison. Eligible participants had non-emergency ocular disease, including corneal, lens and vitreoretinal conditions. The registry explicitly excluded ten ocular emergencies, such as globe perforation, chemical injury, acute angle-closure glaucoma, central retinal artery occlusion and orbital cellulitis. The trial therefore cannot show that the agent recognizes time-critical eye disease safely, manages a complete appointment or replaces an ophthalmologist. It asks whether structured software can do one information-gathering task before the clinician takes over.[1][2]
The Impact Brief · Free
Follow the evidence in health & life sciences.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
The agent collected a more complete structured history
The primary outcome was the MedHistory score, a 0-to-100 assessment based on the standard outpatient record. According to the registry, clinicians from multiple specialties and researchers developed the scoring criteria, and senior specialists scored records independently while blinded to allocation. The paper reports an adjusted mean difference of 17.7 points in favour of the agent, with a 95% confidence interval from 14.1 to 21.3 and p below .001. That interval indicates a substantial difference on this study's rubric rather than a borderline statistical result.
The agent used a dual-agent architecture and symptom-oriented logic trees to standardize questioning. That can reduce variability and prompt follow-up questions that a busy clinician might omit. But the outcome measures completeness according to a paper-defined score, not whether every collected detail was clinically necessary or whether the resulting encounter improved diagnosis, treatment or vision. External validation of the scoring rubric and independent replication would help establish whether the measured advantage generalizes beyond the investigators' workflow.[1][2]
More complete interviews took more than three times as long
The median interaction lasted 11.2 minutes with the agent and 3.1 minutes with residents. A longer interview can be valuable when it uncovers relevant symptoms, treatment history or risk factors, and a conversational system may let patients answer without feeling hurried. Participants also gave the agent higher median ratings for patience and empathy: five versus four for patience and five versus three for empathy, both with p below .001. Those ratings describe the experience of this pre-consultation task; they do not show that software understood a patient's condition or delivered relational care.
Time is also a deployment cost. If the software runs while patients wait and produces a record that clinicians can rapidly verify, eleven minutes may improve preparation without reducing clinic capacity. If staff must supervise, correct or repeat the interview, the extra time could add friction. The publication does not establish staff workload, downstream consultation length, record-editing burden, accessibility for people with visual or cognitive impairments, abandonment, privacy risk or total appointment time. Those practical measures determine whether a higher history score translates into a better service.[1]
Examination findings narrowed the diagnostic gap
Test-recommendation performance did not differ significantly between groups across the study's stages. An exploratory diagnostic analysis was more revealing about the boundary between conversation and clinical observation. Using history alone, the agent's F1 score was 0.85 and the residents' was 0.68. Once ocular signs from examination were added, residents improved more and the relative gap narrowed; the reported group-by-stage interaction was minus 0.19, with p equal to .003.
That pattern is consistent with the agent being strong at structured verbal intake while clinicians gain more from physical findings. It should not be read as a head-to-head test of final diagnostic competence. F1 combines precision and recall for the categories evaluated but does not convey the severity of individual errors, calibration, rare-disease performance or consequences for treatment. The diagnostic analysis was exploratory, and the trial was neither powered nor designed to prove non-inferiority on clinical outcomes.[1]
The evidence is stronger than a benchmark, but still local
Random allocation, a registered protocol, a contemporaneous resident comparator and blinded specialist scoring make this more informative than asking a model to answer a static examination bank. It involved real patients with a range of non-emergency eye conditions in a functioning specialist hospital. The original registry was first posted on 13 February 2025, before recruitment began, and lists medical-history scoring as the primary outcome along with test accuracy, duration and satisfaction as secondary outcomes.
The denominator remains small and the setting concentrated: 172 patients at one tertiary centre in Guangzhou, compared with residents from that institution. Residents are a relevant early-career comparator, but not the only normal intake pathway; senior ophthalmologists, nurses, optometrists and structured electronic forms could perform differently. Language, local record conventions, case mix, health literacy and referral patterns may also affect the result. A workflow trained around this hospital's questions may not transfer intact to a community clinic or another country.[1][2]
Funding and disclosure need to travel with the result
The authors report support from the National Natural Science Foundation of China, Guangdong provincial programmes, the State Key Laboratory of Ophthalmology and Sun Yat-sen University. They declare no competing interests. Public research funding and a no-conflict declaration reduce some concerns, but neither substitutes for independent validation, especially when the system architecture, clinical scoring process and local workflow were developed by the evaluating team.
The publication reports the result as an accepted peer-reviewed article. Editorial formatting may still change, while the DOI and original publication date should remain the reference points. The ClinicalTrials.gov record lists the study as completed and reports no results on the registry itself; the paper is therefore the source for numerical outcomes, while the registry provides the prospectively posted design and outcome definitions.[1]
What would change the assessment
Confidence would increase with a larger, independently led multicentre trial using a locked version of the agent, a prespecified error taxonomy and external adjudicators. It should compare the agent not only with residents but also with normal nurse or optometrist intake and a well-designed digital questionnaire. Results need to be stratified by age, visual impairment, language, literacy, diagnosis and complexity, with missing data and crossovers reported clearly.
The decisive evidence is downstream. Future studies should measure whether clinicians accept or rewrite the generated history, whether total appointment time changes, whether necessary tests are ordered without unnecessary investigation, and whether diagnostic delay, treatment choice, safety incidents, patient understanding and visual outcomes improve. Emergency presentations require a separate safety study. Until those data exist, the trial supports an AI-assisted history-taking role under clinician review; it does not establish autonomous diagnosis or better eye care.[1][2]
What this means for people
- A structured agent could give patients more time to describe symptoms and reduce omissions before a short specialist consultation.
- Longer digital interviews or inaccessible interfaces could burden people with poor vision, low digital confidence or complex needs.
- The study supports clinician-reviewed pre-consultation assistance, not replacing eye examination, urgent-care pathways or professional judgment.
Global context
The trial comes from a high-volume specialist eye hospital in southern China. Its randomized real-patient design is more clinically relevant than a static model benchmark, but resident training, documentation standards, language, accessibility and referral pathways differ internationally. Health systems considering a similar tool would need local validation, privacy and security review, emergency safeguards and evidence that better documentation improves care rather than simply adding another step.
What the evidence does not yet show
- The trial enrolled 172 patients at one tertiary ophthalmic hospital in Guangzhou and may not generalize to primary care, community clinics, other languages or other health systems.
- Ten ocular emergencies were excluded, so the study provides no evidence that the agent safely recognizes or triages urgent eye disease.
- The primary endpoint was a study-defined history-completeness score, not diagnostic delay, treatment quality, vision, harm or another patient outcome.
- The agent interview took a median 11.2 minutes versus 3.1 minutes for residents; total workflow time and staff correction burden were not established.
- Residents were the comparator. Performance against nurses, optometrists, senior ophthalmologists or a structured electronic questionnaire remains unknown.
- Diagnostic results were exploratory, and F1 scores do not describe calibration, error severity or consequences for individual patients.
What to watch next
- Independent multicentre replication with external adjudication and a locked, versioned agent.
- Clinician editing burden, total appointment time and whether a structured history changes test ordering or management.
- Safety testing for urgent symptoms and performance across visual impairment, age, language and literacy groups.
- Patient outcomes, diagnostic delay, unnecessary testing and adverse events rather than documentation scores alone.
Living evidence record
Impact record IAI-0AMIBOG
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 October 2026
Source trail
2 direct sources across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 5 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Health & Life Sciences
Can a respiratory chatbot improve lay triage?
A preregistered Chinese trial found that 2,176 compliant participants answered simulated respiratory questions more accurately with a task-constrained GPT-4o chatbot than with ordinary web search. The largest gain was diagnosis; triage accuracy remained modest, and the trial did not measure real care-seeking or patient outcomes.
7 min · 3 sources
Health & Life Sciences
Can chatbots match urologists on cancer decisions?
In 100 synthetic cancer cases, three leading chatbots were less often fully successful than two early-career urologists and made safety-critical errors far more often. Repeated answers also changed substantially.
7 min · 1 source
Health & Life Sciences
Can a medical AI support a clinical board without becoming the decision-maker?
A peer-reviewed German study put OpenEvidence into discussions of 100 real inflammatory-disease cases. Specialists found the answers useful, but complete agreement with the board was limited, prompt tuning was inconclusive and outputs changed over time. The study tested workflow feasibility—not patient benefit or safety.
9 min · 1 source
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.