Can an eye-care LLM improve referrals?
A peer-reviewed Shanghai study randomly assigned 180 community screening participants to standard reports with or without EyeSeek. Among 170 people analysed, referral-related action was higher with the tool, but the single-centre, Chinese-language study was small and short.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1At a Shanghai community health centre, 180 people were randomly assigned to printed screening reports and standard recommendations with or without access to EyeSeek; 84 assisted and 86 unassisted participants remained in the final three-week analysis.
- 2Referral completion or stated intent within three weeks was 14.28% with EyeSeek versus 4.65% without it (P = 0.037); completed visits alone were 11.90% versus 3.49% (P = 0.039).
- 3The trial was single-centre, Chinese-language and modest in size. It did not measure long-term vision outcomes, formally validated health literacy, cost-effectiveness or the separate contribution of each system component.
Research topic
Whether a locally deployed, ophthalmology-specific language model can explain screening results, adapt readability and change referral behaviour in community eye care
The strongest evidence is a small randomized community comparison
Researchers in Shanghai built EyeSeek to address a practical gap after community eye screening: a printed result can identify risk without ensuring that a resident understands it or acts on a referral. At the Sunqiao Community Health Service Center, 180 eligible people receiving routine health examinations were randomly assigned to standard printed screening reports and recommendations. One group could also use a community terminal to ask EyeSeek for personalised explanations and guidance; the other could seek routine physician consultation or leave after receiving the report.
Loss to follow-up left 84 people in the EyeSeek-assisted group and 86 in the unassisted group for the primary analysis. Within three weeks, 14.28% of assisted participants had completed an ophthalmology referral or expressed an intention to do so, compared with 4.65% without EyeSeek, a difference reported at P = 0.037. Using the stricter endpoint of a completed visit, the rates were 11.90% and 3.49% (P = 0.039). The result is encouraging because allocation was random and behaviour, not just model preference, was measured. It is not proof of a population-wide effect: the absolute event counts were 10 completed visits plus two stated intentions in the assisted group, versus three visits plus one intention in the comparison group.[1]
The Impact Brief · Free
Follow the evidence in health & life sciences.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
Six-month records support attendance, but not clinical benefit
The authors also searched medical-system records over six months. They found documented ophthalmology visits for 23 of 84 assisted participants and five of 86 unassisted participants, or 27.38% versus 5.81%. Every attendee with a record had a specific eye disease or symptom diagnosis, and the authors report no obvious signal of over-referral among those records. Assisted participants who completed referrals also did so earlier on average: 8.5 days compared with 15.5 days.
Those follow-up records make the finding harder to dismiss as a survey artefact, but they still do not show that EyeSeek preserved sight, controlled disease or improved quality of life. Records were available only for people who attended within the observed system, and the paper did not run a powered comparison of clinical endpoints. A referral can be appropriate without producing better health if specialist capacity, cost, transport or treatment access remains constrained. The tool should therefore be read as post-screening communication support, not evidence of autonomous diagnosis or treatment.[1]
EyeSeek combines a specialised model, abstention and tailored language
The system starts from Qwen2-7B and was fine-tuned with 306,425 ophthalmology samples assembled from medical knowledge graphs, licensing-exam data and dialogue corpora. Six clinicians ranked candidate responses for reinforcement learning from human feedback. A separate role-playing set covered children, adults with lower educational attainment and highly educated adults; at inference time, similar examples guide the model toward language intended for that audience. The paper reports that its Chinese outputs ranged from roughly Grade 3 to Grade 8 readability across those groups.
A second mechanism teaches the model to abstain when semantically equivalent versions of a question produce unstable answers. The authors screened 10,324 questions, labelling 7,123 answerable and 3,201 as requiring abstention at a safety-oriented consistency threshold. In a later offline loop, GPT-4 agents generated and solved simulated questions, while ophthalmology experts checked candidate answers before verified examples entered further training. That human review matters, but the architecture does not guarantee truth: the uncertainty detector is the model itself, and the simulation depends partly on another commercial model. The public code helps scrutiny, while participant-level trial data remain restricted for privacy and ethics reasons.[1][2]
Experts and residents liked the answers under controlled tests
Six senior ophthalmologists scored responses to 200 community questions from EyeSeek, two primary-care physicians and three other language models. The paper reports that 90.5% of EyeSeek responses were rated basically or completely consistent with clinical consensus, while 89.0% were judged low risk. These proportions exceeded the named comparators in the study. Inter-rater agreement was high, with a kappa of 0.832 and a 95% confidence interval from 0.733 to 0.931.
A separate preference study enrolled 100 residents across several Shanghai community health centres. Each person compared five anonymised response pairs, creating 500 comparisons. EyeSeek was preferred to Huatuo, EyeGPT and GPT-4 in 87.2%, 76.8% and 69.6% of comparisons, and to a randomly selected written primary-care-physician response in 60.8%. These are useful signals about clarity and usefulness in standardised written answers. They are not a substitute for error monitoring in open-ended deployment, and the physician comparator consisted of two written responders rather than a live consultation with examination and follow-up questions.[1]
The literacy result is promising but partly indirect
In the prospective comparison, assisted participants’ mean eye-health knowledge score rose from 6.08 to 7.70 out of 10, while the unassisted group moved from 5.86 to 5.93. Among people offered EyeSeek, 86.5% reported satisfaction, 84.3% said it improved their understanding and 80.9% would continue using or recommend it. Locally deployed responses took an average of 3.97 seconds. These measures indicate usability and short-term learning in that setting.
The paper also states that formal health literacy was not directly measured. Education and automated readability acted as proxies for comprehension needs, while the ten-item knowledge score captured eye-health knowledge rather than a validated, broad literacy construct. Satisfaction was collected only in the intervention group, so it cannot establish a comparative experience advantage. The trial evaluated the complete package—screening workflow, terminal access, tailored explanation and interaction—so it cannot isolate whether medical accuracy, simplified language or the extra attention drove the behaviour change.[1]
What would change the assessment
Confidence would rise with a preregistered, multicentre trial across rural and urban systems, larger and more diverse populations, concealed allocation, complete outcome capture and blinded adjudication of appropriate versus unnecessary referral. Replication should test other languages, lower-connectivity settings and people with disabilities, using validated health-literacy instruments and reporting costs, clinician workload, digital exclusion and adverse advice. Longer follow-up should measure diagnosis, treatment, vision and quality of life rather than stopping at referral behaviour.
The study was funded by Shanghai science, postdoctoral, ophthalmic-engineering and Tongji University interdisciplinary programmes; the authors declared no competing interests. The present evidence supports a narrow conclusion: in one Shanghai community screening workflow, access to EyeSeek was associated with more and earlier referral activity and better short-term knowledge among a small randomized sample. It does not yet establish that the model is safe across populations, superior to a properly resourced clinician conversation, cost-effective or capable of improving long-term eye health.[1]
What this means for people
- Clearer explanations may help residents understand a screening result and reach specialist care sooner.
- A terminal cannot remove referral costs, transport barriers, digital exclusion or specialist-capacity limits.
- Clinical use requires visible abstention, escalation and audit because the study does not prove safety for all questions or populations.
Global context
The trial was conducted in a peri-urban Shanghai primary-care setting serving agricultural residents and migrant workers. The communication problem is global, but transfer is not automatic: languages, literacy measures, referral systems, clinician availability, device access and costs vary substantially. The work is most relevant as a locally tested design for post-screening explanation, not a universal medical chatbot result.
What the evidence does not yet show
- The prospective study was conducted at one Shanghai community centre and analysed 170 participants after loss to follow-up.
- Referral events were few, and the primary endpoint combined completed visits with stated referral intent.
- Evaluation was exclusively in Chinese; transfer to other languages, cultures and health systems is untested.
- Formal health literacy was not directly measured, and satisfaction had no control-group comparison.
- The integrated intervention cannot reveal which component caused the observed change.
- No long-term outcome such as vision preservation, disease control, cost-effectiveness or equity was measured.
What to watch next
- Independent multicentre replication with preregistration, larger samples and appropriate-referral adjudication.
- Performance in other languages and among people with low digital access or disability.
- Long-term diagnosis, treatment, vision, workload, cost and safety outcomes.
- Component-level tests separating accuracy, readability adaptation, interaction and extra attention.
Living evidence record
Impact record IAI-1MWL5Y6
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 October 2026
Source trail
2 direct sources across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 5 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Health & Life Sciences
Can a health chatbot pass an equity audit?
A peer-reviewed audit generated 1,350 mental-wellness responses from three open models. It found complex reading levels, uneven crisis language and almost no explicit clinician-referral wording—but three repeated prompts and no community validation sharply limit the claim.
8 min · 2 sources
Health & Life Sciences
Did the AI write a better clinic note?
A prospective single-centre study of 62 adults with abdominal pain found that AI-generated notes were more complete than clinicians' notes but scored worse on factuality. The assistant also produced only 50% diagnostic accuracy in the prospective cohort, so the result supports supervised documentation research—not autonomous clinical use.
9 min · 2 sources
Health & Life Sciences
Does primary-care AI improve outcomes?
Not yet on the available evidence. A peer-reviewed review found 10 real-world studies of clinician-facing AI in primary care: some improved detection or care processes, but neither trial measuring patient-important outcomes demonstrated benefit, and the evidence was low or very low certainty.
10 min · 2 sources
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.