Abstract Details

Title Automated Review of Longitudinal Electronic Health Records: Privacy-preserving Agentic LLMs for Identifying Incident NAION at Scale

Topic Neuro-ophthalmology/Neuro-otology

Presentation(s) P10 - Poster Session 10 (8:00 AM-9:00 AM)

Poster/Presentation
Number 17-016

Objective To evaluate whether privacy-preserving local large language models (LLMs) using different architectural and prompting approaches can accurately identify non-arteritic anterior ischemic optic neuropathy (NAION) from longitudinal electronic health records (L-EHR).

Background Accurate retrospective identification of acute NAION cases is important for research on risk factors, but ICD-10 coding has limited positive predictive value (PPV), and manual EHR review is time intensive. LLM-based automated review of L-EHR with patient privacy preserved may offer scalable, accurate case identification.

Design/Methods This retrospective single-center study included 161 subjects with ≥ 1 ION (ischemic optic neuropathy) ICD-10 code(s) in the L-EHR. An expert neuro-ophthalmologist classified each subject for presence of acute NAION based on manual L-EHR review.Three locally-deployed (in order to protect privacy) LLM models via llama.cpp were used to implement four methods of NAION diagnostic classification using unstructured L-EHR data: basic prompting, retrieval-augmented generation (RAG), two-step agentic workflow, and three-step agentic workflow. LLM and expert performance were compared.

Results Expert review confirmed acute NAION in 41/161 subjects (25.5%). Agentic LLM workflows substantially outperformed basic and RAG approaches. For direct NAION diagnosis, agentic methods achieved a mean PPV of 86.7% , compared to 61.2% for basic prompting and 60.5% for RAG. Feature extraction with agentic workflows demonstrated high reliability: afferent pupillary defect detection reached 82.6-100% PPV, optic disc edema 78.1-94.1% PPV, and sudden onset 67.0-88.9% PPV across models. Three-step workflows consistently achieved the most balanced performance across all three models.

Conclusions Privacy-preserving agentic LLM workflows achieved high PPV for acute NAION case identification among subjects with ≥1 ICD-10 code for ION, exceeding ICD-10 code PPV and approaching expert PPV of 100%, while maintaining complete local data control and patient confidentiality. These methods offer a scalable approach for case identification in retrospective clinical research.

Authors/Disclosures
Tuyet Thao Nguyen PRESENTER	Ms. Nguyen has nothing to disclose.
Kelvin Z. Li, MBBS, MMed, MTech	Dr. Li has nothing to disclose.
Pareena Chaitanuwong, MD	Dr. Chaitanuwong has nothing to disclose.
Heather Moss, MD, PhD, FAAN (Spencer Center for Vision Research at Stanford)	Dr. Moss has received personal compensation in the range of $100,000-$499,999 for serving as a Consultant for Verana Health. Dr. Moss has received personal compensation in the range of $10,000-$49,999 for serving as an Expert Witness for Legal Firms. The institution of Dr. Moss has received research support from NIH. The institution of Dr. Moss has received research support from Research to Prevent Blindness. Dr. Moss has received intellectual property interests from a discovery or technology relating to health care. Dr. Moss has received personal compensation in the range of $0-$499 for serving as a grant review panel with National Institutes of Health. Dr. Moss has a non-compensated relationship as a Board of Directors with North American Neuro-ophthalmology Society that is relevant to AAN interests or activities.

��ɫ��

��ɫ��