Abstract Details

Title Large Language Models to Infer Depression in Patients with Neurological Conditions: Applications and Limitations

Topic Multiple Sclerosis

Presentation(s) P2 - Poster Session 2 (8:00 AM-9:00 AM)

Poster/Presentation
Number 1-006

Objective To develop a large language model (LLM) prompt capable of inferring a patient’s categorization of depression from a singular clinical note written by their multiple sclerosis (MS) neurologist.

Background LLMs are proving invaluable in organizing electronic health records (EHRs) by facilitating the extraction of discrete variables from unstructured data. In the care of individuals with chronic neurological conditions such as MS, the timely recognition and treatment of depression represents a vexing problem, which could be supported by LLM applications.

Design/Methods This single-center retrospective study compared three depression indicators from single routine neurology visits (n=278 adults with MS). (1) A patient-reported outcome [PRO: Hospital Anxiety and Depression Scale (HADS-D) or Patient Health Questionnaire-9 (PHQ9)] categorized based on severity. (2) A manual annotation of the neurologist’s impression (depression: present, absent, no mention). (3) An LLM prompt developed utilizing an institutionally secure version of ChatGPT4, refined in five successive rounds, to detect depression in the neurologist’s note.

Results Overall, the LLM prompt detected depression in 60.4% of notes (168/278). When compared to the neurologist’s impression, accuracy was 84.4% (97.3% sensitivity; 68.3% specificity; 79.3% positive predictive value; 95.3% negative predictive value). The major discrepancy arose from the prompt’s additional inference of prior/treated depression (based on associated symptoms, history, and medications) even without explicit neurologist mention. When the prompt and neurologist impression disagreed, the prompt tended to match the PRO (61.9%). The neurologist’s impression of depression was “absent” or “no mention” in 59.1% (91/154) visits with PROs indicating depression.

Conclusions The LLM depression prompt was highly sensitive to a neurologist’s documentation of depression in unstructured, under-utilized, clinical notes; it, additionally, inferred both present/treated depression from other note components. Potential applications include quality improvement initiatives aiming to improve depression care on a cohort level. Discrepancies observed between neurologists’ notes and patient self-reports underscore the importance of systematic assessments and documentation.

Authors/Disclosures
Shane Poole (UCSF) PRESENTER	Shane Poole has nothing to disclose.
Maya L. Julian-Kwong	Ms. Julian-Kwong has nothing to disclose.
Kyra Henderson (UCSF)	Kyra Henderson has nothing to disclose.
Jaeleene Wijangco	Jaeleene Wijangco has nothing to disclose.
Nikki Sisodia (University of California San Francisco)	Nikki Sisodia has nothing to disclose.
Chu-Yueh Guo, MD (UCSF Medical Center)	Dr. Guo has received personal compensation in the range of $500-$4,999 for serving on a Scientific Advisory or Data Safety Monitoring board for TG therapeutics.
Riley Bove, MD, FAAN (University of California, San Francisco)	Dr. Bove has received personal compensation in the range of $5,000-$9,999 for serving as a Consultant for Alexion. Dr. Bove has received personal compensation in the range of $5,000-$9,999 for serving as a Consultant for Amgen. Dr. Bove has received personal compensation in the range of $5,000-$9,999 for serving as a Consultant for Genzyme-Sanofi. Dr. Bove has received personal compensation in the range of $10,000-$49,999 for serving as a Consultant for TG Therapeutics. Dr. Bove has received personal compensation in the range of $10,000-$49,999 for serving as a Consultant for EMD-Serono. Dr. Bove has received personal compensation in the range of $500-$4,999 for serving on a Scientific Advisory or Data Safety Monitoring board for Cadenza. The institution of Dr. Bove has received research support from Biogen. The institution of Dr. Bove has received research support from Eli Lilly. The institution of Dr. Bove has received research support from Novartis. The institution of Dr. Bove has received research support from Roche Genentech.

��ɫ��

��ɫ��