好色先生

好色先生

Explore the latest content from across our publications
Join The AAN

Log In

Forgot Password?
Create New Account

Loading... please wait

Abstract Details

AI-assisted versus Human-generated Personal Statements for Neurology Residency Applications
好色先生, Research, and Methodology
P5 - Poster Session 5 (11:45 AM-12:45 PM)
15-019
To evaluate whether neurology faculty can distinguish AI-assisted (AI) or human-generated(HG) personal statements(PS) in residency applications.
Large language models (LLMs) can help brainstorm, draft or edit PS for residency applications. No studies to date have examined the effect of LLM use on PS for neurology residency applications. Prior work in other specialties compared fully AI-generated versus HG PS, but most applicants are likely to LLMs as an assistive tool, which our study evaluates.
Eleven anonymized AI-PS were collected from medical student volunteers (three post-match, eight pre-match). Volunteers received a brief introduction to LLM basics and had two weeks to submit PS using ChatGPT 4o without restriction. A pool of 86 deidentified PS from neurology applicants selected for interview in the 2023 Neurology Match (predating widespread use of ChatGPT) provided the HG source; 11 HG-PS were selected for comparison. The 22 PS were mixed, randomized and independently rated by six blinded neurology faculty using sliding scales for various qualities. Two-sided t-tests compared AI and HG PS.
Faculty reviewers had a median of 3.5 years’ of experience on residency selection committees. There was no statistically significant differences between the AI-PS and HG-PS  for readability (p < 0.119), originality (p < 0.072), authenticity (p < 0.341), overall quality (p < 0.695) or the “why neurology?” item (p < 0.730). Three reviewers suspected AI use in some PS and  three were unsure. On average faculty estimated 2.5 PS as fully AI generated while 8.1 as AI-assisted, with an average confidence level of 41/100. Four of six reviewers were unaware of AAMC guidelines on AI use in residency PS.
AI-assisted PS  were indistinguishable from human-generated PS by blinded neurology faculty across multiple  quality measures with low confidence in detecting AI use.
Authors/Disclosures
Natalie Erickson
PRESENTER 		Miss Erickson has nothing to disclose.
Scott Millis Scott Millis has received personal compensation in the range of $500-$4,999 for serving as an Editor, Associate Editor, or Editorial Advisory Board Member for Taylor & Francis. The institution of Scott Millis has received research support from NIH & NIDILRR.
Waleed Raheem Abood Abood, MD (Wayne State University) Dr. Abood has nothing to disclose.
Maysaa M. Basha, MD, FAAN (Wayne State University, Detroit Medical Center) Dr. Basha has nothing to disclose.
Peter A. LeWitt, MD (Henry Ford Hospital - Franklin Pointe) Dr. LeWitt has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Amneal. Dr. LeWitt has received personal compensation in the range of $5,000-$9,999 for serving as a Consultant for ONO Pharmaceuticals. Dr. LeWitt has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Supernus. The institution of Dr. LeWitt has received research support from Michael J Fox Foundation.
Anza B. Memon, MD, FAAN (Wayne State University, SOM, Detroit, MI) Dr. Memon has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Inlightened. Dr. Memon has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Connected Research. Dr. Memon has received personal compensation in the range of $5,000-$9,999 for serving on a Scientific Advisory or Data Safety Monitoring board for Horizon Therapeutic . The institution of Dr. Memon has received research support from Genentech. The institution of Dr. Memon has received research support from TG Therapeutics.
Philip M. Ross, DO (Wayne State University Physicians Group, Department of Neurology) Dr. Ross has received personal compensation in the range of $100,000-$499,999 for serving as a Staff Physician/Direct Patient Care with Veteran's Health Administration at the John D. Dingell VA Medical Center .
Jacob C. Rube, MD (University Health Center) Dr. Rube has nothing to disclose.
Carla Watson, MD Dr. Watson has nothing to disclose.
Deepti Zutshi, MD, FAAN (Wayne State University School of Medicine) Dr. Zutshi has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Xenon pharmaceuticals. An immediate family member of Dr. Zutshi has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Boston Scientific. Dr. Zutshi has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Aucta Pharmaceuticals. Dr. Zutshi has received personal compensation in the range of $500-$4,999 for serving on a Speakers Bureau for Aucta.
Deepa Raghavan Miss Raghavan has nothing to disclose.
Rohit A. Marawar, MD, FAAN (Wayne State University - Detroit Medical Center) Dr. Marawar has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Jazz Pharmaceuticals. Dr. Marawar has received personal compensation in the range of $500-$4,999 for serving on a Scientific Advisory or Data Safety Monitoring board for SK Pharma. Dr. Marawar has received personal compensation in the range of $500-$4,999 for serving on a Scientific Advisory or Data Safety Monitoring board for Xenon. Dr. Marawar has received personal compensation in the range of $10,000-$49,999 for serving on a Speakers Bureau for Neurelis.