To examine whether a multi-pronged faculty development effort could improve the quality of neurology clerks' narrative evaluations.

Narrative-based evaluations are increasingly used to discriminate between levels of trainee performance, yet they are often low quality. The evidence has shown mixed results regarding the effectiveness of faculty development efforts on improving narrative evaluation quality.

We used a historical control study to examine the effectiveness of a multi-pronged faculty development effort on narrative evaluation quality. Efforts included: (1) annual neurology grand rounds sessions on developing high quality narratives, (2) adding a rotating neurohospitalist faculty to the neurology grading committee, (3) reporting overall narrative quality data to the department, and (4) instituting an Excellence in Neurology Clerkship Evaluation award to faculty and residents. Following the intervention, we reviewed 120 narrative evaluations from the 2020-2021 academic year, and compared them to 123 evaluations from the 2016-2017 academic year. We evaluated narrative evaluation quality using the previously validated Narrative Evaluation Quality Instrument (NEQI). Linear mixed effects modelling was used to compare NEQI scores (maximum 12 points) across the control and intervention groups. We used multilevel generalized linear models to compare NEQI subcomponent scores (domain, specificity, usefulness) across study groups.

Intervention group was a significant predictor of NEQI scores, with results revealing means of 6.4 (95% CI: 5.90, 6.9) and 7.6 (95% CI: 7.15, 8.10) for the historical control and intervention groups, respectively. Additionally, the intervention was associated with statistically significant improvement in specificity and usefulness NEQI component scores, but not performance domain score.

The multi-pronged faculty development effort was associated with improvements in the quality of neurology clerks' narrative evaluations, suggesting such initiatives may better facilitate improvements relative to single interventions. Future work includes identifying faculty-level factors associated with narrative evaluation quality to inform focused, individual-level interventions.