To develop, refine and test a natural language processing (NLP) algorithm to extract multiple sclerosis (MS) relapses (MS  Relapse  Sentiment Algorithm) from clinical notes with high sensitivity.

Clinical relapses in MS have been dramatically reduced by effective therapies, but remain a focus of scientific inquiries (e.g. natural history, health economics). Historically, these relapses were recorded as unstructured data in clinical  notes,  requiring  time-consuming review of the electronic medical record (EMR) for extraction.

A combination of string matching, sentiment analysis and classification were implemented to identify MS relapses from three sets of notes for patients seen in the UCSF MS Center. The training set included 500 notes from research participants (relapses recorded annually); validation set included 450 notes from women aged 18-50 (high relapse rates); test set included 150 notes from the latter group and 250 notes from patients beginning anti-CD20 therapy (relapsing and progressive MS). Altogether, the notes spanned 502 patients, 13 years (2005-2018), and 14 clinicians. Algorithms were developed to extract relevant information from long notes, which were then passed into a Naïve Bayes probabilistic classifier that determined whether or not a note indicated a relapse. This output was compared to categorizations made by manual review for MS expert indication of clinical relapse since prior visit.

In the final test set (400 notes), the algorithm correctly identified 47/48 relapses (sensitivity = 97.9%). Negative predictive value was 99.6% (274/275 notes). However, positive predictive value was 37.6% (47/125) and specificity was 78%. The NLP algorithm classified an additional 3,000 notes in <1 minute, identifying possible relapses in <1/3 of notes.

The MS Relapse Sentiment Algorithm could rapidly scan a clinic’s historical EMR notes with high sensitivity, allowing researchers to focus their manual review for false positives in the lower proportion of notes marked as clinical relapses.