Close more deals with
our awesome app.

P-72: Development and validation of a novel model for characterizing migraine outcomes within electronic health records utilizing artificial intelligence

Presented at the 64th Annual Scientific Meeting American Headache Society (June 9-12, 2022)

Kymes, S.¹; Cady, R.²; Hindiyeh, N.³; Alexander, K.⁴; Riskin, D.⁴

¹Lundbeck, LLC, Deerfield, IL, United States, Deerfield, IL, USA; ²Lundbeck LLC,RK Consults, Bothell, WA, USA; ³Stanford Headache Clinic at Hoover, Stanford, CA, USA; ⁴Verantos Inc. Menlo Park, CA, USA, USA

Summary

A migraine outcome model was generated to score migraine disease severity using routinely collected clinical data and automated feature extraction using artificial intelligence; measuring recall, precision, and F1 score indicated that algorithms using artificial intelligence can identify migraine-related features to generate migraine outcome scores with high accuracy.

Background

Information required to assess migraine endpoints used in clinical trials are not consistently captured in routine care, limiting the utility of these endpoints in real-world evidence (RWE) studies. In addition, there is currently no validated approach that enables studying migraine outcomes at scale in routine care. We report the development and validation of a novel outcome model for migraine based on headache severity and symptoms captured in routinely collected narrative data in electronic health records (EHRs). This study tests the accuracy of a migraine outcome model to score migraine disease severity based on features that exist in routinely collected clinical data and extracted from physician narratives using artificial intelligence (AI).

Methods

A migraine outcome model was defined by a panel of headache specialists with the selection of clinical features based on relevance to migraine disease state and likely presence in routinely collected data. Encounters captured in primary care and neurology EHR data were reviewed by two clinical annotators to create a manual reference standard for features included in the migraine outcome model. Cohen's kappa score of 0.7 was required to consider the reference standard adequate. Accuracy of a migraine outcome score based on automated feature extraction using AI compared to a migraine outcome score based on a manual reference standard was evaluated. Automated feature extraction performance metrics included recall, precision, and F1 score. Accuracy was measured as the percentage of encounters with matching migraine outcome scores based on automated feature extraction and manual feature extraction according to an exact match (matching manual feature exactly on 10-point scale) or close match (matching manual feature within 1 point on 10-point scale). Variability in migraine severity between encounters was evaluated as the difference in migraine outcome scores between 2 encounters. Differential item functioning (DIF) analysis of gender, age, comorbidities, and weight were performed to evaluate trends in migraine outcome score based on patient characteristics.

Results

Features in the migraine outcome model included headache severity and associated symptoms (nausea, vomiting, and light or sound sensitivity). Accuracy of feature extraction using AI was evaluated in 2,006 encounters. Recall, precision, and F1 scores were >90% for migraine and headache, >80% for all symptoms, >70% for mild headache, and >65% for severe headache. Outcome model scores based on automated and manual feature extraction were an exact match and close match for 77.2% and 82.2% of encounters, respectively. Migraine severity worsened, had no change, and improved in 18.8%, 63.0%, and 18.2% of paired encounters, respectively. DIF analysis showed differential reporting of headache and nausea according to gender. DIF was negligible for age, comorbidities, and weight.

Conclusion

These data indicate that algorithms using AI techniques can identify migraine-related features to generate migraine outcome scores with high accuracy. Application of these methods could provide a scalable approach to EHR-based clinical studies and could be used to generate RWE to support migraine prevention and treatment.

(2022), American Headache Society 64th Annual Scientific Meeting June 9–12, 2022 Denver, Colorado. Headache, 62: 1-170. https://doi.org/10.1111/head.14317