Validation of Verantos AI in asthma

Karynsa Kilpatrick¹, Keri Monda¹, Daniel Riskin²

Purpose

Regulatory, market access, and prescribing decisions increasingly rely on insights derived from analyses of health data collected in routine care, also known as real-world evidence (RWE). As RWE can potentially influence the standard of care, underlying data quality is necessarily scrutinized. Rigorous approaches are required to quantitatively understand data quality, particularly in research questions intended to influence care and in diseases that are phenotypically complex. In this study, asthma is used as a testbed to compare data quality between traditional RWE approaches (structured data) and advanced RWE approaches (narrative electronic health record [EHR] data, natural language processing [NLP], artificial intelligence [AI]-based inference, and optimization for accuracy).

Methods

Eighteen asthma-related concepts were extracted from 6,037 health care encounters using the longitudinal EHRs of 3,481 patients. The concepts included asthma-related conditions (e.g., asthma and its severity), co-morbidities, symptoms, findings, and procedures. A manual reference standard was created through chart abstraction, with two annotators reviewing each record. Inter-rater reliability was measured by Cohen’s kappa score with a threshold of 80% considered sufficient. Accuracy was measured as an F1 score. A two-sided p-value of 0.05 and a Chi-squared test were used to compare the statistical differences between the advanced and traditional RWE approaches.

Results

In the traditional study arm, the average recall, precision, and F1-score across concepts were 40.7%, 72.4%, and 52.1%, respectively. In the advanced study arm, the average recall, precision, and F1-score were 95.6%, 93.8%, and 94.7%, respectively. There was an absolute increase of 42.6% and a relative increase of 81.8% in the F1-score between traditional and advanced approaches, with a p-value <0.001 for all concepts. Cohen's kappa score indicated 0.8 inter-rater reliability, reflecting a credible reference standard.

Conclusions

By applying NLP and AI-based inference, high-reliability data were obtained to enable high-validity RWE generation in asthma. Varied data types and technologies led to highly variable data quality. The results highlight the importance of measuring accuracy for key variables once specific data types and technologies are selected.

Clinical implications

High-validity RWE in asthma could enable the generation of increasingly reliable evidence to improve patient care.

Presented at CHEST 2023, October 10, 2023

¹Amgen, Inc.
²Verantos Inc.