Heart failure phenotyping

Arthur Reshad Garan¹, Keri L Monda², Ricardo E Dent-Acosta², Daniel J Riskin³, Ty J Gluckman⁴

Objective

Quantitatively evaluate the quality of data underlying real-world evidence (RWE) in heart failure (HF).

Design

Retrospective comparison of accuracy in identifying patients with HF and phenotypic information was made using traditional (ie, structured query language applied to structured electronic health record (EHR) data) and advanced (ie, artificial intelligence (AI) applied to unstructured EHR data) RWE approaches. The performance of each approach was measured by the harmonic mean of precision and recall (F₁ score) using manual annotation of medical records as a reference standard.

Setting

EHR data from a large academic healthcare system in North America between 2015 and 2019, with an expected catchment of approximately 500,000 patients.

Population

4288 encounters for 1155 patients aged 18–85 years, with 472 patients identified as having HF.

Outcome measures

HF and associated concepts, such as comorbidities, left ventricular ejection fraction, and selected medications.

Results

The average F₁ scores across 19 HF-specific concepts were 49.0% and 94.1% for the traditional and advanced approaches, respectively (p<0.001 for all concepts with available data). The absolute difference in F₁ score between approaches was 45.1% (98.1% relative increase in F₁ score using the advanced approach). The advanced approach achieved superior F₁ scores for HF presence, phenotype and associated comorbidities. Some phenotypes, such as HF with preserved ejection fraction, revealed dramatic differences in extraction accuracy based on technology applied, with a 4.9% F₁ score when using natural language processing (NLP) alone and a 91.0% F₁ score when using NLP plus AI-based inference.

Conclusions

A traditional RWE generation approach resulted in low data quality in patients with HF. While an advanced approach demonstrated high accuracy, the results varied dramatically based on extraction techniques. For future studies, advanced approaches and accuracy measurement may be required to ensure data are fit-for-purpose.

1. Beth Israel Deaconess Medical Center, Department of Medicine, Division of Cardiology, Harvard Medical School, Boston, Massachusetts, USA
2. Amgen Inc, Thousand Oaks, California, USA
3. Verantos, Menlo Park, California, USA
4. Center for Cardiovascular Analytics, Research and Data Science (CARDS), Providence Heart Institute, Providence Research Network, Portland, Oregon, USA