External validation of prognostic rules for early post-pulmonary embolism mortality: assessment of a claims-based and three clinical-based approaches
Thrombosis Journal volume 14, Article number: 7 (2016)
Studies show the In-hospital Mortality for Pulmonary embolism using Claims daTa (IMPACT) rule can accurately identify pulmonary embolism (PE) patients at low-risk of early mortality in a retrospective setting using only claims for the index admission. We sought to externally validate IMPACT, Pulmonary Embolism Severity Index (PESI), simplified PESI (sPESI) and Hestia for predicting early mortality.
We identified consecutive adults admitted for objectively-confirmed PE between 10/21/2010 and 5/12/2015. Patients undergoing thrombolysis/embolectomy within 48 h were excluded. All-cause in-hospital and 30 day mortality (using available Social Security Death Index data through January 2014) were assessed and prognostic accuracies of IMPACT, PESI, sPESI and Hestia were determined.
Twenty-one (2.6 %) of the 807 PE patients died before discharge. All rules classified 26.1–38.3 % of patients as low-risk for early mortality. Fatality among low-risk patients was 0 % (sPESI and Hestia), 0.4 % (IMPACT) and 0.6 % (PESI). IMPACT’s sensitivity was 95.2 % (95 % confidence interval [CI] = 74.1–99.8 %), and the sensitivities of clinical rules ranged from 91 (PESI)-100 % (sPESI and Hestia). Specificities of all rules ranged between 26.8 and 39.1 %. Of 573 consecutive patients in the 30 day mortality analysis, 33 (5.8 %) died. All rules classified 27.9–38.0 % of patients as low-risk, and fatality occurred in 0 (Hestia)-1.4 % (PESI) of low-risk patients. IMPACT’s sensitivity was 97.0 % (95%CI = 82.5–99.8 %), while sensitivities for clinical rules ranged from 91 (PESI)-100 % (Hestia). Specificities of rules ranged between 29.6 and 39.8 %.
In this analysis, IMPACT identified low-risk PE patients with similar accuracy as clinical rules. While not intended for prospective clinical decision-making, IMPACT appears useful for identification of low-risk PE patient in retrospective claims-based studies.
Guidelines suggest that patients with pulmonary embolism (PE) who are identified to have a low-risk of early post-PE all-cause mortality may be candidates for abbreviated hospital admission or outpatient treatment if appropriate follow-up can be arranged [1, 2]. Data from randomized trials and observational studies suggest that early discharge or outpatient treatment of low-risk PE patients is feasible and safe [3, 4].
A prior meta-analysis suggested at least one-third of acute PE patients could be classified as low-risk for early mortality according to clinical prediction rules . This same meta-analysis identified the Pulmonary Embolism Severity Index (PESI), simplified PESI (sPESI) and Hestia clinical prediction rules as having high sensitivities and negative predictive values (NPVs) for identifying low-risk PE patients. In order to use PESI , sPESI  and Hestia  in the risk stratification of a patient with PE, access to vital signs, laboratory values, comorbid conditions and a cognitive evaluation at presentation is necessary. While PESI, sPESI and Hestia can be helpful in clinical practice, the extensive clinical data required to score these rules are not commonly found in claims databases or easily accessible to individual hospitals/health-systems. As a result, the utility of PESI, sPESI or Hestia for retrospective identification of low-risk patients with PE is limited.
While not originally derived to aid in prognostication in a prospective clinical setting, the In-hospital Mortality for Pulmonary embolism using Claims daTa (IMPACT) multivariable prediction rule utilizes data accessible within claims databases and/or claims from individual hospitals to retrospectively risk stratify patients with PE for early mortality . Prior validation studies suggest IMPACT can accurately identify pulmonary embolism (PE) patients at low-risk of early mortality [10, 11]. The ability of IMPACT to correctly identify patients at low- and higher-risk of early mortality has not previously been compared to analogous clinical prediction rules. Therefore, using data from a single center, this study sought to externally validate IMPACT, PESI, sPESI and Hestia for predicting in-hospital and 30 day post-PE mortality.
Preparation of this study report was in accordance with the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement . For this retrospective cohort study, we identified consecutive patients diagnosed with acute PE between October 21, 2010 and May 12, 2015 using computerized claims records for admissions to Hartford Hospital (Hartford, Connecticut, USA). The hospital’s computerized claims system contains information on source of admission, International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes, admission and discharge dates and discharge status. To be eligible for inclusion into this study, patients ≥18 years of age presenting to our institution had to have a primary diagnosis of PE (ICD-9-CM code = 415.1x). Consistent with prior studies, we excluded patients lacking objective confirmation of acute PE according to clinical guidelines. The following were considered confirmatory studies for the diagnosis of acute PE: high probability perfusion-ventilation lung scan (V/Q scan), computed tomography pulmonary angiography (CTPA) or pulmonary angiography diagnostic for PE, or a non-diagnostic V/Q scan or CTPA in combination with an abnormal compression ultrasonography of the lower extremities. Consistent with many prior studies of PE clinical prediction rules , subjects that received thrombolysis and/or pulmonary embolectomy within the 48 h of presentation were excluded as clinical guidelines do not consider such patients low-risk [1, 2]. All patients included in this study were managed according to usual clinical practice for our institution.
Risk stratification of patients with acute PE using IMPACT, PESI, sPESI and Hestia was performed according to published methods (Additional file 1) [6–9]. Patients with an IMPACT predicted mortality risk ≤1.5 % , PESI score ≤85  or a sPESI or Hestia scores =0 [7, 8] were classified as low-risk for early mortality. Estimated mortality risk according to the claims-based IMPACT prediction rule [estimated % absolute risk = 1/(1 + exp(-x); where x = −5.833 + (0.026*age) + (0.402*myocardial infarction) + (0.368*chronic lung disease) + (0.464*stroke) + (0.638*prior major bleeding) + (0.298*atrial fibrillation) + (1.061*cognitive impairment) + (0.554*heart failure) + (0.364*renal failure) + (0.484*liver disease) + (0.523*coagulopathy) + (1.068*cancer)] was determined using all available hospital claims data (i.e., ICD-9-CM diagnosis and procedural codes) for each patient’s index PE encounter along with their age at time of presentation. ICD-9-CM coding for relevant IMPACT co-morbidities were performed according to the original IMPACT derivation paper . Data necessary to classify patients as low- or high-risk of early mortality according to the PESI, sPESI and Hestia clinical prediction rules [6–8] were obtained by linking all included patients identified through hospital claims to the hospital’s electronic health record (EHR). We used vital signs (heart rate, blood pressure, respiratory rate, oxygen [O2] saturation, body temperature), laboratory values (serum creatinine, platelet count, total bilirubin) and mental status assessments obtained as close to the time of presentation for the index PE encounter as possible to score each of the clinical prediction rules . For all patients admitted through the emergency department, the first vital sign, laboratory value and/or mental status assessment upon presentation (but within 24 h) was utilized. For patients directly admitted to the hospital we used the first values recorded on the day of hospital admission. Consistent with previous studies of this type, missing vital, laboratory and mental status assessment data were assumed to be normal . For PESI, sPESI and Hestia, the presence of cancer, heart failure, chronic lung disease, severe liver disease (defined as a total bilirubin ≥2.5 mg/dL), heparin-induced thrombocytopenia and recent clinical events (gastrointestinal bleeding within 14 days, stroke with 4 weeks, surgery with 2 weeks) were assessed at time of hospital admission for the index PE encounter. All required data was abstracted from the electronic health record (including vital signs, laboratory values and emergency department, admission and consult notes) by trained study personnel blinded to study outcome.
All-cause in-hospital and 30 day post-PE mortality served as a priori endpoints for this study. In-hospital mortality was determined using the discharge status field for the index admission within the hospital billing system and electronic health record. Thirty-day mortality was based upon searches of the Social Security Death Index (SSDI)  performed >6 months after the last day of eligible inclusion in this analysis. Computerized health-system encounter data from our hospital’s billing records were queried for subsequent emergency visit claims, clinic visits and/or hospital readmissions outside of 30 days. These were used as confirmatory markers of vital status at 30 days. Beginning in March 2014, rules regarding access to data within the SSDI changed; restricting the release of three most recent years of data [13, 14]. For this reason, our 30 day mortality endpoint was assessed only in the subset of consecutive patients presenting to the hospital prior to January 31, 2014.
Baseline characteristics are described with means ± SDs for continuous data and counts and proportions for categorical data. Sensitivity, specificity and negative and positive predictive values for predicting early mortality were calculated for IMPACT, PESI, sPESI and Hestia along with 95 % confidence intervals (CIs). C-statistics were computed to evaluate each rule’s overall discriminative power. All database management and statistical analyses were performed using IBM SPSS Statistics version 22.0 (IBM Corp., Armonk, NY, USA). The study was approved by the Hartford Hospital institutional review board.
A total of 861 patients with a primary ICD-9-CM code for acute PE and objective confirmation of diagnosis were identified (Fig. 1). Of these, 54 received thrombolytic therapy/embolectomy within 48 h of presentation, leaving 807 for analysis. Baseline characteristics of the cohort, stratified by low- and high-risk for each rule are shown in Table 1. Two-hundred and thirty-four patients presented with PE after January 2014 and were excluded from the 30 day mortality endpoint analysis, as SSDI data are not available for patients past this time point. The baseline characteristics of the 30 day mortality analysis patient subset were similar to the overall population (Additional file 1: Table S1).
The IMPACT, PESI, sPESI and Hestia scores (mean ± SD) for the complete cohort of 807 PE patients were 3.9 ± 4.3, 96.4 ± 33.3, 1.1 ± 0.9 and 1.3 ± 1.1, respectively. While mean age for patients was 64.1 ± 16.57; when dichotomized into risk groups, high-risk patients were considerably older, regardless of prediction rule used. At time of presentation, mean vital sign values were within normal ranges in the overall analysis population; but more than half of the subjects required O2 supplementation to maintain saturations >90 %. The use of thrombolysis and/or embolectomy after 48 h was infrequent, occurring in only 4 (0.5 %) of patients.
The overall incidence of all-cause in-hospital mortality was 2.6 % (21/807). IMPACT, PESI, sPESI and Hestia classified 26 % (Hestia)-38 % (PESI) of the cohort as low-risk for early post-PE mortality. Fatality among low-risk patients was low (0–0.6 %), corresponding to NPVs of 99.4 (PESI)-100 % (sPESI and Hestia) (Table 2). IMPACT’s sensitivity was 95.2 % (95 % CI = 74.1–99.8 %), and the sensitivities of clinical rules ranged from 91 (PESI)-100 % (sPESI and Hestia). Specificities of all rules ranged between 26.8 (Hestia)-39.1 % (PESI) and C-statistics from 0.76 (sPESI)-0.86 (Hestia). Additional file 1: Table S2 describes the characteristics of patients who died in-hospital and had discordant risk categorization across any of the four prediction rules.
Among the subset of 573 patients accessible within the SSDI, 33 (5.8 %) died of any cause within 30 days of presentation for PE. All rules classified 27.9 (Hestia)-38.0 % (PESI) of PE patients as low-risk, and fatality occurred in 0.0 % (Hestia)-1.4 % (PESI) of low-risk patients (NPVs = 98.6–100 %) (Table 3). IMPACT’s sensitivity for predicting 30 day mortality was 97.0 % (95%CI = 82.5–99.8 %), while sensitivities for clinical rules ranged from 91 (PESI)-100 % (Hestia). Specificities of rules ranged between 29.6 % (Hestia)-39.8 % (PESI) and their C statistics ranged from 0.73 (sPESI)-0.81 (PESI). Additional file 1: Table S3 describes the characteristics of patients who died within 30 days of presentation and had discordant risk categorization across any of the 4 prediction rules.
In this analysis, the claims-based IMPACT prediction rule displayed prognostic accuracy similar to that of commonly used clinical risk stratification rules, including PESI and sPESI (which have been prospectively validated for identification of low-risk PE patients) and the Hestia criteria (which was prospectively designed to identify patients whom could be treated as outpatients). The 4 rules evaluated in this study classified between 1/4th and 2/5ths of patients as low-risk. Each exhibited sensitivities >90 %, NPVs >98.6 % and specificities <40 % for predicting in-hospital or 30 day all-cause mortality, and these findings are consistent with prior derivation and validation studies. To our knowledge, this is the first external validation study of Hestia [5, 9–11]. Taken together, our results suggests each of the four rules can identify a cohort of low-risk PE patients whom are unlikely to die within the first 30 days of presentation. However, because a minority of patients with PE (<6 % in our study) die within 30 days, these prognostic rules classify a substantial number of patients who ultimately survive into higher-risk groups (hence their lower specificities). Prognostic tests seldom have both high sensitivity and specificity. Therefore, when using a prognostic test to decide upon implementing a less conservative treatment strategy (e.g., discharging a patient with acute PE directly from the emergency department) higher sensitivity and NPV values are preferable.
The American College of Chest Physicians and the European Society of Cardiology guidelines support early discharge and/or home treatment of PE patients at low-risk for early mortality as long as they have adequate home circumstances [1, 2]. These guidelines suggest that clinicians use validated clinical prediction rules to assist in identification and selection of low-risk patients. Although in this analysis IMPACT displayed a similar ability to identify low-risk PE patients as PESI, sPESI and Hestia; IMPACT was not originally derived or validated to assist in prospective clinical decision-making (and is relatively more computationally complex compared to clinical prediction rules), and it should not be used for individual patient decision-making . However, IMPACT appears valid for retrospective identification of low-risk PE patients and therefore could be used to aid in the performance of real-world outcomes studies and to enable payer/institution benchmarking of rates of low-risk PE patients treated at home or following an abbreviated admission. Using claim s data as described by IMPACT may have advantages over obtaining highly granular clinical information from the EHR (including reduced time and effort requirements).
Our study has limitations that require consideration. First, this validation study was performed retrospectively and therefore may be subject to biases, particularly due to missing data. Nonetheless, our study had similar rates of missing data than reported in prior prospective derivation/external validation papers of clinical prediction rules [5–7]. Second, this was a single-center study limiting its generalizability and sample size. However, baseline characteristics and mortality rates were consistent with national estimates [10, 15] and our sample size (573–807 patients) was large relative to many previously published external validation studies of PE clinical prediction rules . Third, we could not assess 30 day mortality in our entire study cohort due to restrictions on the availability of SSDI data . Despite this, the sample size of patients with objectively confirmed PE available for final analysis was robust. Moreover, we are unaware of any programmatic changes in evaluation and treatment of patients with PE at our institution since February 2014. Consequently, the likelihood of selection bias resulting from the unavoidable exclusion of patients after this date is low. Next, the more subjective nature of certain criteria in Hestia (i.e., medical or social reason for hospital admission and the need for intravenous pain medication for >24 h), make retrospective scoring challenging. This being said, the proportion of patients in our study classified as higher-risk because of these “subjective” criteria was not inconsistent with the Hestia derivation study . Lastly, the 48 h cut-off used to exclude patients undergoing thrombolysis and/or embolectomy is somewhat arbitrary. We excluded patients receiving thrombolysis and/or embolectomy in less than 48 h because such patients likely had hemodynamic instability at presentation and would not be considered low-risk per guidelines [1, 2]. Of note, numerous studies evaluating the prognostic accuracy of clinical prediction rules have similarly excluded patients undergoing early thrombolysis and/or embolectomy . However, when these procedures are performed later in a hospital stay (day 3 onwards), they are more likely an indicator of a therapeutic failure resulting in a poor clinical course (i.e., respiratory failure or cardiac arrest). In addition, the need for and timing of thrombolysis and/or embolectomy can easily be detected in a claims database and a clinical setting, allowing it to be implemented in our evaluation of IMPACT and clinical prediction rules. The 48 h cut-off was chosen a priori to approximate the likely timing used for assessing the similar Hestia criterion (i.e., the Hestia study required discharge within 24 h of PE diagnosis, likely resulting in the assessment of the 11 Hestia criteria within 48 h of initial PE presentation) .
IMPACT identified low-risk PE patients with similar accuracy as PESI, sPESI and Hestia. While not designed for prospective clinical decision-making, IMPACT appears useful for identification of low-risk PE patient in retrospective claims-based studies.
Ethics approval and consent to participate
This study was approved by the Hartford Hospital institutional review board with a waiver of informed consent.
Consent for publication
Consent for publication: not applicable.
Computed tomography pulmonary angiography
Electronic health record
International classification of diseases, ninth revision, clinical modification
In-hospital mortality for pulmonary embolism using claims daTa
Negative predictive value
Pulmonary embolism severity index
Positive predictive value
Simplified pulmonary embolism severity index
Social security death index
Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis
United States of America
Kearon C, Akl EA, Ornelas J, Blaivas A, Jimenez D, Bounameaux H, Huisman M, King CS, Morris T, Sood N, Stevens SM, Vintch JRE, Wells P, Woller SC, Moores CL, Antithrombotic Therapy for VTE Disease: CHEST Guideline, CHEST (2016), doi: 10.1016/j.chest.2015.11.026.
Konstantinides SV, Torbicki A, Agnelli G, Danchin N, Fitzmaurice D, Galiè N, et al. 2014 ESC guidelines on the diagnosis and management of acute pulmonary embolism. Eur Heart J. 2014;35:3033–69.
Zondag W, Kooiman J, Klok FA, Dekkers OM, Huisman MV. Outpatient versus inpatient treatment in patients with pulmonary embolism: a meta-analysis. Eur Respir J. 2013;42:134–44.
Aujesky D, Roy PM, Verschuren F, Righini M, Osterwalder J, Egloff M, et al. Outpatient versus inpatient treatment for patients with acute pulmonary embolism: an international, open-label, randomised, non-inferiority trial. Lancet. 2011;378:41–8.
Kohn CG, Mearns ES, Parker MW, Hernandez AV, Coleman CI. Prognostic accuracy of clinical prediction rules for early post-pulmonary embolism all-cause mortality: A bivariate meta-analysis. Chest. 2015;147:1043–62.
Aujesky D, Obrosky DS, Stone RA, Auble TE, Perrier A, Cornuz J, et al. Derivation and validation of a prognostic model for pulmonary embolism. Am J Respir Crit Care Med. 2005;172:1041–6.
Jiménez D, Aujesky D, Moores L, Gómez V, Lobo JL, Uresandi F, et al. Simplification of the pulmonary embolism severity index for prognostication in patients with acute symptomatic pulmonary embolism. Arch Intern Med. 2010;170:1383–9.
Zondag W, den Exter PL, Crobach MJ, Dolsma A, Donker ML, Eijsvogel M, et al. Comparison of two methods for selection of out of hospital treatment in patients with acute pulmonary embolism. Thromb Haemost. 2013;109:47–52.
Coleman CI, Kohn CG, Bunz TJ. Derivation and validation of the In-hospital mortality for pulmonary embolism using claims daTa (IMPACT) prediction rule. Curr Med Res Opin. 2015;31:1461–8.
Coleman CI, Kohn CG, Crivera C, Schein JR, Peacock WF. Validation of the multivariable in-hospital mortality for pulmonary embolism using claims daTa (IMPACT) prediction rule within an all-payer inpatient administrative claims database. BMJ Open. 2015;5:e009251.
Kohn CG, Peacock WF, Fermann GJ, Bunz TJ, Crivera C, Schein JR, et al. External validation of the in-hospital mortality for pulmonary embolism using claims data (IMPACT) multivariable prediction rule. Int J Clin Pract. 2016;70:82–8.
Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1–73.
Social Security Administration. Social Security Death Index, Master File. Social Security Administration. Available at: http://www.ntis.gov/products/ssa-dmf.aspx (Last accessed on August 2, 2015).
Bipartisan Budget Act of 2013. Public Law 113–67. Sec. 203. Restriction on access to the death master file. Available at: https://www.congress.gov/113/plaws/publ67/PLAW-113publ67.pdf (Last accessed on August 2, 2015).
Wiener RS, Schwartz LM, Woloshin S. Time trends in pulmonary embolism in the United States: evidence of overdiagnosis. Arch Intern Med. 2011;171:831–7.
This study was funded by Janssen Scientific Affairs, LLC, Raritan, NJ, USA
CIC has received grant funding and consultancy fees from Janssen Scientific Affairs, LLC; Bayer Pharma AG; and Boehringer-Ingelheim Pharmaceuticals, Inc. CC, JRS are employees of Janssen Scientific Affairs LLC. GJF has received grant funding for Novartis, Cardiorentis, Cardioxyl, Cempra Pharmaceuticals, Trevena, Intersection Medical, Siemens, The Mayday Foundation, Pfizer; and is on the advisory board and speakers bureau for Janssen Scientific Affairs. FWP has received grant funding and consultancy fees from Abbott, Alere, Banyan, Cardiorentis, Janssen Pharmaceuticals, Portola, Roche, The Medicine’s Company, Prevencio and Singulex.
The authors declare that they have no competing interests.
Study concept and design: ERW, CGK, GJF, WFP, CC, JRS, CIC. Acquisition of data: ERW, CGK, CT, DM, CIC. Analysis and interpretation of data: ERW, CGK, GJF, WFP, CT, DM, CC, JRS, CIC. Drafting the manuscript: ERW, CGK, CT, DM, CIC. Critical Revision of the manuscript for important intellectual content: ERW, CGK, GJF, WFP, CT, DM, CC, JRS, CIC. Administrative, technical or material support: ERW, CGK, CIC. Study supervision: CIC. CIC had full access to all the study data and take full responsibility for the integrity of the data and the accuracy of the data analysis. All authors read and approved the final manuscript. The authors meet criteria for authorship as recommended by the International Committee of Medical Journal Editors (ICJME) and were fully responsible for all content and editorial decisions, and were involved in all stages of manuscript development. All authors read and approved the final manuscript.
Description of the IMPACT, PESI, sPESI and Hestia Prediction Rules. Table S1. Characteristics of Pulmonary Embolism Patients in the 30-Day Mortality Cohort. Table S2. Description of Patients Whom Died In-Hospital and Had Discordant Risk Categorization. Table S3. Description of Patients Whom Died After Discharge and Within 30-days of Presentation and Had Discordant Risk Categorization. (DOCX 35 kb)
About this article
Cite this article
Weeda, E.R., Kohn, C.G., Fermann, G.J. et al. External validation of prognostic rules for early post-pulmonary embolism mortality: assessment of a claims-based and three clinical-based approaches. Thrombosis J 14, 7 (2016). https://doi.org/10.1186/s12959-016-0081-5