Propensity Scores and Observational Research

We read with interest the study by Khanal et al published in the August issue of The American Journal of Medicine. Using a large comprehensive clinical registry of over 28000 subjects, these authors found that the use of hydroxymethylglutaryl coenzyme A (HmgCoA) inhibitors was associated with decreased odds of developing contrast-induced nephropathy (CIN) following cardiac catheterization. While the results of this observational study are interesting, we must point out some significant methodological limitations. Specifically, we are concerned that the definition of CIN was not appropriately applied, that substantial differences in baseline characteristics and study population sources led to incorrect inferences, and that the propensity score adjustment for selection bias was insufficient to account for these substantial differences.

This study attempts to compare the incidence of contrast-induced nephropathy in patients who were receiving versus patients who were not receiving HmgCoA inhibitors. The primary definition of CIN is stated as “an increase in serum creatinine of ≥0.5 mg/dL after the procedure.” While CIN is often defined as a rise in serum creatinine of this magnitude, a time course of 48-72 hours has been previously used as a cutoff for azotemia related to intravenous contrast.^{,  ,  ,  ,}  It is unclear from the methods used whether the incidence of azotemia noted in this population is purely related to the use of contrast, to other factors such as acute tubular necrosis, or to cardio-renal syndrome due to poor left ventricular function.

A second and potentially more important limitation relates to the comparability of the 2 populations studied. In Table 1, the authors provide descriptive data for “prestatin” and “no prestatin” patients. Although many of the baseline demographic and clinical characteristics (age, sex distribution, baseline renal function) are quite similar, there are a large number of factors that are both statistically and clinically different. Substantial differences exist in rate of cardiogenic shock, presence, and time course of previous myocardial infarction and indication for catheterization. For example, while 31% of the “no prestatin” group had cardiac catheterization performed for an emergent indication or for cardiac arrest, only 14.2% of the “prestatin” group shared these indications. It is conceivable that the “prestatin” group may represent primarily outpatients with known or suspected coronary artery disease, who underwent a planned interventional study, and would have, therefore, received precatheterization hydration or other potentially protective interventions. Those patients in the “No prestatin” group, on the other hand, may have been primarily emergency department referrals who had failed lytic therapy and had rescue percutaneous coronary interventions. The baseline “risk” for adverse outcomes among these populations is therefore quite different based on a number of factors (ie, confounders) other than the use of HmgCoA inhibitors. Many of these baseline differences are strong independent risks for developing renal failure following cardiac catheterization. Clearly, the source populations for the 2 groups are not comparable without careful adjustment for selection bias.

While the authors did attempt to statistically control for baseline differences through the use of a propensity score, several key aspects of regression modeling using a propensity score were insufficiently explored. First, it is unclear how specific variables were chosen for entry into the model for propensity to receive statin therapy. Several variables in the model, such as fluoroscopy time, maximal contrast volume, and intra-aortic balloon pump use, are not pretreatment covariates (and could thus easily introduce a substantial bias) and have nothing to do with the use or nonuse of statin therapy prior to presentation for cardiac catheterization. Secondly, to justify the use of a propensity adjustment in a regression model, baseline characteristics must substantially overlap between the exposure groups. For instance, the difference in the means of the propensity scores of the 2 groups being compared must be fairly small (a generally accepted rule is <1/2 of 1 standard deviation). While the mean propensity scores of the 2 groups are not presented, the bivariate odds ratio of 0.67 (P <.0001) in Table 4 suggests that the 2 groups substantially differ in their propensity to receive statin therapy at baseline. Verifying the overlap between groups is an essential part of using propensity methods to adjust for selection bias. If minimal overlap is observed, as in this case, a regression analysis on the entire sample using the propensity score is inappropriate. Either propensity matching or stratification would be a better choice, where the propensity score would be used to identify limited groups of patients from the populations who did have similar likelihood of receiving statin therapy. In this case, we cannot tell whether the observed differences in outcomes are solely a function of baseline differences in the patient populations and, thus, cannot draw any causal conclusion about the use of HmgCoA inhibitors and the odds of developing contrast induced nephropathy.