I n recent years, many studies have shown that evidence-based therapies do not always make it to the bedside. There are substantial variations in practice throughout the United States, and important gaps in the treatment of patients with common medical conditions (
- Ellerbeck E.F.
- Jencks S.F.
- Radford M.J.
- et al.
Quality of care for Medicare patients with acute myocardial infarction. A four-state pilot study from the Cooperative Cardiovascular Project.
- Krumholz H.M.
- Wang Y.
- Parent E.M.
- et al.
Quality of care for elderly patients hospitalized with heart failure.
- Krumholz H.M.
- Radford M.J.
- Wang Y.
- et al.
National use and effectiveness of beta-blockers for the treatment of elderly patients after acute myocardial infarction. National Cooperative Cardiovascular Project.
). However, the best way to reduce this variation and increase the implementation of evidence-based therapies is not well understood.
Major purchasers of care, such as the Health Care Financing Administration, have responded to concerns about quality of care by developing medical quality measures and by urging health care professionals and institutions to recognize opportunities to improve care and to develop interventions that do so (
- Jencks S.F.
- Wilensky G.R.
The health care quality improvement initiative. A new approach to quality assurance in Medicare.
). However, hospitals and health care professionals have little scientifically valid knowledge with which to guide their efforts. Medical research has generally focused on advancing clinical diagnosis and treatment. Less attention has been paid to studying how to implement this knowledge. Efforts to change patterns of care have generally been relegated to pharmaceutical representatives, quality assurance experts, and medical advocacy organizations and consultants. Recognizing the need for greater translation of medical knowledge to the bedside and the need to attract scientific inquiry to this topic, the Agency for Healthcare Research and Quality has devoted funds to projects designed to address this issue and improve the dissemination of best practices. These funds, however, are insufficient for current needs. Thus, in an era of evidence-based medicine, we have a lack of evidence-based strategies for implementing best practice.
In this issue of the American Journal of Medicine
, Philbin and colleagues (
- Philbin E.F.
- Rocco Jr, T.A.
- Lindenmuth N.W.
- et al.
The results of a randomized trial of a quality improvement intervention in the care of patients with heart failure.
) report the results of a randomized trial of a multifaceted quality improvement intervention directed at heart failure. Such studies are crucial for generating new knowledge about quality improvement strategies and specifically defining the essential features of successful programs for improving the care and outcomes of patients. The authors should be commended for organizing the study, enlisting the hospitals in the effort, and targeting a prevalent condition (heart failure) with clearly defined quality measures. The study was not able to demonstrate the value of the intervention, but does illustrate many of the challenges of performing these types of studies, thereby highlighting the need for novel methods that can generate knowledge necessary to enhance hospital performance. In particular, five challenges face those who undertake quality improvement studies.
First, and perhaps foremost, quality improvement studies often suffer from a lack of sufficient statistical power. The unit of analysis for these studies is the provider of care, not the patient. This has implications for the design and the analysis of a study. For example, Philbin and colleagues conducted their study with only ten hospitals. While bringing together ten hospitals to participate in the study was surely an organizational feat, the small number of randomized units limited the researchers’ ability to detect important differences between the groups. Perhaps as a result of limited power, a 1.1-day difference in length of stay and an $817 difference in hospital costs were not statistically significant. Thus, their disheartening findings may have been the result of design limitations, rather than indicating that the intervention was ineffective.
Randomization of clusters of patients—hospitals, clinics, or practices—is common in quality improvement studies. However, so-called “cluster-randomized” trials cannot, in general, be analyzed with methods used for patient-level randomization (
Statistical considerations in the design and analysis of community intervention trials.
). Ignoring clustering during analysis has the effect of “inflating” the sample size, typically producing exceedingly narrow confidence intervals and P values that are too small (
Analysis of a trial randomized in clusters.
). Analytic methods specific to clustered data are appropriate whenever the variance between clusters relative to the total variation in patient outcomes—the “intra-cluster correlation”—is much different from zero. This condition would be a concern in this study if, for example, variation in patients’ length of stay were partly explained by variation between hospitals, or, to put it differently, if the length of stay for patients with heart failure depended to some non-trivial extent on the hospital to which they were admitted. Philbin and colleagues did not report the intra-cluster correlation in their study; in the absence of such information, we should assume that the clustered analysis techniques that they used were necessary and appropriate. However, assuming only a moderate intra-cluster correlation of 1%, we can use their results to estimate (
The intracluster correlation coefficient in cluster randomisation.
) roughly that they had about 90% power to detect a 3.8-day difference in length of stay; the probability of not detecting a real 1-day difference in length of stay was almost 90%. The fundamental point is that using a large number of patients in a hospital-based quality improvement study cannot overcome the problem of having a small number of sites.
The small sample size also increases the difficulty of achieving two similar groups of hospitals. The effectiveness of a quality-improvement intervention depends not only on the intervention itself, but also the environment, culture, and organization of each hospital. The benefits of randomization depend on the law of large numbers, such that both measured and unmeasured factors are equally distributed among the two study groups. With small numbers it is possible that the intervention and usual care hospitals differed in important characteristics that were not incorporated into the analysis, potentially biasing the results. Unfortunately, the authors present little information about the hospital characteristics and environmental factors that may have influenced the success of their intervention.
The second difficulty with quality of care implementation studies is their vulnerability to contamination. With a baseline observation period for all hospitals before the intervention, it is not possible to have a true “usual care” group, since all hospitals knew that they were in a study assessing performance and were presumably exposed to some information about the study. If the contamination were substantial, intervention and non-intervention hospitals would perform similarly, and bias the study towards not finding an effect of the intervention. Indeed, the most important intervention may have been participation in the trial. One method of addressing this concern would be a comparison of performance at similar hospitals that did not participate in the study; without this information, it is difficult to assess whether contamination of the non-interventional hospitals may have affected the results.
Third, the ideal time to assess the effects of an intervention is not clear. How long does it take for a quality improvement intervention to change practice at an institution? In this study, the investigators wisely began their assessment of the post-intervention phase after all components of the intervention had been introduced. However, the results of the intervention period were presented as a single result. While this approach may have been necessary for design issues, it obscures potentially important information. Information about trends in practice during the study could demonstrate if an effort is gaining momentum—as evidenced by improving practice—or that it is not sustainable, as would be shown by diminishing performance.
Fourth, the generalizability of quality improvement studies requires careful scrutiny. Practice is often regional and all politics are local. Lichtman and colleagues (
Lichtman JH, Roumanis SA, Radford MJ, et al. Can practice guidelines be transported effectively to different settings? Results from a multicenter interventional study. Jt Comm J Qual Improv. (In Press).
) have described the difficulty of transporting guideline-based interventions beyond their original setting even in the best of circumstances. Philbin and colleagues included ten community hospitals, but provide no information about the hospital selection process. While they demonstrated that the patients in the study hospitals were similar to those admitted statewide, the more relevant issue concerns how the hospitals’ cultures and environments compare with others.
Fifth, quality improvement studies often contain so many different improvement strategies that it is impossible to know which aspects of the intervention were effective. Had this study yielded a positive result, readers may have wondered if the full combination of interventions were necessary to achieve the results, which interventions were more or less effective than others, or even if some aspects of the intervention might have been counterproductive. The lure of a multifaceted intervention is a result of uncertainty about what works and the interest in producing a positive result. However, the next generation of projects must identify the essential elements of a successful implementation program, with an emphasis on efficiency in an era of limited health care resources.
Hospitals and health care professionals require evidence-based approaches to guide their efforts to improve quality of care. The challenges of testing interventions and identifying effective strategies require the use of suitable methods, so that research resources are well-used and conclusions reliable, particularly with regard to adequate sample sizes and methods appropriate for the clustered design. We need to defend against contamination and other threats to the validity of a study. We need to attract investigators who are focused on generating new knowledge about how to ensure that high-quality medical evidence is disseminated quickly and effectively, and that all patients have the opportunity to receive the very best care. Only then can we expect to see the “more favorable changes in process and outcomes” that Philbin and colleagues hope to find.