Advertisement

Assessing hospital outcomes: it’s time to get inside the black box

  • Jonathan Showstack
    Correspondence
    Requests for reprints should be addressed to Jonathan Showstack, PhD, MPH, Institute for Health Policy Studies and Department of Medicine, University of California, San Francisco, School of Medicine, Box 0936, San Francisco, California 94143-0936, USA
    Affiliations
    Institute for Health Policy Studies and Department of Medicine, School of Medicine, University of California, San Francisco, San Francisco, California, USA
    Search for articles by this author
      The efficacy of advances in clinical medicine is often less clear in actual practice than in clinical trials. Outcomes research, which examines effectiveness, particularly the structures and processes that are associated with better outcomes (
      • Brook R.H.
      • McGlynn E.A.
      • Cleary P.D.
      Quality of health care. Part 2 Measuring quality of care.
      ), has found that the wide variations observed in outcomes and use of services often seem to be associated more with factors such as where a patient lives than with the clinical characteristics of a patient (
      • Wennberg J.E.
      Dealing with medical practice variations a proposal for action.
      ). As pointed out in two recent Institute of Medicine reports, quality of care (or the lack of it) has become an important issue in the U.S. health care system (
      ,
      Committee on Quality of Health Care in America, Institute of Medicine
      ).
      While several provider characteristics, such as hospital teaching status and the volume of certain procedures, appear to be associated with quality of care and outcomes, linking individual providers and practices to these outcomes remains a difficult and challenging task (
      • Luft H.S.
      • Hunt S.S.
      Evaluating individual hospital quality through outcomes statistics.
      ). After almost two decades of results that suggest that the volume of certain procedures (such as open heart surgery) is positively associated with outcomes, we are not much closer to being able to identify the causal mechanism for this finding. Perhaps because of the lack of certainty about specific causation, the provider community has been reluctant to be judged on the basis of a corollary factor (such as volume) as compared with direct clinical observation of outcomes (
      • Sheikh K.
      Utility of provider volume as an indicator of medical care quality and for policy decisions.
      ). On the other hand, academic health centers and teaching hospitals are more than eager to announce the results of studies, whether of empirical data or based on opinion, that suggest their superiority to nonteaching hospitals (
      • Kassirer J.P.
      Hospitals, heal yourselves.
      ).
      The article by Polanczyk and colleagues in this issue of The American Journal of Medicine adds new evidence to the debate about the value of teaching hospitals (
      • Polanczyk C.A.
      • Lane A.
      • Coburn M.
      Hospital outcomes in major teaching, minor teaching, and nonteaching hospitals in New York State.
      ). Based on an analysis of secondary data, they suggest that “large” teaching hospitals may have better outcomes on average than do nonteaching hospitals, but that outcomes at “small” teaching hospitals may actually be worse than at nonteaching hospitals. The authors, and we, are left to speculate about the causal factors that may have led to these intriguing, if somewhat contradictory, results.
      One of the basic methodological problems associated with studies of the association of teaching status with outcomes is the lack of a precise definition of “teaching status.” Which characteristics of teaching hospitals, and in what amounts, are associated with improved outcomes? Even with relatively precise definitions of teaching status derived from secondary data, for example, the number of residents per occupied bed, the causal model remains unclear. Teaching status is associated with factors other than the presence of students and residents that may be causally associated with outcomes, including the availability of specialists and advanced technologies, the size of the hospital, the number of cases seen and procedures performed, and the availability of similar facilities in the same geographic area. While outcomes studies have attempted to adjust for these other factors using available secondary data, “teaching status” has remained poorly defined and an unopened black box.
      The use of broad, nonspecific terms to define complex concepts can lead to both logical and scientific problems. In recent years, many articles have addressed the effects of “managed care” on patient access and health care costs, as well as of service “volume” and “teaching status” on hospital outcomes. The use of these usually poorly defined concepts does little to enhance our understanding of why patients do or do not have access to needed care, the factors that affect the costs of care, or the structures and processes that lead to better outcomes. Similarly, public opinion can be swayed, and policies and programs may be based on the misperceptions that are created, or at least encouraged, by loose definitions of key concepts and issues.
      Investigations into quality of care have tended to err on the side of specificity, often to the neglect and detriment of sensitivity. This focus is logical given the relatively insensitive and nonspecific instruments available to identify individual providers as poor performers. In general, what we are able to detect is a statistically significant variation from the average; yet providers argue, with some cause, that a significant (poor) result may be due as much or more to circumstances that are beyond the control of the provider as to the quality of the care provided. Overall relations that are based on average outcomes may be informative, but the predictive value of a positive result for a specific provider is often poor. While we should not abandon the analysis of secondary data, progress in assessing outcomes can be made only if we open the black box and undertake studies that are designed to identify the specific causal mechanisms that lead to better outcomes (
      • Halm E.A.
      • Chassin M.R.
      Why do hospital death rates vary?.
      ).
      One possible answer to this dilemma is to adopt the lessons and techniques of epidemiology and the population health sciences. When an event is unlikely, mass screening is rarely useful, particularly because low prevalence is associated with poor predictive values, even with relatively sensitive and specific tests (
      • Galen R.S.
      • Gambino S.R.
      ). Given the poor predictive value of many diagnostic tests (and outcomes studies) when applied to broad populations, and within defined and often constrained resources, what type of surveillance systems should be created to identify and improve poor quality care? Two basic and related answers are to identify and respond to “sentinel indicators” of a possible problem and to enrich the population in which the testing is performed.
      Applying these lessons to outcomes and effectiveness research suggests that a poorer than average outcome by a provider should not be necessarily defined as an indicator of poor quality, but rather as a sentinel event that deserves further investigation. Sentinel events can be used to narrow the focus to sites that have a relatively high probability of poor quality care and then to investigate these instances in greater depth. Secondary data might be used for identifying the suspect site; once identified, however, the equivalent of the Centers for Disease Control’s Epidemic Intelligence Service (perhaps an “Effectiveness Intelligence Service”) could be sent to the site to conduct a more in-depth investigation (), focusing on whether the event occurred and identifying ways to prevent further occurrences. The site could then be monitored for further “outbreaks” of poor outcomes.
      The new Effectiveness Intelligence Service would have several advantages. The investigation of a sentinel indicator would be routine and indicate suspicion, not guilt. The investigation would be independent; hospitals would not be required to judge themselves, thus avoiding the associated potential for subjective and possibly biased conclusions. Furthermore, the investigation could avoid a priori assumptions about the possible effects of structures and processes on outcomes. The focus would be on the correction of specific structural and process deficiencies as well as the identification of broader lessons and models. The new service would not replace, but rather complement, current evidence-based methods to ensure quality care for patients (
      • Dudley R.A.
      • Mangione C.M.
      Physician responses to evidence-based hospital referral programs.
      ).
      Opening up the black box requires that the complex causal paths that lead to patient outcomes be defined and that summary concepts be abandoned. Patients and providers should not settle for treatment recommendations that are based on personal knowledge and little empirical data or on assumptions about the value of a general characteristic of a provider (e.g., teaching status). Ultimately, treatment decisions should be based on an understanding of the probability of a specific outcome given the characteristics of individual patients and providers. Patients and providers deserve no less.

      References

        • Brook R.H.
        • McGlynn E.A.
        • Cleary P.D.
        Quality of health care. Part 2.
        N Engl J Med. 1996; 335: 966-970
        • Wennberg J.E.
        Dealing with medical practice variations.
        Health Aff (Millwood). 1984; 3: 6-32
      1. Kohn L.T. Corrigan J.M. Donaldson M.S. To Err Is Human Building a Safer Health System. National Academy Press, Washington, DC2000
        • Committee on Quality of Health Care in America, Institute of Medicine
        Crossing the Quality Chasm. National Academy Press, Washington, DC2001
        • Luft H.S.
        • Hunt S.S.
        Evaluating individual hospital quality through outcomes statistics.
        JAMA. 1986; 255: 2780-2784
        • Sheikh K.
        Utility of provider volume as an indicator of medical care quality and for policy decisions.
        Am J Med. 2001; 111: 712-715
        • Kassirer J.P.
        Hospitals, heal yourselves.
        N Engl J Med. 1999; 340: 309-310
        • Polanczyk C.A.
        • Lane A.
        • Coburn M.
        Hospital outcomes in major teaching, minor teaching, and nonteaching hospitals in New York State.
        Am J Med. 2002; 112: 255-261
        • Halm E.A.
        • Chassin M.R.
        Why do hospital death rates vary?.
        N Engl J Med. 2001; 345: 692-694
        • Galen R.S.
        • Gambino S.R.
        Beyond Normality. John Wiley & Sons, New York, New York1975
        • Centers for Disease Control
        (Accessed December 28, 2001)
        • Dudley R.A.
        • Mangione C.M.
        Physician responses to evidence-based hospital referral programs.
        Am J Med. 2001; 111: 731-733