Heterogeneity of Treatment Effects: Implications for Guidelines, Payment, and Quality Assessment


      Randomized controlled trial results are needed for developing guidelines, payment rules, and quality-of-care measures; however, 2 phenomena reduce the usefulness of randomized controlled trial findings. First, these studies now enroll patients with less severe disease, who are less likely to benefit from a drug or treatment. Second, patients are living longer but, as a result, have more chronic diseases. Although randomized controlled trials often exclude these older patients, trial findings continue to be generalized to them. Together, these phenomena impose challenges to the usefulness of the results of randomized controlled trials for clinical and policy applications. The convergence of these phenomena makes the current research paradigm underlying evidence-based medicine, guideline development and quality assessment fundamentally flawed in 2 ways. First, the “evidence” includes patients who may have a minimal benefit from the treatment being tested. This could reduce the power for the trial and yield negative or null results, leading to undertreatment of a group of patients with potential for a greater-than-observed benefit. Second, attempts to generalize the results from positive trials to patients who have been excluded from those trials may result in the overtreatment of those who could not benefit (e.g., because they will die from other causes before the benefit of treatment would occur) and therefore represents a parallel hazard. In this article, we describe sources of heterogeneity of treatment effects (HTE) within trials, which can compromise the interpretation and generalizability of results. We also examine strategies for understanding and managing HTE in practice, to increase the usefulness of trial results.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to The American Journal of Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Goldstein D.B.
        Pharmacogenetics in the laboratory and the clinic.
        N Engl J Med. 2003; 348: 553-556
        • Spear B.B.
        • Heath-Chiozzi M.
        • Huff J.
        Clinical application of pharmacogenetics.
        Trends Mol Med. 2001; 7: 201-204
        • Hayward R.A.
        • Kent D.M.
        • Vijan S.
        • Hofer T.P.
        Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis.
        BMC Med Res Methodol. 2006; 6: 18
        • Kravitz R.L.
        • Duan N.
        • Braslow J.
        Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages.
        Milbank Q. 2004; 82: 661-687
        • Bhatt D.L.
        • Fox K.A.
        • Hacke W.
        • et al.
        • CHARISMA Investigators
        Clopidogrel and aspirin versus aspirin alone for the prevention of atherothrombotic events.
        N Engl J Med. 2006; 354: 1706-1717
        • Terris S.
        Calcium plus vitamin D and the risk of fractures.
        N Engl J Med. 2006; 354: 2285-2287
        • Clegg D.O.
        • Reda D.J.
        • Harris C.L.
        • et al.
        Glucosamine, chondroitin sulfate, and the two in combination for painful knee osteoarthritis.
        N Engl J Med. 2006; 354: 795-808
        • Ernst E.
        Glucosamine and chondroitin sulfate for knee osteoarthritis.
        N Engl J Med. 2006; 354: 2184-2185
        • Jackson R.D.
        • LaCroix A.Z.
        • Gass M.
        • et al.
        Calcium plus vitamin D supplementation and the risk of fractures.
        N Engl J Med. 2006; 354: 669-683
        • Pfeffer M.A.
        • Jarcho J.A.
        The charisma of subgroups and the subgroups of CHARISMA.
        N Engl J Med. 2006; 354: 1744-1746
        • Brindis R.G.
        • Spertus J.
        The role of academic medicine in improving health care quality.
        Acad Med. 2006; 81: 802-806
        • Pronovost P.
        • Holzmueller C.G.
        • Needham D.M.
        • et al.
        How will we know patients are safer?.
        Crit Care Med. 2006; 34: 1988-1995
        • Chassin M.R.
        Does paying for performance improve the quality of health care?.
        Med Care Res Rev. 2006; 63: 122S-125S
        • Levenson S.
        “Pay for performance”: can it help improve long-term care?.
        J Am Med Dir Assoc. 2006; 7: 262-264
        • Goldfield N.
        • Burford R.
        • Averill R.
        • et al.
        Pay for performance: an excellent idea that simply needs implementation.
        Qual Manag Health Care. 2005; 14: 31-44
        • Petersen L.A.
        • Woodard L.D.
        • Urech T.
        • Daw C.
        • Sookanan S.
        Does pay-for-performance improve the quality of health care?.
        Ann Intern Med. 2006; 145: 265-272
        • Boyd C.M.
        • Darer J.
        • Boult C.
        • et al.
        Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance.
        JAMA. 2005; 294: 716-724
        • Tan K.B.
        Clinical practice guidelines: a critical review.
        Int J Health Care Qual Assur Inc Leadersh Health Serv. 2006; 19: 195-220
        • Philips Z.
        • Ginnelly L.
        • Sculpher M.
        • et al.
        Review of guidelines for good practice in decision-analytic modelling in health technology assessment.
        Health Technol Assess. 2004; 8 (1): iii-xi
        • Jamtvedt G.
        • Young J.M.
        • Kristoffersen D.T.
        • O’Brien M.A.
        • Oxman A.D.
        Audit and feedback: effects on professional practice and health care outcomes.
        Cochrane Database Syst Rev. 2006; : CD000259
        • Frolich A.
        • Talavera J.A.
        • Broadhead P.
        • Adams D.R.
        A behavioral model of clinician responses to incentives to improve quality.
        Health Policy. 2007; 80: 179-193
        • Albertsen P.C.
        • Fryback D.G.
        • Storer B.E.
        • Kolon T.F.
        • Fine J.
        Long-term survival among men with conservatively treated localized prostate cancer.
        JAMA. 1995; 274: 626-631
        • Litwin M.S.
        • Greenfield S.
        • Elkin E.P.
        • et al.
        Assessment of prognosis with the Total Illness Burden Index for Prostate Cancer (TIBI-CaP): aiding clinicians in treatment choice. 2006 (Presented at the annual meeting of the American Urological Association, May 20; Atlanta, Georgia.)
        • Greenfield S.
        • Kaplan S.H.
        • Nicolucci A.
        The diabetes patient’s “other” diseases lead to 5-year mortality. 2006 (Presented at the Midwest Society of General Internal Medicine, Quality Summit II. September 29; Chicago, Illinois.)
        • Longford N.T.
        Selection bias and treatment heterogeneity in clinical trials.
        Stat Med. 1999; 18: 1467-1474
        • Roses A.D.
        Pharmacogenetics and the practice of medicine.
        Nature. 2000; 405: 857-865
        • Vijan S.
        • Hofer T.P.
        • Hayward R.A.
        Estimated benefits of glycemic control in microvascular complications in type 2 diabetes.
        Ann Intern Med. 1997; 127: 788-795
        • CDC Diabetes Cost-effectiveness Group
        Cost-effectiveness of intensive glycemic control, intensified hypertension control, and serum cholesterol level reduction for type 2 diabetes.
        JAMA. 2002; 287: 2542-2551
        • Lip G.Y.
        • Lowe G.D.
        ABC of atrial fibrillation: antithrombotic treatment for atrial fibrillation.
        BMJ. 1996; 312: 45-49
        • Hayward R.A.
        • Kent D.M.
        • Vijan S.
        • Hofer T.P.
        Reporting clinical trial results to inform providers, payers, and consumers.
        Health Aff (Millwood). 2005; 24: 1571-1581
        • Garber K.
        Tamoxifen pharmacogenetics moves closer to reality.
        J Natl Cancer Inst. 2005; 97: 412-413
        • Sindrup S.H.
        • Brosen K.
        The pharmacogenetics of codeine hypoalgesia.
        Pharmacogenetics. 1995; 5: 335-346
        • McCormick A.W.
        • Whitney C.G.
        • Farley M.M.
        • et al.
        Geographic diversity and temporal trends of antimicrobial resistance in Streptococcus pneumoniae in the United States.
        Nat Med. 2003; 9: 424-430
        • Huppert J.D.
        • Schultz L.T.
        • Foa E.B.
        • et al.
        Differential response to placebo among patients with social phobia, panic disorder, and obsessive-compulsive disorder.
        Am J Psychiatry. 2004; 161: 1485-1487
        • Senn S.
        Individual response to treatment: is it a valid assumption?.
        BMJ. 2004; 329: 966-968
        • Materson B.J.
        Variability in response to antihypertensive drugs.
        Am J Med. 2007; 120: S10-S20
        • Carson III, C.C.
        Cardiac safety in clinical trials of phosphodiesterase 5 inhibitors.
        Am J Cardiol. 2005; 96: 37M-41M
        • Lassila H.C.
        • Stoehr G.P.
        • Ganguli M.
        • et al.
        Use of prescription medications in an elderly rural population: the MoVIES Project.
        Ann Pharmacother. 1996; 30: 589-595
        • Stoehr G.P.
        • Ganguli M.
        • Seaberg E.C.
        • Echement D.A.
        • Belle S.
        Over-the-counter medication use in an older rural community: the MoVIES Project.
        J Am Geriatr Soc. 1997; 45: 158-165
        • Routledge P.A.
        • O’Mahony M.S.
        • Woodhouse K.W.
        Adverse drug reactions in elderly patients.
        Br J Clin Pharmacol. 2004; 57: 121-126
        • Fortin M.
        • Dionne J.
        • Pinho G.
        • et al.
        Randomized controlled trials: do they have external validity for patients with multiple comorbidities?.
        Ann Fam Med. 2006; 4: 104-108
        • Rothwell P.M.
        • Warlow C.P.
        • European Carotid Surgery Trialists’ Collaborative Group
        Prediction of benefit from carotid endarterectomy in individual patients: a risk-modelling study.
        Lancet. 1999; 353: 2105-2110
        • Kent D.M.
        • Ruthazer R.
        • Selker H.P.
        Are some patients likely to benefit from recombinant tissue-type plasminogen activator for acute ischemic stroke even beyond 3 hours from symptom onset?.
        Stroke. 2003; 34: 464-467
        • Pocock S.J.
        • McCormack V.
        • Gueyffier F.
        • et al.
        A score for predicting risk of death from cardiovascular disease in adults with raised blood pressure, based on individual patient data from randomised controlled trials.
        BMJ. 2001; 323: 75-81
        • Slotman G.J.
        Prospectively validated prediction of organ failure and hypotension in patients with septic shock: the Systemic Mediator Associated Response Test (SMART).
        Shock. 2000; 14: 101-106
        • Stier D.M.
        • Greenfield S.
        • Lubeck D.P.
        • et al.
        Quantifying comorbidity in a disease-specific cohort: adaptation of the total illness burden index to prostate cancer.
        Urology. 1999; 54: 424-429
        • Zimmerman J.E.
        • Wagner D.P.
        • Draper E.A.
        • et al.
        Evaluation of acute physiology and chronic health evaluation III predictions of hospital mortality in an independent database.
        Crit Care Med. 1998; 26: 1317-1326
        • Selker H.P.
        • Griffith J.L.
        • Beshansky J.R.
        • et al.
        Patient-specific predictions of outcomes in myocardial infarction for real-time emergency use: a thrombolytic predictive instrument.
        Ann Intern Med. 1997; 127: 538-556
        • Lagakos S.W.
        The challenge of subgroup analyses—reporting without distorting.
        N Engl J Med. 2006; 354: 1667-1669
        • Kraemer H.C.
        • Kupfer D.J.
        Size of treatment effects and their importance to clinical research and practice.
        Biol Psychiatry. 2006; 59: 990-996
        • Kraemer H.C.
        • Frank E.
        • Kupfer D.J.
        Moderators of treatment outcomes: clinical, research, and policy importance.
        JAMA. 2006; 296: 1286-1289
        • Vijan S.
        • Hayward R.A.
        Treatment of hypertension in type 2 diabetes mellitus: blood pressure goals, choice of agents, and setting priorities in diabetes care.
        Ann Intern Med. 2003; 138: 593-602