Advertisement

Optimize Your Electronic Medical Record to Increase Value: Reducing Laboratory Overutilization

Published:October 14, 2015DOI:https://doi.org/10.1016/j.amjmed.2015.09.009

      Abstract

      Purpose

      The purpose of this study is to decrease overutilization of laboratory testing by eliminating a feature of the electronic ordering system that allowed providers to order laboratory tests to occur daily without review.

      Methods

      We collected rates of utilization of a group of commonly ordered laboratory tests (number of tests per patient per day) throughout the entire hospital from June 10, 2013 through June 10, 2015. Our intervention, which eliminated the ability to order daily recurring tests, was implemented on June 11, 2014. We compared pre- and postintervention rates in order to assess the impact and surveyed providers about their experience with the intervention.

      Results

      We examined 1,296,742 laboratory tests performed on 92,799 unique patients over 434,059 patient days. Before the intervention, the target tests were ordered using this daily recurring mechanism 33% of the time. After the intervention we observed an 8.5% (P <.001) to 20.9% (P <.001) reduction in tests per patient per day. The reduction in rate for some of the target tests persisted during the study period, but not for the 2 most commonly ordered tests. We estimated an approximate reduction in hospital costs of $300,000 due to the intervention.

      Conclusion

      A simple modification to the order entry system significantly and immediately altered provider practices throughout a large tertiary care academic center. This strategy is replicable by the many hospitals that use the same electronic health record system, and possibly, by users of other systems. Future areas of study include evaluating the additive effects of education and real-time decision support.

      Keywords

      Clinical Significance
      • Waste and overuse of diagnostic testing have severe effects on both quality of care and costs to the health care system.
      • Some electronic order entry systems may have features that facilitate laboratory overutilization by allowing providers to order daily recurring tests.
      • Elimination of features in electronic health record systems that facilitate overutilization is likely safe and can lead to reduction in hospital costs.
      Hospitals are under increased pressure to lower costs without compromising care. Changing how and why providers order laboratory tests presents an opportunity for hospitals to improve value. Laboratory service utilization has increased in recent decades,
      • Mindemark M.
      • Larsson A.
      Longitudinal trends in laboratory test utilization at a large tertiary care university hospital in Sweden.
      • Valenstein P.N.
      • Praestgaard A.H.
      • Lepoff R.B.
      Six-year trends in productivity and utilization of 73 clinical laboratories: a College of American Pathologists Laboratory Management Index Program study.
      but a substantial proportion of tests may be unnecessary. Waste and overuse of diagnostic testing have severe effects on both quality of care and costs to the health care system, and their reduction is a focus of the American Board of Internal Medicine's “Choosing Wisely Campaign.”

      ABIM Foundation. Choosing wisely. Available at: http://choosingwisely.org. Accessed August 10, 2015.

      Results from a pilot study at New York University Langone Medical Center (NYULMC) demonstrated that >95% of patients on the Internal Medicine service had laboratory tests ordered on the first day of admission as daily recurring orders in the electronic health record (EHR). As a result, the same laboratory tests were repeated every day during the hospitalization without any requirement to review the appropriateness of the test. While viewed as a time-saving measure by providers, this practice possibly led to increased unnecessary testing.
      There have been several studies that have evaluated approaches to address the problem of overutilization of laboratory testing, some of which include provider education, audit and feedback of lab ordering, implementation of lab-ordering guidelines, restrictions placed on lab ordering, and displaying the cost of labs at time of order placement in the EHR.
      • Grivell A.R.
      • Forgie H.J.
      • Fraser C.G.
      • Berry M.N.
      Effect of feedback to clinical staff of information on clinical biochemistry requesting patterns.
      • Sidel V.W.
      • Mahler D.M.
      • Veatch R.M.
      Modification of residents' test-ordering behavior.
      • Cummings K.M.
      • Frisof K.B.
      • Long M.J.
      • Hrynkiewich G.
      The effects of price information on physicians' test-ordering behavior. Ordering of diagnostic tests.
      • Kroenke K.
      • Hanley J.F.
      • Copley J.B.
      • et al.
      Improving house staff ordering of three common laboratory tests. Reductions in test ordering need not result in underutilization.
      • Davidoff F.
      • Goodspeed R.
      • Clive J.
      Changing test ordering behavior. A randomized controlled trial comparing probabilistic reasoning with cost-containment education.
      • Bareford D.
      • Hayling A.
      Inappropriate use of laboratory services: long term combined approach to modify request patterns.
      • Studnicki J.
      • Bradham D.D.
      • Marshburn J.
      • Foulis P.R.
      • Straumfjord J.V.
      A feedback system for reducing excessive laboratory tests.
      • Beilby J.J.
      • Silagy C.A.
      Trials of providing costing information to general practitioners: a systematic review.
      • Bates D.W.
      • Kuperman G.J.
      • Rittenberg E.
      • et al.
      A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests.
      • Hampers L.C.
      • Cha S.
      • Gutglass D.J.
      • Krug S.E.
      • Binns H.J.
      The effect of price information on test-ordering behavior and patient outcomes in a pediatric emergency department.
      • Sucov A.
      • Bazarian J.J.
      • deLahunta E.A.
      • Spillane L.
      Test ordering guidelines can alter ordering patterns in an academic emergency department.
      • Bunting P.S.
      • Van Walraven C.
      Effect of a controlled feedback intervention on laboratory test ordering by community physicians.
      • Neilson E.G.
      • Johnson K.B.
      • Rosenbloom S.T.
      • et al.
      The impact of peer management on test-ordering behavior.
      • Calderon-Margalit R.
      • Mor-Yosef S.
      • Mayer M.
      • Adler B.
      • Shapira S.C.
      An administrative intervention to improve the utilization of laboratory tests within a university hospital.
      • May T.A.
      • Clancy M.
      • Critchfield J.
      • et al.
      Reducing unnecessary inpatient laboratory testing in a teaching hospital.
      • Roshanov P.S.
      • You J.J.
      • Dhaliwal J.
      • et al.
      Can computerized clinical decision support systems improve practitioners' diagnostic test ordering behavior? A decision-maker-researcher partnership systematic review.
      • Pageler N.M.
      • Franzon D.
      • Longhurst C.A.
      • et al.
      Embedding time-limited laboratory orders within computerized provider order entry reduces laboratory utilization.
      • Waldron J.L.
      • Ford C.
      • Dobie D.
      • et al.
      An automated minimum retest interval rejection rule reduces repeat CRP workload and expenditure, and influences clinician-requesting behaviour.
      • Goetz C.
      • Rotman S.R.
      • Hartoularos G.
      • Bishop T.F.
      The effect of charge display on cost of care and physician practice behaviors: a systematic review.
      • Vidyarthi A.R.
      • Hamill T.
      • Green A.L.
      • Rosenbluth G.
      • Baron R.B.
      Changing resident test ordering behavior: a multilevel intervention to decrease laboratory utilization at an academic medical center.
      • Corson A.H.
      • Fan V.S.
      • White T.
      • et al.
      A multifaceted hospitalist quality improvement intervention: decreased frequency of common labs.
      There have been varying degrees of success shown with each of the approaches, with some suggestion that combined approaches have the highest likelihood of success.
      • Solomon D.H.
      • Hashimoto H.
      • Daltroy L.
      • Liang M.H.
      Techniques to improve physicians' use of diagnostic tests: a new conceptual framework.
      • Baird G.
      The laboratory test utilization management toolbox.
      Combined approaches, however, are limited by their additional requirements for labor and resources. We found 2 studies that reported reductions in laboratory utilization after elimination of the ability to order repeating tests in the computerized order entry system: one localized to a pediatric intensive care unit that showed approximately 30% significant reduction in testing of 3 commonly ordered labs,
      • Pageler N.M.
      • Franzon D.
      • Longhurst C.A.
      • et al.
      Embedding time-limited laboratory orders within computerized provider order entry reduces laboratory utilization.
      and one on the general inpatient wards, which showed a 12% reduction in laboratory test utilization overall.
      • May T.A.
      • Clancy M.
      • Critchfield J.
      • et al.
      Reducing unnecessary inpatient laboratory testing in a teaching hospital.
      The objective of this study was to reduce the utilization of laboratory tests by eliminating the ability to order daily recurring labs in the EHR.

      Patients and Methods

      We conducted a mixed-methods study to assess the impact of this intervention. The study took place at NYULMC, an academic medical center with a main hospital that has 705 acute inpatient beds and an orthopedic/rehabilitation subspecialty hospital with an additional 225 beds. In March 2013, NYULMC implemented Epic (Verona, WI) as the EHR and order entry system for all inpatient services. Notification to hospital clinical staff impacted by this change was made at department service chiefs' meetings, via e-mail notification to all staff, and through multiple departmental meetings. The change in the EHR took place on June 11, 2014. This intervention was institution wide and impacted every patient on all inpatient services in the medical center.
      Our primary outcome measure was the rate of laboratory utilization. We defined our target tests for this study as a set of commonly ordered laboratory studies that had a high likelihood of being ordered in this daily recurring manner based on our pilot study: Basic Metabolic Profile, Magnesium, Phosphorus, Complete Blood Count, Hepatic Panel, and Prothrombin Time with International Normalized Ratio. We collected a dataset of all these laboratory tests performed from June 10, 2013 through June 10, 2015.
      When making comparisons of laboratory test utilization before and after the intervention, we examined the ratio of laboratory tests performed to number of inpatients in the hospital on a given day in order to account for variation in hospital census. This generated a metric defined as the number of tests per patient per day.
      Secondary outcomes include hospital costs and possible adverse patient outcomes. Because the direct costs associated with laboratory testing can vary substantially between hospitals, we chose to derive an order of magnitude estimate of possible cost savings associated with this intervention by using prices for laboratory tests obtained from the Centers for Medicare and Medicaid Services Clinical Laboratory Fee Schedule.

      Centers for Medicare & Medicaid Services. Clinical laboratory fee schedule. Available at: http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/ClinicalLabFeeSched/. Accessed August 10, 2015.

      We calculated and report the cost of the tests that would have occurred had rates before the intervention persisted.
      In addition to the quantitative assessment, we also performed a qualitative evaluation of the intervention via a survey of providers who were impacted by the intervention (Table 1). The objective of the survey was to assess provider experience and perception of the EHR modification. We also asked them to report any observed adverse patient outcomes associated with the change. The survey was sent out via e-mail to 200 house officers, nurse practitioners, and hospitalists who represent the range of medical and surgical subspecialties throughout the hospital. Within this group of 200 providers there is some unidentified number who did not experience ordering in the system before the intervention.
      Table 1Qualitative Survey of Experience with the Intervention
      • 1.
        Role? Nurse Practitioner, Housestaff, Attending
      • 2.
        Did you order lab tests on patients prior to 6/11/2014 and afterward? Yes/No
      • 3.
        Were you in favor of the change that eliminated the ability to order daily labs? Yes/No
      • 4.
        What has been your experience with this change?
      • 5.
        Have there been any negative patient outcomes?
      • 6.
        Did this change make you more thoughtful about which labs are necessary for inpatient care?
      Finally, we obtained data from NYULMC's quality dashboard about average risk-adjusted length of stay and mortality, as well as rates of readmission within 30 days in the pre- and postintervention time periods. The hospital utilizes the University Health System Consortium's methodology for defining risk adjustment.

      University Health System Consortium. 2014 Risk-adjustment models. Available at: https://www.uhc.edu/26295. Accessed August 11, 2015.

      Statistical Analysis

      The ratio of tests per patient per day was compared pre- and postintervention as a continuous variable with an approximately normal distribution using Student's t test. Additionally, we used an interrupted time series design and performed segmented regression analysis, which divided our time series into pre- and postintervention segments.
      • Wagner A.K.
      • Soumerai S.B.
      • Zhang F.
      • Ross-Degnan D.
      Segmented regression analysis of interrupted time series studies in medication use research.
      We selected June 11 to be the intersection between our pre and post segments because that is when our intervention was initiated. The differences between the 2 time periods were assessed by evaluating 2 parameters: the change in level of ordering tests after the intervention and the trends in ordering tests before and after the intervention. We also included a term in our model for the day of the week that the tests were performed in order to account for the weekly periodic ordering pattern that emerged in our dataset. Additionally, we corrected for autocorrelation by using a second-order autoregressive model. All statistical analysis was performed using R.
      • Venables W.N.
      • Smith D.M.
      R Development Core Team
      An Introduction to R: Notes on R: A Programming Environment for Data Analysis and Graphics, version 1.4.1.

      Results

      We examined 1,296,742 of our target lab tests performed on 92,799 unique patients over 434,059 patient days between June 1, 2013 and June 10, 2015. Before the intervention, our target labs were ordered using the daily recurring mechanism 33% of the time, and this ranged from 26.2% to 48.9%.
      Using Student's t test we noted a significant reduction in the number of target labs performed per patient per day as a group (−8.5%, P-value <.001). Neither the basic metabolic profile nor the complete blood count showed a significant change, while the rest of the individual components of the target lab tests showed statistically significant reductions. Our estimate of cost savings for the year using the percent reduction of only the laboratory tests that were significantly changed was $323,489 (Table 2).
      Table 2Pre and Post Lab Test Utilization Rates per Patient Day
      Ordered as Daily Recurring PreinterventionPreinterventionPostinterventionChangeP-Value (t Test)Estimated Cost Savings
      Basic metabolic profile30.5%0.820.830.08%.890
      Complete blood count26.2%0.930.92−1.32%.100
      PT/INR36.2%0.320.26−19.71%<.001$62,974
      Hepatic panel31.0%0.290.25−11.63%<.001$76,159
      Magnesium41.9%0.460.42−10.28%<.001$85,367
      Phosphorus48.9%0.290.18−37.34%<.001$98,989
      All target lab tests33.0%3.122.86−8.52%<.001$323,489
      INR = international normalized ratio; PT = prothrombin time.
      Using segmented regression analysis, we found that there was a 20.9% reduction in the utilization of the target lab tests (P-value <.001) as a result of the intervention. There was a small but significant downward trend in the utilization of these tests before the intervention (−0.0002, P = .003). After the intervention there was a small but significant upward trend (0.0004, P ≤.001) (Figure 1).
      Figure thumbnail gr1
      Figure 1Segmented regression analysis, target laboratory tests.
      All individual target laboratory tests showed a statistically significant level change in rate of ordering after the intervention. Magnesium and phosphorus showed a small but significant negative trend in the rate of ordering before the intervention; other tests showed no significant trend. All tests, with the exception of international normalized ratio, showed small but statistically significant positive trends in utilization after the intervention. The rates of both basic metabolic profile and complete blood count returned to preintervention levels (Figure 2).
      Figure thumbnail gr2
      Figure 2Segmented regression analysis, individual laboratory tests. INR = international normalized ratio; PT = prothrombin time.
      There were 28 survey respondents, 10 house officers, 17 nurse practitioners, and one attending. The one attending physician documented not having ordered lab tests before and after the intervention, whereas all other respondents experienced the system before and after the change. Eight of 28 respondents reported that they were in favor of this intervention. Twelve respondents reported negatively about their experience with the intervention. These negative experiences included having more work created for the end users who are very busy, as well as statements about how in specific cases, managing postoperative patients, it is necessary to order daily laboratory tests. The 6 respondents who answered with a favorable description of the experience described their opinion that there are a lot of wasteful laboratory test-ordering practices and that the intervention increased thoughtfulness. Six respondents stated that there have been negative patient outcomes, and describe these as delays in patient care when tests were accidentally not ordered on patients, increased needlesticks when lab tests were ordered after the routine morning phlebotomy time, and decreased patient satisfaction. The 13 respondents who described no negative patient outcome reported that were no “serious” adverse events, but they too reported that in some cases laboratory tests were ordered later in the morning because tests had not been ordered the night before. Thirteen respondents stated that the intervention made them more thoughtful, whereas 11 stated that it had not. Those that reported no increased thoughtfulness reiterated their discontent about reduced productivity. One respondent simply stated, “Please bring back daily labs.”
      We observed that expected length of stay (pre: 0.97, post: 0.93) and mortality (pre: 0.5, post: 0.49), as well as rates of readmission within 30 days (pre: 7.39%, post: 7%), were all lower after the intervention.

      Discussion

      Eliminating the ability to order daily recurring laboratory tests across the board for all tests and in all inpatient locations resulted in a significant decrease in the use of the commonly ordered set of target laboratory tests that we identified. This study shows that a simple change to the user interface of the electronic order entry system can decrease the rate of laboratory utilization in the short run and in the case of some of our laboratory tests retain the reduction over time.
      Not surprisingly, there was an initial decrease in the rate of utilization of the target laboratory tests right after the intervention. During the postintervention period the rates of utilization for the 2 most commonly ordered tests—basic metabolic panel and complete blood count—returned to the preintervention levels. Right after the intervention, rates for the other tests had a trend toward returning to the prior levels, but appear to plateau at a lower rate than before the intervention. This suggests that after the initial decrease in ordering after the intervention, providers made a considered decision that the rate of utilization for the basic metabolic panel and complete blood count was likely at an appropriate level before the intervention. Their behavior seems to suggest, however, that the other laboratory tests were in fact overutilized before the intervention.
      In order to consider possible adverse consequences related to the intervention, we examined patient outcomes that are routinely collected in hospitals. Because these metrics are influenced by many confounding variables such as other interventions that are taking place in the hospital to actively reduce these metrics, it was unlikely that the study would reveal a causal relationship to the intervention and these metrics. We reviewed them, however, to make sure that there did not appear to be a large negative impact of the intervention reflected in increases in these metrics around the time of the intervention. We were in fact reassured because these patient outcome metrics decreased in the postintervention time period.
      With the limitations in the quantitative evaluation in mind, we attempted to assess adverse patient outcomes related to the intervention by asking providers to describe negative patient outcomes as they have seen them. This qualitative approach is limited by responder bias and recollection bias, as well as the subjectivity of interpretation of the responses, however, it provides us with a way to capture some aspect of negative outcomes that may have occurred.
      The survey did not reveal specific patient harm due to the intervention, although it elucidated concern on the part of the providers that patients have suffered a blood test being performed later than the usual morning phlebotomy time because laboratory tests were accidentally not ordered the night before. The survey of providers' experience with the intervention reveals that a substantial proportion of providers do not agree with the need for the intervention because their perception is that the change contributes to decreased productivity because it added daily tasks that were formerly managed one time per patient admission.
      A limitation of the study is that we did not have a control group of laboratory tests because the intervention was across the board for all tests and for all inpatients in the hospital. We looked at a set of laboratory tests that were not commonly ordered using the daily ordering mechanism and noted no reduction in rate of utilization after the intervention. This is a flawed control, however, because these tests are fundamentally different from our target labs in that they were ordered much less frequently in general. Despite lacking a control, the clear chronological juxtaposition of the intervention and the decrease in testing rate suggests that the intervention was likely responsible for the reductions that we have elucidated.
      An additional observation we made when analyzing the data is that there is a weekly ordering pattern that was consistent throughout all the tests, which showed a decrease in laboratory test utilization per patient day on weekends with a spike on Monday. This pattern has been described before
      • Cheng C.K.
      • Lee T.
      • Cembrowski G.S.
      Temporal approach to hematological test usage in a major teaching hospital.
      and further reinforces the idea that there is overutilization of laboratory testing because fewer tests are deemed necessary on weekends for the same patients with the same medical conditions.
      This study demonstrates that a simple modification to the electronic order entry system has the capacity to alter provider practices that ultimately decreased costs of hospitalizations and likely did not cause serious patient harm. The intervention did create dissatisfaction with a substantial proportion of providers who view it as causing them to have more trivial time consuming tasks added to their workload. This technique is replicable by the many hospitals that use the same EHR system and possibly by users of other systems. Future areas of study include evaluating the additive effects of education, real-time decision support, and cost reporting to this simple intervention.

      Acknowledgment

      We thank Meg Ferrauiola, Senior Epic Analyst at NYU Langone Medical Center, for her technical assistance.

      References

        • Mindemark M.
        • Larsson A.
        Longitudinal trends in laboratory test utilization at a large tertiary care university hospital in Sweden.
        Ups J Med Sci. 2011; 116: 34-38
        • Valenstein P.N.
        • Praestgaard A.H.
        • Lepoff R.B.
        Six-year trends in productivity and utilization of 73 clinical laboratories: a College of American Pathologists Laboratory Management Index Program study.
        Arch Pathol Lab Med. 2001; 125: 1153-1161
      1. ABIM Foundation. Choosing wisely. Available at: http://choosingwisely.org. Accessed August 10, 2015.

        • Grivell A.R.
        • Forgie H.J.
        • Fraser C.G.
        • Berry M.N.
        Effect of feedback to clinical staff of information on clinical biochemistry requesting patterns.
        Clin Chem. 1981; 27: 1717-1720
        • Sidel V.W.
        • Mahler D.M.
        • Veatch R.M.
        Modification of residents' test-ordering behavior.
        N Engl J Med. 1981; 304: 918-919
        • Cummings K.M.
        • Frisof K.B.
        • Long M.J.
        • Hrynkiewich G.
        The effects of price information on physicians' test-ordering behavior. Ordering of diagnostic tests.
        Med Care. 1982; 20: 293-301
        • Kroenke K.
        • Hanley J.F.
        • Copley J.B.
        • et al.
        Improving house staff ordering of three common laboratory tests. Reductions in test ordering need not result in underutilization.
        Med Care. 1987; 25: 928-935
        • Davidoff F.
        • Goodspeed R.
        • Clive J.
        Changing test ordering behavior. A randomized controlled trial comparing probabilistic reasoning with cost-containment education.
        Med Care. 1989; 27: 45-58
        • Bareford D.
        • Hayling A.
        Inappropriate use of laboratory services: long term combined approach to modify request patterns.
        BMJ. 1990; 301: 1305-1307
        • Studnicki J.
        • Bradham D.D.
        • Marshburn J.
        • Foulis P.R.
        • Straumfjord J.V.
        A feedback system for reducing excessive laboratory tests.
        Arch Pathol Lab Med. 1993; 117: 35-39
        • Beilby J.J.
        • Silagy C.A.
        Trials of providing costing information to general practitioners: a systematic review.
        Med J Aust. 1997; 167: 89-92
        • Bates D.W.
        • Kuperman G.J.
        • Rittenberg E.
        • et al.
        A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests.
        Am J Med. 1999; 106: 144-150
        • Hampers L.C.
        • Cha S.
        • Gutglass D.J.
        • Krug S.E.
        • Binns H.J.
        The effect of price information on test-ordering behavior and patient outcomes in a pediatric emergency department.
        Pediatrics. 1999; 103: 877-882
        • Sucov A.
        • Bazarian J.J.
        • deLahunta E.A.
        • Spillane L.
        Test ordering guidelines can alter ordering patterns in an academic emergency department.
        J Emerg Med. 1999; 17: 391-397
        • Bunting P.S.
        • Van Walraven C.
        Effect of a controlled feedback intervention on laboratory test ordering by community physicians.
        Clin Chem. 2004; 50: 321-326
        • Neilson E.G.
        • Johnson K.B.
        • Rosenbloom S.T.
        • et al.
        The impact of peer management on test-ordering behavior.
        Ann Intern Med. 2004; 141: 196-204
        • Calderon-Margalit R.
        • Mor-Yosef S.
        • Mayer M.
        • Adler B.
        • Shapira S.C.
        An administrative intervention to improve the utilization of laboratory tests within a university hospital.
        Int J Qual Health Care. 2005; 17: 243-248
        • May T.A.
        • Clancy M.
        • Critchfield J.
        • et al.
        Reducing unnecessary inpatient laboratory testing in a teaching hospital.
        Am J Clin Pathol. 2006; 126: 200-206
        • Roshanov P.S.
        • You J.J.
        • Dhaliwal J.
        • et al.
        Can computerized clinical decision support systems improve practitioners' diagnostic test ordering behavior? A decision-maker-researcher partnership systematic review.
        Implement Sci. 2011; 6: 88
        • Pageler N.M.
        • Franzon D.
        • Longhurst C.A.
        • et al.
        Embedding time-limited laboratory orders within computerized provider order entry reduces laboratory utilization.
        Pediatr Crit Care Med. 2013; 14: 413-419
        • Waldron J.L.
        • Ford C.
        • Dobie D.
        • et al.
        An automated minimum retest interval rejection rule reduces repeat CRP workload and expenditure, and influences clinician-requesting behaviour.
        J Clin Pathol. 2014; 67: 731-733
        • Goetz C.
        • Rotman S.R.
        • Hartoularos G.
        • Bishop T.F.
        The effect of charge display on cost of care and physician practice behaviors: a systematic review.
        J Gen Intern Med. 2015; 30: 835-842
        • Vidyarthi A.R.
        • Hamill T.
        • Green A.L.
        • Rosenbluth G.
        • Baron R.B.
        Changing resident test ordering behavior: a multilevel intervention to decrease laboratory utilization at an academic medical center.
        Am J Med Qual. 2015; 30: 81-87
        • Corson A.H.
        • Fan V.S.
        • White T.
        • et al.
        A multifaceted hospitalist quality improvement intervention: decreased frequency of common labs.
        J Hosp Med. 2015; 10: 390-395
        • Solomon D.H.
        • Hashimoto H.
        • Daltroy L.
        • Liang M.H.
        Techniques to improve physicians' use of diagnostic tests: a new conceptual framework.
        JAMA. 1998; 280: 2020-2027
        • Baird G.
        The laboratory test utilization management toolbox.
        Biochem Med (Zagreb). 2014; 24: 223-234
      2. Centers for Medicare & Medicaid Services. Clinical laboratory fee schedule. Available at: http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/ClinicalLabFeeSched/. Accessed August 10, 2015.

      3. University Health System Consortium. 2014 Risk-adjustment models. Available at: https://www.uhc.edu/26295. Accessed August 11, 2015.

        • Wagner A.K.
        • Soumerai S.B.
        • Zhang F.
        • Ross-Degnan D.
        Segmented regression analysis of interrupted time series studies in medication use research.
        J Clin Pharm Ther. 2002; 27: 299-309
        • Venables W.N.
        • Smith D.M.
        • R Development Core Team
        An Introduction to R: Notes on R: A Programming Environment for Data Analysis and Graphics, version 1.4.1.
        Network Theory, Bristol, UK2002
        • Cheng C.K.
        • Lee T.
        • Cembrowski G.S.
        Temporal approach to hematological test usage in a major teaching hospital.
        Lab Hematol. 2003; 9: 207-213