Advertisement

Maintenance of Certification: 20 Years Later

      A voluntary physician-led, nongovernmental process of setting standards is a unique strength of American medicine. Founded in 1936, the American Board of Internal Medicine (ABIM) has set the standards for practice in internal medicine and its subspecialties. ABIM is a physician-run organization but is independent of any physician societies or membership organizations. The standards, measured by the “certification” process, demonstrate to the public that internists have met the knowledge and practice requirements that ensure a high level of quality of care. It is a model for other countries that have standards set by either a government agency or a physician membership organization—which is often seen as less credible by the public.
      • Shaw K.
      • Cassel C.K.
      • Black C.
      • Levinson W.
      Shared medical regulation in a time of increasing calls for accountability and transparency: comparison of recertification in the United States, Canada, and the United Kingdom.
      • Levinson W.
      Revalidation of physicians in Canada: are we passing the test?.
      • Approximately 98% of internists who completed an accredited training program between 1990 and 2007 attempted certification; 96% ultimately achieved certification in internal medicine.
      • In 1990, the American Board of Internal Medicine (ABIM) made certification time limited: all certifications in internal medicine and its subspecialties would be valid for a period of 10 years, and internists would be required to maintain their certification through a Maintenance of Certification program.
      • ABIM plans to institute a more continuous process in keeping with external medical regulators and public expectations that internists will update their knowledge and skills regularly.
      While participation in the certification process is entirely voluntary, 98% of internists who completed an accredited training program between 1990 and 2007 attempted certification. Of these individuals, 96% ultimately achieved certification in internal medicine. Until 1990, internists completing the ABIM standards were issued certifications that were valid indefinitely. However, there is growing evidence that physicians' knowledge and skills decline in the years after training,
      • Choudhry N.K.
      • Fletcher R.H.
      • Soumerai S.B.
      Systematic review: the relationship between clinical experience and quality of health care.
      and that traditional continuing medical education is not sufficient to ensure that physicians continue to practice at an acceptable level of quality.
      • Mazmanian P.E.
      • Davis D.A.
      Continuing medical education and the physician as a learner: guide to the evidence.
      • Davis D.A.
      • Thomson M.A.
      • Oxman A.D.
      • Haynes R.B.
      Changing physician performance A systematic review of the effect of continuing medical education strategies.
      • Davis D.A.
      • Mazmanian P.E.
      • Fordis M.
      • Van Harrison R.
      • Thorpe K.E.
      • Perrier L.
      Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
      ABIM instituted a significant change to the program in 1990: all certifications in internal medicine and its subspecialties would be valid for a period of 10 years, and internists would be required to maintain their certification through a Maintenance of Certification (MOC) program. Diplomates who did not maintain their certification would no longer be “board certified.” While the change to the program was and remains contentious
      • Levinson W.
      • King Jr, T.E.
      • Goldman L.
      • Goroll A.H.
      • Kessler B.
      Clinical decisions American Board of Internal Medicine maintenance of certification program.
      —with many internists objecting to these new and more periodic requirements—approximately 85% chose to participate in MOC. It is now 2 decades since MOC was introduced. This article describes the rationale, the components of the present program, the evolving research evidence, characteristics of internists who have participated over the last 20 years, and the limitations and future of MOC.

      Rationale for MOC

      There is significant evidence supporting the importance of board certification. Initial ABIM certification, which consists of training in an accredited program and passing the ABIM examination, is associated with the provision of higher-quality care by certified internists compared with physicians who are not certified. For example, studies have found that certified internists provide patients with higher rates of preventive services, improved care for hypertension, and lower mortality rates in patients with myocardial infarction compared with care delivered by noncertified internists.
      • Norcini J.J.
      • Lipner R.S.
      • Kimball H.R.
      Certifying examination performance and patient outcomes following acute myocardial infarction.
      • Norcini J.J.
      • Kimball H.R.
      • Lipner R.S.
      Certification and specialization: do they matter in the outcome of acute myocardial infarction?.
      • Chen J.
      • Rathore S.S.
      • Wang Y.
      • Radford M.J.
      • Krumholz H.M.
      Physician board certification and the care and outcomes of elderly patients with acute myocardial infarction.
      • Pham H.H.
      • Schrag D.
      • Hargraves J.L.
      • Bach P.B.
      Delivery of preventive services to older adults by primary care physicians.
      • Ramsey P.G.
      • Carline J.D.
      • Inui T.S.
      • Larson E.B.
      • LoGerfo J.P.
      • Wenrich M.D.
      Predictive validity of certification by the American Board of Internal Medicine.
      However, certification is limited to assessing physicians at one point in time and does not ensure that they maintain their knowledge and skills in an environment of rapidly growing and expanding research evidence. Evidence suggests that physicians do not maintain their knowledge and skills or effectively self-assess their performance on their own.
      • Choudhry N.K.
      • Fletcher R.H.
      • Soumerai S.B.
      Systematic review: the relationship between clinical experience and quality of health care.
      • Davis D.A.
      • Mazmanian P.E.
      • Fordis M.
      • Van Harrison R.
      • Thorpe K.E.
      • Perrier L.
      Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
      Choudhry et al
      • Choudhry N.K.
      • Fletcher R.H.
      • Soumerai S.B.
      Systematic review: the relationship between clinical experience and quality of health care.
      conducted a systematic review of 62 articles about physician knowledge or a quality-of-care outcome and the years since graduation or the physician's age. Overall, 73% of the articles reported decreasing performance on some or all of the outcomes assessed with increasing years in practice. The studies assessed outcomes that are widely accepted as important to quality care, including general medical and specialized knowledge relevant to physicians' practice; adherence to standards of appropriate diagnostic, screening, or preventive care; adherence to standards of appropriate treatment; and actually measured patient outcomes. Furthermore, studies assessing how physicians actually perform in practice demonstrate major gaps between known appropriate standards of practice and what physicians actually do. Often, basic standards of care are performed in only 50% of patients.
      • McGlynn E.A.
      • Asch S.M.
      • Adams J.
      • et al.
      The quality of health care delivered to adults in the United States.
      Institute of Medicine, Committee on Quality of Health Care in America
      Crossing the Quality Chasm: A New Health System for the 21st Century.
      Despite best intentions to deliver high-quality care, physicians and the organizations in which they practice often fall short.
      Furthermore, public opinion polling finds that the public expects physicians to be current in medical knowledge and think they should be evaluated regularly.
      • Brennan T.A.
      • Horwitz R.I.
      • Duffy F.D.
      • Cassel C.K.
      • Goode L.D.
      • Lipner R.S.
      The role of physician specialty board certification status in the quality movement.
      A recent MSNBC poll asked the public whether “all specialists should be required to take a test to renew their certification.” Eighty percent agreed that to provide the public with a credible standard, the profession must have a systematic way to assess physician knowledge, skills, and actual care delivery in a transparent and ongoing fashion.

      MOC in 2010

      ABIM is a member of the American Board of Medical Specialties (ABMS) – the umbrella organization for 26 specialty boards. ABMS articulates requirements for elements of the MOC program and allows the individual boards the flexibility to design their own programs to meet those requirements under the leadership of physicians in academia and practice. ABMS requires 4 components for every MOC program: evidence of professional standing (license to practice); participation in lifelong learning and self-assessment; evidence of cognitive expertise (examination); and assessment of practice performance.
      The ABIM MOC program is designed to be responsive to new requirements from ABMS and feedback from the diplomates who participate in the program. Currently, the ABIM program requires diplomates to pass a secure examination in their subspecialty once every 10 years. Evidence supports that physicians should possess a certain level of knowledge and reasoning ability on their own to effectively make clinical decisions.
      • Gruppen L.D.
      • Frohna A.Z.
      Clinical reasoning..
      • Holmboe E.S.
      • Lipner R.
      • Greiner A.
      Assessing quality of care: knowledge matters.
      • Bowen J.L.
      Educational strategies to promote clinical diagnostic reasoning.
      The examination is a computer-delivered test typically composed of 180 multiple-choice questions. The examination in each specialty is designed by a test-writing committee comprising specialists in that area; ABIM psychometricians ensure that each examination is reliable and valid.
      Other elements of the ABIM MOC program allow physicians to choose either ABIM products or other products or programs offered by their society, health system, or hospital to meet those requirements. These non-ABIM programs are evaluated by ABIM physician-boards to ensure that they meet high standards for formative assessment. For example, while ABIM offers self-evaluation of medical knowledge assessments in internal medicine and each of its subspecialties, more than 30 products are ABIM-approved for credit towards self-evaluation of medical knowledge.
      In response to the growing literature on the importance of self-assessment in practice and the idea that knowledge alone was not enough to determine if physicians had the skills and judgment to deliver quality care, ABIM added a self-assessment of practice performance requirement to the program in 2006. To help physicians meet this requirement, ABIM developed web-based tools—Practice Improvement Modules (PIMs) —that allowed physicians to examine elements of their practice and to get feedback from peers and patients. PIMs allow internists to assess how they are actually performing in their work setting and contain a clinical care audit with feedback, a survey of patient experience, and an assessment of the “systems” of a practice. Data collected by the physician or staff via the web are analyzed automatically by unique ABIM statistical software, which provides feedback about how a physician's performance compares with standards in the field. The internist then reflects on their performance and creates their own quality-improvement plan. At the end of the process, physicians resubmit data and can compare their performance before and after the quality-improvement intervention. This participation in a performance assessment and improvement cycle, often called a Plan-Do-Study-Act cycle, was developed out of the science of quality improvement.
      • Nelson E.C.
      • Batalden P.B.
      • Godfrey M.M.
      Quality by Design: A Clinical Microsystems Approach.

      Science and Theory of MOC

      Experience and assessment drive adult learning. Assessment helps physicians to recognize and address gaps in knowledge and performance. Although there are many models, Kolb's model of experiential learning (Figure 1) provides one example to highlight how assessment is a necessary element of learning and personal change.
      • Kolb D.A.
      Experimental Learning: Experience as the Source of Learning and Development.
      In this model, assessment is a particularly important aspect of the concrete experience and observation and reflection phases. Physicians require “good data” to ensure that their interpretations and reflections are accurate around their performance and learning needs. However, research demonstrates that physicians are not accurate in generating this information through isolated self-assessment, and the least competent physicians are unfortunately also the least skilled in assessing their abilities and performance.
      • Davis D.A.
      • Mazmanian P.E.
      • Fordis M.
      • Van Harrison R.
      • Thorpe K.E.
      • Perrier L.
      Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
      Furthermore, physicians often confuse experience with expertise. Evidence exists that overall competence may actually decline with age; older physicians may fail to “keep up” and are prone to “premature closure” when making a diagnosis.
      • Eva K.W.
      The aging physician: changes in cognitive processing and their impact on medical practice.
      However, another well-described phenomenon is that of the experienced “nonexpert” —just because you do something often doesn't necessarily mean you are an expert.
      • Bereiter C.
      • Scardamalia M.
      Surpassing Ourselves: An Inquiry into the Nature and Implications of Expertise.
      Many graduating residents possess significant deficiencies in clinical skills.
      • Holmboe E.S.
      Faculty and the observation of trainees' clinical skills: problems and opportunities.
      It is not surprising, therefore, that these deficiencies may not get corrected in practice, especially if the physician does not participate in meaningful, ongoing, and rigorous assessment processes. MOC provides a useful assessment vehicle for practicing physicians to accurately and rigorously assess their knowledge and performance gaps.

      What Is the Evidence for ABIM's MOC Program?

      A small but evolving body of literature relates elements of MOC to physician performance. Further research is imperative to understand both the relationship of MOC to outcomes and the best assessment tools. The MOC secure examination is highly reliable, with coefficients consistently above 0.90.
      • Norcini J.J.
      Setting standards on educational tests.
      Two recent articles provided validity evidence by correlating performance on the examination with performance in practice.
      • Turchin A.
      • Shubina M.
      • Chodos A.H.
      • Einbinder J.S.
      • Pendergrass M.L.
      Effect of board certification on antihypertensive treatment intensification in patients with diabetes mellitus.
      • Holmboe E.S.
      • Wang Y.
      • Meehan T.P.
      • et al.
      Association between maintenance of certification examination scores and quality of care for Medicare beneficiaries.
      These results are consistent with a body of evidence from 16 separate studies on the relationship between initial certification examinations and subsequent clinical practice.
      • Sharp L.K.
      • Bashook P.G.
      • Lipsky M.S.
      • Horowitz S.D.
      • Miller S.H.
      Specialty board certification and clinical outcomes: the missing link.
      Related to the PIMs, diplomates report that they value the information they receive about their practice, often describing the “aha” moment when they review their performance data and uncover an area of suboptimal practice where they are doing less well than expected.
      • Bernabeo E.C.
      • Conforti L.N.
      • Holmboe E.S.
      The impact of a preventive cardiology quality improvement intervention on residents and clinics: a qualitative exploration.
      • Holmboe E.S.
      • Meehan T.P.
      • Lynn L.
      • Doyle P.
      • Sherwin T.
      • Duffy F.D.
      Promoting physicians' self-assessment and quality improvement: the ABIM diabetes practice improvement module.
      • Duffy F.D.
      • Lynn L.A.
      • Didura H.
      • et al.
      Self-assessment of practice performance: development of the ABIM Practice Improvement Module (PIM).
      This phenomenon has been well described by others who have argued that physicians must engage in guided assessment and self-directed assessment seeking.
      • Davis D.A.
      • Mazmanian P.E.
      • Fordis M.
      • Van Harrison R.
      • Thorpe K.E.
      • Perrier L.
      Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
      • Eva K.W.
      • Regehr G.
      “I’ll never play professional football” and other fallacies of self-assessment.
      • Duffy F.D.
      • Holmboe E.S.
      Self-assessment in lifelong learning and improving performance in practice: physician know thyself.
      The reliability and validity of the measures in the PIM have been well described.
      • Lipner R.S.
      • Weng W.
      • Arnold G.K.
      • Duffy F.D.
      • Lynn L.A.
      • Holmboe E.S.
      A three-part model for measuring diabetes care in physician practice.
      Research studies on the PIMs have found that it is often the physician's first experience with performance measurement and improvement.
      • Bernabeo E.C.
      • Conforti L.N.
      • Holmboe E.S.
      The impact of a preventive cardiology quality improvement intervention on residents and clinics: a qualitative exploration.
      • Holmboe E.S.
      • Meehan T.P.
      • Lynn L.
      • Doyle P.
      • Sherwin T.
      • Duffy F.D.
      Promoting physicians' self-assessment and quality improvement: the ABIM diabetes practice improvement module.
      • Duffy F.D.
      • Lynn L.A.
      • Didura H.
      • et al.
      Self-assessment of practice performance: development of the ABIM Practice Improvement Module (PIM).
      • Mladenovic J.
      • Shea J.A.
      • Duffy F.D.
      • Lynn L.A.
      • Holmboe E.S.
      • Lipner R.S.
      Variation in internal medicine residency clinic practices: assessing practice environments and quality of care.
      • Oyler J.
      • Vinci L.
      • Arora V.
      • Johnson J.
      Teaching internal medicine residents quality improvement techniques using the ABIM's practice improvement modules.
      Several studies have found direct improvements in patient care.
      • Oyler J.
      • Vinci L.
      • Arora V.
      • Johnson J.
      Teaching internal medicine residents quality improvement techniques using the ABIM's practice improvement modules.
      • Williams M.V.
      Hospitalists and the hospital medicine system of care are good for patient care.
      • Simpkins J.
      • Divine G.
      • Wang M.
      • Holmboe E.
      • Pladevall M.
      • Williams L.K.
      Improving asthma care through recertification: a cluster randomized trial.
      • Lipner R.S.
      • Bylsma W.H.
      • Arnold G.K.
      • Fortna G.S.
      • Tooker J.
      • Cassel C.K.
      Who is maintaining certification in internal medicine—and why? A national survey 10 years after initial certification.
      • Shunk R.L.
      • Dulay M.
      • Julian K.
      • et al.
      Using the American Board of Internal Medicine practice improvement modules to teach internal medicine residents practice improvement.
      In all, the 13 studies published since 2006 have evaluated PIMs in diabetes, preventive clinical services, preventive cardiology, care of the vulnerable elderly, asthma, communication with patients, communication with referring physicians, and the comprehensive care. Ongoing studies are exploring the impact of the PIM on patients, how to assess physician performance across multiple conditions, and the interaction between health system and physician competence.

      Participants and Feedback

      In 2010, approximately 70% of ABIM diplomates have at least one time-limited certificate with a 10-year duration. The other 30% completed their certification before the introduction of MOC and hold certification that is valid indefinitely; they are often called “grandfathers.” Several subspecialties, such as critical care and geriatrics, became subspecialties after 1987 and never had “grandfathers.” ABIM encourages grandfathers to recertify voluntarily,
      • Levinson W.
      • King Jr, T.E.
      • Goldman L.
      • Goroll A.H.
      • Kessler B.
      Clinical decisions American Board of Internal Medicine maintenance of certification program.
      but only 1.7% enroll in MOC and only one-half of that percentage complete the process. The information available about the MOC process is primarily based on time-limited certification holders.
      Overall, diplomates who certified only in general internal medicine between 1990 and 1997 have a high MOC completion rate (80%).
      • Bylsma W.H.
      • Arnold G.K.
      • Fortna G.S.
      • Lipner R.S.
      Where have all the general internists gone?.
      Subspecialists can chose MOC for general internal medicine, their specialty, or both. Overall, the rates of completion of MOC in subspecialty is high (82%), but this rate varies from 50% in geriatrics to 89% in gastroenterology (Figure 2). In addition, a significant number of subspecialists also complete MOC in general internal medicine, ranging from 33% for cardiology to 86% for geriatrics. These high participation rates for a voluntary process indicate that internists and the organizations within which they practice place value on maintaining board-certification status.
      Figure thumbnail gr2
      Figure 2Maintenance of Certification completion rates among physicians who initially certified in internal medicine (IM) or a subspecialty (SS) 1990-1997 (as of February 2009).
      ABIM asks diplomates to complete a survey about the value and experience regarding their participation on each component of MOC. For the self-assessment of knowledge modules, diplomates provided feedback for 85% of more than 74,000 modules completed between April 2007 and March 2009. Seventy-eight percent agreed or strongly agreed that the modules provided a valuable overall learning experience and helped them identify important areas for further study, while 71% said the knowledge modules raised their awareness on how to improve patient care.
      Although the examination is one of the more anxiety-provoking aspects of the MOC program, diplomates rate the overall experience once they have taken the examination highly (84%), and most believe that their MOC examination is a relevant test of their knowledge (61%). Of the 23,646 diplomates who took an MOC examination between 2007 and 2009, 15% disagreed or strongly disagreed that their MOC examination was a fair assessment of clinical knowledge in their discipline, and 4% were dissatisfied with the testing experience.
      Diplomates completed over 9800 PIMs between January 2008 and June 2009, with 79% of PIM completers providing feedback. For the disease- or condition-specific PIMs (eg, diabetes, hypertension), 90% of diplomates agreed or strongly agreed that reviewing their patient charts as part of the audit process raised their awareness of the quality of care their practice provided, while 88% said the audit provided ideas on how to improve their practice. Overall, 73% of diplomates reported that they changed their practice as a direct result of completing a PIM, and 82% reported they would recommend the PIM to a colleague.

      Limitations and the Future of MOC

      ABIM believes the MOC process must continue to improve and evolve. In a recent dialogue related to whether grandfathers should recertify, diplomates suggested areas of deficiency in the present MOC process and recommended changes.
      • Levinson W.
      • King Jr, T.E.
      • Goldman L.
      • Goroll A.H.
      • Kessler B.
      Clinical decisions American Board of Internal Medicine maintenance of certification program.
      Diplomates point out the need for the secure examination to be more tailored to their specific practice and provide electronic resources to look up information as one does in practice. They are concerned about the time required to complete the process and they call for better research to understand whether the time and effort are worth it in terms of patient outcomes. They raise concerns about the cost of the process.
      The most important upcoming change is that the MOC program will become more continuous, requiring internists to participate on a regular basis. Currently, under the 10-year MOC program, the vast majority (74%) wait until the ninth year of their certification before they prepare any of the components of MOC. A more continuous process is consistent with public expectations that internists will update their knowledge and skills regularly,
      • Brennan T.A.
      • Horwitz R.I.
      • Duffy F.D.
      • Cassel C.K.
      • Goode L.D.
      • Lipner R.S.
      The role of physician specialty board certification status in the quality movement.
      and consistent with external medical regulators and medical organizations that require physicians to report their participation in continuing medical education on a yearly or biyearly basis. For example, most state licensing boards request information from all physicians about their continuing medical education biyearly to renew state licenses. Harmonizing MOC with these external stakeholders will ultimately make it easier for internists to participate in MOC and meet other requirements simultaneously. Furthermore, a continuous MOC process is aligned with educational theory that supports the idea that most effective learning is ongoing.
      • Ericsson K.A.
      The influence of experience and deliberate practice on the development of superior expert performance.
      • Ericsson K.A.
      An expert-performance perspective of research on medical expertise: the study of clinical performance.
      The MOC process should evaluate internists' competencies in a variety of domains. The ABMS competencies include patient care, medical knowledge, practice-based learning and improvement, systems-based learning, professionalism and interpersonal skills, and communication. Skill in working collaboratively with a team of health care providers is increasingly recognized as key for internists. However, the present MOC process emphasizes some competencies more than others; for example, professionalism and clinical skills are minimally evaluated. ABIM is developing new tools to help internists assess each of these competencies, but development of reliable and valid tools will require significant resources. ABIM recently launched efforts to develop useful assessment tools in teamwork and professionalism. In addition, ABMS now requires a component related to patient safety and ABIM is exploring options to incorporate it into MOC.
      Specific elements are undergoing changes. The secure examination will be changed to improve fidelity, and research is under way to explore the inclusion of resources so that internists can look up information, simulating real practice. ABIM is evaluating other assessments of medical knowledge, including a tool that allows physicians to answer clinical questions generated at the point of care
      • Green M.L.
      • Reddy S.G.
      • Holmboe E.
      Teaching and evaluating point of care learning with an internet-based clinical-question portfolio.
      and increased use of simulation in cardiology and other specialties. PIMs are being improved by incorporating more specific feedback about a diplomate's performance compared with national and regional benchmarks and by developing new standards. ABIM and the specialty societies are working on the self assessment of knowledge to increase products tailored to individual needs. Recognizing that supervision of and feedback to learners is an important component of the quality calculus, ABIM last year introduced the Clinical Supervision PIM for academic faculty that combines assessment of learner clinical skills with a focused chart audit of key safety measures and diagnostic reasoning. Already more than 450 faculty members have completed the Clinical Supervision PIM.

      Conclusion

      As MOC enters its second decade, it is clear that internists have effectively transitioned from a “point-in-time” certification to a maintenance program. The large majority of internists with time-limited certifications actively participate in the program, and newer internists expect to participate throughout their practice careers. As ABIM responds to the changing face of medicine and feedback from diplomates, the program must evolve and improve to remain accountable to the first and most important constituency, the American public.

      References

        • Shaw K.
        • Cassel C.K.
        • Black C.
        • Levinson W.
        Shared medical regulation in a time of increasing calls for accountability and transparency: comparison of recertification in the United States, Canada, and the United Kingdom.
        JAMA. 2009; 302: 2008-2014
        • Levinson W.
        Revalidation of physicians in Canada: are we passing the test?.
        CMAJ. 2008; 179: 979-980
        • Choudhry N.K.
        • Fletcher R.H.
        • Soumerai S.B.
        Systematic review: the relationship between clinical experience and quality of health care.
        Ann Intern Med. 2005; 142: 260-273
        • Mazmanian P.E.
        • Davis D.A.
        Continuing medical education and the physician as a learner: guide to the evidence.
        JAMA. 2002; 288: 1057-1060
        • Davis D.A.
        • Thomson M.A.
        • Oxman A.D.
        • Haynes R.B.
        Changing physician performance.
        JAMA. 1995; 274: 700-705
        • Davis D.A.
        • Mazmanian P.E.
        • Fordis M.
        • Van Harrison R.
        • Thorpe K.E.
        • Perrier L.
        Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
        JAMA. 2006; 296: 1094-1102
        • Levinson W.
        • King Jr, T.E.
        • Goldman L.
        • Goroll A.H.
        • Kessler B.
        Clinical decisions.
        N Engl J Med. 2010; 362: 948-952
        • Norcini J.J.
        • Lipner R.S.
        • Kimball H.R.
        Certifying examination performance and patient outcomes following acute myocardial infarction.
        Med Educ. 2002; 36: 853-859
        • Norcini J.J.
        • Kimball H.R.
        • Lipner R.S.
        Certification and specialization: do they matter in the outcome of acute myocardial infarction?.
        Acad Med. 2000; 75: 1193-1198
        • Chen J.
        • Rathore S.S.
        • Wang Y.
        • Radford M.J.
        • Krumholz H.M.
        Physician board certification and the care and outcomes of elderly patients with acute myocardial infarction.
        J Gen Intern Med. 2006; 21: 238-244
        • Pham H.H.
        • Schrag D.
        • Hargraves J.L.
        • Bach P.B.
        Delivery of preventive services to older adults by primary care physicians.
        JAMA. 2005; 294: 473-481
        • Ramsey P.G.
        • Carline J.D.
        • Inui T.S.
        • Larson E.B.
        • LoGerfo J.P.
        • Wenrich M.D.
        Predictive validity of certification by the American Board of Internal Medicine.
        Ann Intern Med. 1989; 110: 719-726
        • McGlynn E.A.
        • Asch S.M.
        • Adams J.
        • et al.
        The quality of health care delivered to adults in the United States.
        N Engl J Med. 2003; 348: 2635-2645
        • Institute of Medicine, Committee on Quality of Health Care in America
        Crossing the Quality Chasm: A New Health System for the 21st Century.
        National Academy Press, Washington, DC2001
        • Brennan T.A.
        • Horwitz R.I.
        • Duffy F.D.
        • Cassel C.K.
        • Goode L.D.
        • Lipner R.S.
        The role of physician specialty board certification status in the quality movement.
        JAMA. 2004; 292: 1038-1043
      1. MSNBC Health.
        (Accessed May 3, 2010)
        • Gruppen L.D.
        • Frohna A.Z.
        Clinical reasoning..
        in: Norman G.R. van der Vleuten C.P.M. Newble D.I. International Handbook of Research in Medical Education. Kluwer Academic, Dordrecht, Netherlands2002: 205-230
        • Holmboe E.S.
        • Lipner R.
        • Greiner A.
        Assessing quality of care: knowledge matters.
        JAMA. 2008; 299: 338-340
        • Bowen J.L.
        Educational strategies to promote clinical diagnostic reasoning.
        N Engl J Med. 2006; 355: 2217-2225
        • Nelson E.C.
        • Batalden P.B.
        • Godfrey M.M.
        Quality by Design: A Clinical Microsystems Approach.
        Jossey-Bass, John Wiley & Sons, Inc, San Francisco, CA2007
        • Kolb D.A.
        Experimental Learning: Experience as the Source of Learning and Development.
        Prentice-Hall, Inc, Englewood Cliffs, NJ1984
        • Eva K.W.
        The aging physician: changes in cognitive processing and their impact on medical practice.
        Acad Med. 2002; 77: S1-S6
        • Bereiter C.
        • Scardamalia M.
        Surpassing Ourselves: An Inquiry into the Nature and Implications of Expertise.
        Open Court Publishing Company, Chicago, IL1993
        • Holmboe E.S.
        Faculty and the observation of trainees' clinical skills: problems and opportunities.
        Acad Med. 2004; 79: 16-22
        • Norcini J.J.
        Setting standards on educational tests.
        Med Educ. 2003; 37: 464-469
        • Turchin A.
        • Shubina M.
        • Chodos A.H.
        • Einbinder J.S.
        • Pendergrass M.L.
        Effect of board certification on antihypertensive treatment intensification in patients with diabetes mellitus.
        Circulation. 2008; 117: 623-628
        • Holmboe E.S.
        • Wang Y.
        • Meehan T.P.
        • et al.
        Association between maintenance of certification examination scores and quality of care for Medicare beneficiaries.
        Arch Intern Med. 2008; 168: 1396-1403
        • Sharp L.K.
        • Bashook P.G.
        • Lipsky M.S.
        • Horowitz S.D.
        • Miller S.H.
        Specialty board certification and clinical outcomes: the missing link.
        Acad Med. 2002; 77: 534-542
        • Bernabeo E.C.
        • Conforti L.N.
        • Holmboe E.S.
        The impact of a preventive cardiology quality improvement intervention on residents and clinics: a qualitative exploration.
        Am J Med Qual. 2009; 24: 99-107
        • Holmboe E.S.
        • Meehan T.P.
        • Lynn L.
        • Doyle P.
        • Sherwin T.
        • Duffy F.D.
        Promoting physicians' self-assessment and quality improvement: the ABIM diabetes practice improvement module.
        J Contin Educ Health Prof. 2006; 26: 109-119
        • Duffy F.D.
        • Lynn L.A.
        • Didura H.
        • et al.
        Self-assessment of practice performance: development of the ABIM Practice Improvement Module (PIM).
        J Contin Educ Health Prof. 2008; 28: 38-46
        • Eva K.W.
        • Regehr G.
        “I’ll never play professional football” and other fallacies of self-assessment.
        J Contin Educ Health Prof. 2008; 28: 14-19
        • Duffy F.D.
        • Holmboe E.S.
        Self-assessment in lifelong learning and improving performance in practice: physician know thyself.
        JAMA. 2006; 296: 1137-1139
        • Lipner R.S.
        • Weng W.
        • Arnold G.K.
        • Duffy F.D.
        • Lynn L.A.
        • Holmboe E.S.
        A three-part model for measuring diabetes care in physician practice.
        Acad Med. 2007; 82: S48-S52
        • Mladenovic J.
        • Shea J.A.
        • Duffy F.D.
        • Lynn L.A.
        • Holmboe E.S.
        • Lipner R.S.
        Variation in internal medicine residency clinic practices: assessing practice environments and quality of care.
        J Gen Intern Med. 2008; 23: 914-920
        • Oyler J.
        • Vinci L.
        • Arora V.
        • Johnson J.
        Teaching internal medicine residents quality improvement techniques using the ABIM's practice improvement modules.
        J Gen Intern Med. 2008; 23: 927-930
        • Williams M.V.
        Hospitalists and the hospital medicine system of care are good for patient care.
        Arch Intern Med. 2008; 168 (; discussion 1259-60): 1254-1256
        • Simpkins J.
        • Divine G.
        • Wang M.
        • Holmboe E.
        • Pladevall M.
        • Williams L.K.
        Improving asthma care through recertification: a cluster randomized trial.
        Arch Intern Med. 2007; 167: 2240-2248
        • Lipner R.S.
        • Bylsma W.H.
        • Arnold G.K.
        • Fortna G.S.
        • Tooker J.
        • Cassel C.K.
        Who is maintaining certification in internal medicine—and why?.
        Ann Intern Med. 2006; 144: 29-36
        • Shunk R.L.
        • Dulay M.
        • Julian K.
        • et al.
        Using the American Board of Internal Medicine practice improvement modules to teach internal medicine residents practice improvement.
        J Grad Med Educ. 2010; 2: 90-95
        • Bylsma W.H.
        • Arnold G.K.
        • Fortna G.S.
        • Lipner R.S.
        Where have all the general internists gone?.
        J Gen Intern Med. 2010; 25: 1020-1023
        • Ericsson K.A.
        The influence of experience and deliberate practice on the development of superior expert performance.
        in: Ericsson K.A. Charness N. Hoffman R.R. Feltovich P.J. The Cambridge Handbook of Expertise and Expert Performance. Cambridge University Press, New York2006: 683-704
        • Ericsson K.A.
        An expert-performance perspective of research on medical expertise: the study of clinical performance.
        Med Educ. 2007; 41: 1124-1130
        • Green M.L.
        • Reddy S.G.
        • Holmboe E.
        Teaching and evaluating point of care learning with an internet-based clinical-question portfolio.
        J Contin Educ Health Prof. 2009; 29: 209-219

      Linked Article

      • Recertification for Grandfathers
        The American Journal of MedicineVol. 124Issue 10
        • Preview
          I read with particular interest the article by Levinson and Holmboe,1 recently published in The American Journal of Medicine. These authors provide a very informative update on the American Board of Internal Medicine (ABIM) Maintenance of Certification (MOC) program.
        • Full-Text
        • PDF
      • Common Sense and the Recertification Process
        The American Journal of MedicineVol. 125Issue 1
        • Preview
          I strongly disagree with the authors' statement that certification is voluntary.1 Because some hospitals require applicants for membership on their staffs to be certified or recertified and because there is a risk that managed care organizations may use certification as a criterion for admitting physicians to their panels or for prorating their fees, many physicians consider it virtually compulsory.
        • Full-Text
        • PDF