The American Journal of Medicine
Volume 119, Issue 8 , Pages 708-714 , August 2006

Outcomes-based Evaluation in Resident Education: Creating Systems and Structured Portfolios

  • Eric S. Holmboe, MD

      Affiliations

    • American Board of Internal Medicine, Philadelphia, Pa
    • Corresponding Author InformationReprint requests should be addressed to Eric S. Holmboe, MD, American Board of Internal Medicine, 510 Walnut Street, Suite 1700, Philadelphia, PA 19106.
    • ,
  • William Rodak, PhD

      Affiliations

    • Accreditation Council for Graduate Medical Education, Chicago, Ill
    • ,
  • Glenn Mills, MD

      Affiliations

    • Louisiana State University Health Science Center, Shreveport, La
    • ,
  • Michael J. McFarlane, MD

      Affiliations

    • Case Western Reserve University, Cleveland, Ohio
    • ,
  • Henry J. Schultz, MD

      Affiliations

    • Mayo Clinic, Rochester, Minn
    • Accreditation Council for Graduate Medical Education and American Board of Medical Specialties Quadrad Committee for Internal Medicine, Chicago, Ill

References 

  1. Accreditation Council for Graduate Medical Education. The Outcome Project. Available at: http://www.acgme.org/Outcome/. Accessed September 30, 2005.
  2. Holmboe ES, Hawkins RE. Evaluating the clinical competence of residents (a review). Ann Intern Med. 1998;129:42–48
  3. Nelson EC, Batalden PB, Huber TP, et al. Microsystems in health care: Part 1. Learning from high-performing front-line clinical units. Jt Comm J Qual Improv. 2002;28:472–493
  4. Donabedian A. Explorations in Quality Assessment and Monitoring. The Definition of Quality and Approaches to its Assessment. Vol. 1. Ann Arbor, MI: Health Administration Press; 1980;
  5. Swing SR. Assessing the ACGME General Competencies (general considerations and assessment methods). Acad Emerg Med. 2002;9:1278–1288
  6. Gray JD. Global rating scales in residency education. Acad Med. 1996;71:S55–S63
  7. Schon DA. The Reflective Practitioner (How Professionals Think In Action). New York, NY: Basic Books; 1983;
  8. Holmboe ES, Bowen JD, Green M, et al. Reforming internal medicine residency training (a report from the Society of General Internal Medicine’s task force for residency reform). J Gen Intern Med. 2005;20:1165–1172
  9. Ende J. Feedback in clinical medical education. JAMA. 1983;250:777–781
  10. Holmboe ES, Williams F, Yepes M, Huot S. Feedback and the mini clinical evaluation exercise. J Gen Intern Med. 2004;19:558–561
  11. Hemmer PA, Hawkins R, Jackson JL, Pangaro LN. Assessing how well three evaluation methods detect deficiencies in medical students’ professionalism in two settings of an internal medicine clerkship. Acad Med. 2000;75:167–173
  12. Hiss RG, MacDonald R, David WR. Identification of physician educational influentials in small community hospitals. Res Med Educ. 1978;17:283–288
  13. Holmboe ES, Bradley ES, Mattera JA, et al. Characteristics of physician leaders working to improve the quality of care in acute myocardial infarction (a qualitative study). Jt Comm J Qual Saf. 2003;29:289–296
  14. Brennan TA. Physicians responsibility to improve the quality of care. Acad Med. 2002;77:973–980
  15. Gruen RL, Pearson SD, Brennan TA. Physician-citizens—public roles and professional obligations. JAMA. 2004;291:94–98
  16. Friedman Ben David M, Davis MH, Harden RM, et al. AAME Educational Guide 24 (portfolios as a method of student assessment). Med Teach. 2001;23:535–551
  17. Challis M. AMEE Medical Education Guide No.11 (portfolio-based learning and assessment in medical education). (revised) Med Teach. 1999;21:370–386
  18. Carraccio C, Englander R. Evaluating competence using a portfolio (a literature review and web-based application to the ACGME competencies). Teach Learn Med. 2004;16:381–387
  19. Campbell CM, Parboosingh JT, Gondocz ST, et al. Study of physicians’ use of a software program to create a portfolio of their self-directed learning. Acad Med. 1996;71:S49–S51
  20. Wilkinson TJ, Challis M, Hobma SO, et al. The use of portfolios for assessment of the competence and performance of doctors in practice. Med Educ. 2002;36:918–924
  21. Turnbull J, Gray J, MacFayden J. Improving in-training evaluation programs. J Gen Intern Med. 1998;13:317–323
  22. Silber CG, Nasca TJ, Paskin DL, et al. Do global rating forms enable program directors to assess the ACGME competencies?. Acad Med. 2004;79:549–556
  23. Schwind CJ, Williams RG, Boehler ML, Dunnington GL. Do individual attendings’ post-rotation performance ratings detect residents’ clinical performance deficiencies?. Acad Med. 2004;79:453–457
  24. Holmboe ES, Fiebach NH, Galaty LA, Huot S. Effectiveness of a focused educational intervention on resident evaluations from faculty (a randomized control trial). J Gen Intern Med. 2001;16:427–434
  25. Pangaro L. A new vocabulary and other innovations for improving descriptive in-training evaluations. Acad Med. 1999;74:1203–1207
  26. Holmboe ES. Faculty and the observation of trainees’ clinical skills (problems and opportunities). Acad Med. 2004;79:16–22
  27. Garibaldi RA, Subhiyah R, Moore ME, Waxman H. The in-training examination in internal medicine (an analysis of resident performance over time). Ann Intern Med. 2002;137:505–510
  28. Hawkins RE, Sumption KF, Gaglione MM, Holmboe ES. The in-training examination in internal medicine (resident perceptions and correlation between resident scores and faculty predictions of resident performance). Am J Med. 1999;106:206–210
  29. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX (a method for assessing clinical skills). Ann Intern Med. 2003;138:476–481
  30. Holmboe ES, Huot S, Hawkins RE. Construct validity of the mini clinical evaluation exercise. Acad Med. 2003;78:826–830
  31. Holmboe ES, Hawkins RE, Huot SJ. Direct observation of competence training (a randomized controlled trial). Ann Intern Med. 2004;140:874–881
  32. Matthews DA, Feinstein AR. A new instrument for patients’ ratings of physician performance in the hospital setting. J Gen Intern Med. 1989;4:14–22
  33. Butterfield PS, Mazzaferri EL. A new rating form for use by nurses in assessing residents’ humanistic behavior. J Gen Intern Med. 1991;6:155–161
  34. Lipner RS, Blank LL, Leas BF, Fortna GS. The value of patient and peer ratings in recertification. Acad Med. 2002;77:S64–S66
  35. Violato C, Lockyer J, Fidler H. Multisource feedback (a method of assessing surgical practice). BMJ. 2003;326:546–548
  36. Lockyer J. Multisource feedback in the assessment of physician competencies. J Contin Educ Health Prof. 2003;23:4–12
  37. Norcini JJ. Peer assessment of competence. Med Educ. 2003;37:539–543
  38. Jamtvedt G, Young JM, Kristoffersen JT, Thomson O’Brien MA, Oxman AD. Audit and feedback (effects on professional practice and health care outcomes). Cochrane Database Syst Rev. 2003;(3):CD000259
  39. Holmboe ES, Prince L, Green ML. Teaching and improving quality of care in a residency clinic. Acad Med. 2005;80:571–577
  40. Ogrinc G, Headrick LA, Mutha S, et al. A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review. Acad Med. 2003;78:748–756
  41. Djuricich AM, Ciccarelli M, Swigonski NL. A continuous quality improvement curriculum for residents (addressing core competencies, improving systems). Acad Med. 2004;79:S65–S67
  42. Ogrinc G, Headrick LA, Morrison LJ, Foster T. Teaching and assessing resident competence in practice-based learning and improvement. J Gen Intern Med. 2004;19:496–500
  43. Wong J, Holmboe ES, Huot S. Teaching and learning in an 80-hour work week (a novel day-float rotation for medical residents). J Gen Intern Med. 2004;19:519–523
  44. American Board of Internal Medicine. Practice improvement modules. Available at: https://www.abim.org/online/pim/DEFAULT.ASPX. Accessed September 30, 2005.
  45. E*value Residency Evaluation System. Available at: http://www.advancedinformatics.com/?id=3. Accessed March 15, 2006.

 Portions of this work were presented at the ACGME/ABMS Extreme Makeover Quadrad Meeting, Chicago, Ill, May 16, 2005.

PII: S0002-9343(06)00620-6

doi: 10.1016/j.amjmed.2006.05.031

The American Journal of Medicine
Volume 119, Issue 8 , Pages 708-714 , August 2006

Article Tools

* Image Usage & Resolution