Advertisement

“Problem Residents”: Prevalence, Problems and Remediation in the Era of Core Competencies

      The American Board of Internal Medicine (ABIM) has defined the “problem resident” as a learner who demonstrates problem behaviors significant enough to require intervention by program leadership, typically the residency program director or chief resident.
      American Board of Internal Medicine. Chief Residents' Workshop on Problem Residents. Association of Program Directors in Internal Medicine (APDIM)
      It has been over a decade since Yao and Wright's report
      • Yao D.C.
      • Wright S.M.
      National survey of internal medicine residency program directors regarding problem residents.
      on the prevalence of “problem residents” in internal medicine residency training programs. Their survey of program directors reported performance problems in 6.9% of residents. We are unaware of any subsequent large studies in internal medicine of this important topic. Although the term “problem resident” has been used frequently, we will refer to these individuals as “residents in difficulty.”
      • Reamy B.V.
      • Harman J.H.
      Residents in trouble: an in-depth assessment of the 25-year experience of a single family medicine residency.
      The purpose of this study was to assess internal medicine program director experiences with residents in difficulty in the era of Accreditation Council for Graduate Medical Education (ACGME) competencies.
      Accreditation Council of Graduate Medical Education
      Common Program Reqiurements: General Competencies.
      • Internal medicine residents requiring remediation often have deficiencies in multiple competencies.
      • Deficiencies were identified in all competencies; remediation is most successful for medical knowledge (85.8%) and least successful for professionalism (41.2%).
      • Application materials rarely help program directors identify individuals at risk for difficulty during residency.
      • Performance deficiencies were rarely (5.6%) self-identified by residents.

      Materials and Methods

      Survey

      The Association of Program Directors in Internal Medicine Survey Committee develops and deploys an annual survey of internal medicine residency program directors to address important issues in graduate medical education. The survey process has been described.
      • Loertshcer L.L.
      • Halvorsen A.J.
      • Beasley B.W.
      • et al.
      Pharmaceutical industry support and residency education: a survey of internal medicine program directors.
      • Singh D.
      • McDonald F.S.
      • Beasley B.W.
      Demographic and work-life study of chief residents: a survey of the program directors in internal medicine residency programs in the United States.
      For this study, e-mail notifications with program-specific hyperlinks to a Web-based questionnaire were sent in August 2008 to each Association of Program Directors in Internal Medicine institutional member (program directors) (372), representing 97.1% of 383 US categoric internal medicine residencies.
      List of programs by specialty—internal medicine [database on the Internet].
      Reminders were sent to nonresponders biweekly until the survey closed in November 2008.
      The survey collected information on characteristics of the program and program directors in addition to the program director's experiences with residents in difficulty during the 2007-2008 academic year. The entire survey and results are available online.
      Survey responses were appended with data from publicly available sources, including the US Census Bureau,
      Census regions and divisions of the United States [database on the Internet].
      American Medical Association Fellowship and Residency Electronic Interactive Database Access System Online,
      Graduate medical education: FREIDA Online [database on the Internet].
      and ABIM.
      Residency program pass rates 2006-2009 [database on the Internet].
      Data from the public ACGME website included accreditation cycle length, government affiliations, number of approved and filled training positions, and program director appointment date.
      Accredited programs—internal medicine [database on the Internet].
      The study was approved by the Mayo Clinic Institutional Review Board.

      Data Analysis

      Program description, geographic region, ABIM 3-year rolling pass rate, government affiliation, accreditation cycle length, size, and program director tenure were compared among survey responders and nonresponders using Fisher exact tests or Welch's t tests. Logistic regression models were used to examine possible associations with program and program director characteristics for 2 outcomes: the proportion of residents in difficulty in a program and the ability of program directors with residents in difficulty to identify warning signs in application materials. All first-order interactions were assessed, and a conservative alpha level of 0.01 was used throughout. Models were fit using SAS statistical software (version 9.1; SAS Institute Inc, Cary, NC).

      Results

      Of 372 program directors, 268 (72.0%) completed the survey. A comparison between responders and nonresponders revealed no significant differences (Table 1).
      Table 1US Internal Medicine Residency Program Characteristics by 2008 Association of Program Directors in Internal Medicine Survey Response
      RespondersNonresponders
      Characteristic(N=268)(N=115)P Value
      Program description, No. (%)
      Fisher exact test.
      .13
       Community-based, university-affiliated hospital143 (53.4)55 (47.8)
       University-based86 (32.1)37 (32.2)
       Community-based35 (13.1)16 (13.9)
       Military4 (1.5)6 (5.2)
       Other0 (0.0)1 (0.9)
      Region, No. (%)
      Fisher exact test.
      .08
       Northeast102 (38.1)37 (32.2)
       South67 (25.0)32 (27.8)
       Midwest59 (22.0)26 (22.6)
       West38 (14.2)14 (12.2)
       Unincorporated territory2 (0.8)6 (5.2)
      Government affiliation, No. (%)
      Fisher exact test.
      .80
       No202 (75.4)85 (73.9)
       Yes66 (24.6)30 (26.1)
      ABIM pass rate, mean (SD)
      Welch's t test.
      %
      92.2 (6.2)92.7 (6.0).42
      Program director tenure, mean (SD)
      Welch's t test.
      y
      7.3 (6.2)6.0 (6.1).07
      Accreditation cycle length, mean (SD)
      Welch's t test.
      y
      4.2 (1.5)4.0 (1.3).18
      Program size, mean (SD)
      Welch's t test.
      approved positions
      62.4 (36.2)62.5 (40.8).98
      ABIM=American Board of Internal Medicine; SD=standard deviation.
      low asterisk Fisher exact test.
      Welch's t test.
      A total of 197 programs (73.5%) reported having residents in difficulty. The point prevalence for the 2007-2008 academic year was 3.5% (532 residents of 15,031 filled positions). The mean number of residents in difficulty per program was 2.9 with a median of 2. The majority of program directors (52.6%) reported that the number of residents in difficulty was unchanged when compared with previous years, with the percentages indicating an increase (18.7%) or decrease (23.9%) being similar.
      The majority of performance deficiencies (83.3%) in residents in difficulty were identified during inpatient rotations. Residents in difficulty also were identified in continuity clinics (32%), outpatient rotations (19.8%), and informal meetings (20.8%). Poor performance on the Internal Medicine In-Training Exam, end-of-rotation examinations, and US Medical Licensing Examination Step 3 identified residents in difficulty in 19.8%, 1.5%, and 1.0%, respectively. The most likely to identify residents in difficulty were faculty (69.5%), supervising residents (including chief residents) (62.9%), peers (52.8%), program directors (39.6%), and fellows (8.1%). Nurses (35.0%), nonclinical staff (14.7%), and nondepartment staff (8.1%) also identified residents in difficulty. Residents in difficulty infrequently were identified through patient complaints (8.1%) and adverse events (6.1%). Residents self-identified performance issues only 5.6% of the time.
      Contributing factors identified in residents in difficulty included depression, anxiety, and personality disorders, which were associated with 32.6% of residents in difficulty reports. Learning disability was a factor in 6.6% of residents in difficulty, whereas illness, substance abuse, and divorce were reported less than 5% of the time.
      Program directors reported multiple interventions to deal with residents in difficulty, most commonly informal discussion (52.3%). Formal remediation was reported in 38.2% of residents in difficulty. Disciplinary actions reported included formal warning (27.4%), “marginal” as defined by ABIM FasTrack
      American Board of Internal Medicine
      FasTrack.
      (13.7%), academic probation (12.6%), and dismissal (4.7%). Residents were required to repeat a rotation or an entire year in 15.4% and 3.9% of cases. A minority of residents, 7.9%, resigned from the programs.
      Deficiencies in performance were noted in all ACGME competencies, most commonly in patient care (53.0%), followed by medical knowledge (47.9%). Deficiencies in organization/prioritization, communication, and professionalism also were observed frequently, each occurring in more than 40% of residents in difficulty (Figure 1) . The majority of program directors (76.9%) reported that residents in difficulty had deficiencies in multiple competencies.
      Figure thumbnail gr1
      Figure 1Comparison of reported competency deficiency frequencies in 532 residents with program directors (n = 268) estimated the likelihood of successful remediation.
      Program directors reported varying success with remediation. Remediation was most successful for deficiencies in medical knowledge (85.9%) and least successful for problems with professionalism (48.6%) (Figure 1).
      Only 68 program directors (34.5%) with residents in difficulty could retrospectively identify warning signs in applications that might have predicted poor performance. Factors considered somewhat or very helpful by program directors included poor performance on US Medical Licensing Examination (57.4%), interview (42.6%), resident input (38.2%), dean's letters (27.9%), and letters of recommendation (16.2%).
      Community-based, university-affiliated programs were associated with a 62% increase in the odds of reporting residents in difficulty when compared with university-based programs (P=.001). Each 1% decrease in the ABIM 2006-2008 rolling pass rate was associated with a 2.6% increase in the odds of reporting residents in difficulty (P=.005). Programs with a program director annual salary of $250,000 or less were associated with a 2.25-fold increase in the odds of reporting residents in difficulty when compared with programs where the program director made more than $250,000 annually (P<.0001). Each additional year of tenure as a program director was associated with a 2.5% decrease in the odds of reporting residents in difficulty (P=.01).

      Discussion

      The majority of programs have residents in difficulty. Differences in methodology prevent a direct comparison of prevalence with that previously reported.
      • Yao D.C.
      • Wright S.M.
      National survey of internal medicine residency program directors regarding problem residents.
      The types of interventions required for remediation of residents in difficulty suggest that substantial resources are used for each resident. The persistence of residents in difficulty in most programs suggests that program directors will continue to need skills and resources for recognition and remediation of residents in difficulty.
      Program directors reported deficiencies in all 6 ACGME competencies, as well as in organization and prioritization. An important finding is that the majority of residents in difficulty have deficiencies in multiple competencies. This finding also has been reported in family medicine
      • Reamy B.V.
      • Harman J.H.
      Residents in trouble: an in-depth assessment of the 25-year experience of a single family medicine residency.
      and surgery.
      • Bergen P.C.
      • Littlefield J.H.
      • O'Keefe G.E.
      • et al.
      Identification of high-risk residents.
      • Resnick A.S.
      • Mullen J.L.
      • Kaiser L.R.
      • Morris J.B.
      Patterns and predictions of resident misbehavior—a 10-year retrospective look.
      • Williams R.G.
      • Roberts N.K.
      • Schwind C.J.
      • Dunnington G.L.
      The nature of general surgery resident performance problems.
      Of great importance, deficiencies in professionalism are relatively common in residents in difficulty (41.2%) but respond poorly to remediation. Likewise, poor outcomes remediating professionalism deficiencies have been reported in obstetric and gynecology
      • Adams K.E.
      • Emmons S.
      • Romm J.
      How resident unprofessional behavior is identified and managed: a program director survey.
      and surgical residents.
      • Williams R.G.
      • Roberts N.K.
      • Schwind C.J.
      • Dunnington G.L.
      The nature of general surgery resident performance problems.
      A recent systematic review found a paucity of literature reporting successful remediation of professionalism,
      • Hauer K.E.
      • Ciccone A.
      • Henzel T.R.
      • et al.
      Remediation of the deficiencies of physicians across the continuum from medical school to practice: a thematic review of the literature.
      which is especially relevant in light of studies noting that unprofessional behavior in both medical students and residents is predictive of subsequent disciplinary action by medical boards.
      • Papadakis M.A.
      • Arnold G.K.
      • Blank L.L.
      • et al.
      Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards.
      • Papadakis M.A.
      • Hodgson C.S.
      • Teherani A.
      • Kohatsu N.D.
      Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board.
      Program directors reported that residents rarely possess self-awareness to identify their own deficiencies. A systematic review of physicians (including learners) found a striking inability to assess their own performance, which supports our findings.
      • Davis D.A.
      • Mazmanian P.E.
      • Fordis M.
      • et al.
      Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
      Unfortunately, those physicians who are least skilled seem incapable of accurate self-assessment,
      • Kruger J.
      • Dunning D.
      Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments.
      supporting the need for multisource assessment.
      • Holmboe E.S.
      • Hawkins R.E.
      Methods for evaluating the clinical competence of residents in internal medicine: a review.
      The survey allowed multiple responses to the question about the highest level of intervention needed for residents in difficulty. A total of 988 actions were reported for the 532 residents in difficulty, suggesting that more than 1 action was required for some. The most frequent remediation methods reported were feedback and mentoring. Surveys of program directors in surgery
      • Torbeck L.
      • Canal D.F.
      Remediation practices for surgery residents.
      and obstetrics and gynecology
      • Adams K.E.
      • Emmons S.
      • Romm J.
      How resident unprofessional behavior is identified and managed: a program director survey.
      have outlined methods of remediation and estimation of success with residents in difficulty.
      Ideally, resident performance problems could be minimized through a more stringent selection of applicants. In this study, only one third of program directors thought that standard application materials were predictive of performance difficulties. A minority of program directors (15/268, 5.6%) reported a decrease in residents in difficulty because of improved selection processes. A systematic review found that academic success in medical school weakly predicted success as a resident.
      • Ferguson E.
      • James D.
      • Madeley L.
      Factors associated with success in medical school: systematic review of the literature.
      Although efforts to standardize the dean's letter may help provide more accurate evaluations of medical students,
      • Wagoner N.E.
      • Suriano J.R.
      A new approach to standardizing the dean's letter.
      less than one third of program directors reported it was somewhat or very helpful, even in retrospect, for identifying residents at risk for performance difficulties. The dean's letter or Medical Student Performance Evaluation may predict better performance when the student is ranked higher in comparative ratings.
      • Lurie S.J.
      • Lambert D.R.
      • Grady-Weliky T.A.
      Relationship between dean's letter rankings and later evaluations by residency program directors.
      • Cullen M.W.
      • Reed D.A.
      • Halvorsen A.J.
      • et al.
      Selection criteria for internal medicine residency applicants and professionalism ratings during internship.
      Conversely, negative comments in particular lapses in professionalism noted in the Medical Student Performance Evaluation may identify applicants at risk for future performance problems.
      • Brenner A.M.
      • Mathai S.
      • Jain S.
      • Mohl P.C.
      Can we predict “problem residents”?.
      Overall, we believe that simple screening strategies are unlikely to consistently and prospectively identify residents in difficulty.
      Four program-related factors were associated with a lower prevalence of residents in difficulty: higher ABIM pass rates, university-based training, higher salaried program directors, and longer tenured program directors. It is not clear why these factors were associated with a decreased risk for residents in difficulty, but it certainly is an area for further investigation.
      To our knowledge, this study is the first to report the relative success of remediation in specific ACGME competencies. One explanation for variable success of remediation may reflect limitations in assessment of specific competencies. For example, medical knowledge can be objectively measured and deficiencies can be addressed through assigned reading and study programs.
      • Edson R.S.
      • Beckman T.J.
      • West C.P.
      • et al.
      A multi-institutional survey of internal medicine residents' learning habits.
      Other competencies, such as professionalism and systems-based practice, are more difficult to assess and reassess after intervention.
      There are limitations to our study. Survey research is by definition observational and can therefore only assess associations rather than causality. There is intrinsic potential for recall bias; program directors could not report on residents with problems who did not come to their attention. The similarity of responders and nonresponders and the high response rate suggest that the measures of association are likely representative of the population of internal medicine residencies. Our survey results of internal medicine program directors demonstrate commonalities with other studies of this topic;
      • Yao D.C.
      • Wright S.M.
      National survey of internal medicine residency program directors regarding problem residents.
      • Bergen P.C.
      • Littlefield J.H.
      • O'Keefe G.E.
      • et al.
      Identification of high-risk residents.
      • Resnick A.S.
      • Mullen J.L.
      • Kaiser L.R.
      • Morris J.B.
      Patterns and predictions of resident misbehavior—a 10-year retrospective look.
      • Williams R.G.
      • Roberts N.K.
      • Schwind C.J.
      • Dunnington G.L.
      The nature of general surgery resident performance problems.
      • Adams K.E.
      • Emmons S.
      • Romm J.
      How resident unprofessional behavior is identified and managed: a program director survey.
      • Torbeck L.
      • Canal D.F.
      Remediation practices for surgery residents.
      • Tabby D.S.
      • Majeed M.H.
      • Schwartzman R.J.
      Problem neurology residents: a national survey.
      however, the findings may not be directly applicable to all disciplines. We cannot comment on how many residents required multiple interventions and which type of interventions were most successful for specific deficiencies. Future studies may help elucidate interventions most helpful for remediation of specific competency deficiencies.

      Conclusions

      Ten years after the landmark study by Yao and Wright,
      • Yao D.C.
      • Wright S.M.
      National survey of internal medicine residency program directors regarding problem residents.
      residents in difficulty remain a concern for internal medicine program directors. Our results suggest that current application materials do not provide the information needed to avoid selecting future residents in difficulty. The majority of residents in difficulty have problems in multiple competencies. A comprehensive and multisource evaluation system is necessary to identify residents in difficulty, because individual residents are unable to see their own deficient behaviors. Remediation remains difficult for all the competencies, but most troubling is the challenge of remediating unprofessional behavior.

      Acknowledgments

      The authors thank the Association of Program Directors in Internal Medicine, the members of the Survey Committee, and the residency program directors who completed this survey. The authors also thank the Mayo Clinic Survey Research Center for assistance with survey design and data collection.

      References

        • American Board of Internal Medicine. Chief Residents' Workshop on Problem Residents. Association of Program Directors in Internal Medicine (APDIM)
        April 19, 1999 (New Orleans, LA)
        • Yao D.C.
        • Wright S.M.
        National survey of internal medicine residency program directors regarding problem residents.
        JAMA. 2000; 284: 1099-1104
        • Reamy B.V.
        • Harman J.H.
        Residents in trouble: an in-depth assessment of the 25-year experience of a single family medicine residency.
        Fam Med. 2006; 38: 252-257
        • Accreditation Council of Graduate Medical Education
        Common Program Reqiurements: General Competencies.
        ([cited 2011 August, 23]) (Accessed December 1, 2008)
        • Loertshcer L.L.
        • Halvorsen A.J.
        • Beasley B.W.
        • et al.
        Pharmaceutical industry support and residency education: a survey of internal medicine program directors.
        Arch Intern Med. 2010; 170: 356-362
        • Singh D.
        • McDonald F.S.
        • Beasley B.W.
        Demographic and work-life study of chief residents: a survey of the program directors in internal medicine residency programs in the United States.
        J Grad Med Educ. 2009; 1: 150-154
      1. List of programs by specialty—internal medicine [database on the Internet].
        ([cited December 1, 2008]) (Accessed January 21, 2012)
        • Halvorsen A.J.
        APDIM 2008 Survey.
        2010 ([cited 2011 August 23]) (Accessed December 1, 2008)
      2. Census regions and divisions of the United States [database on the Internet].
        ([cited December 1, 2008]) (Accessed December 1, 2008)
      3. Graduate medical education: FREIDA Online [database on the Internet].
        ([cited December 1, 2008]) (Accessed January 21, 2012)
      4. Residency program pass rates 2006-2009 [database on the Internet].
        ([cited December 1, 2008]) (Accessed December 1, 2008)
      5. Accredited programs—internal medicine [database on the Internet].
        ([cited December 1, 2008]) (Accessed January 21, 2012)
        • American Board of Internal Medicine
        FasTrack.
        ([cited 2011 August 23]) American Board of Internal Medicine, Philadelphia2010 (Accessed December 1, 2008)
        • Bergen P.C.
        • Littlefield J.H.
        • O'Keefe G.E.
        • et al.
        Identification of high-risk residents.
        J Surg Res. 2000; 92: 239-244
        • Resnick A.S.
        • Mullen J.L.
        • Kaiser L.R.
        • Morris J.B.
        Patterns and predictions of resident misbehavior—a 10-year retrospective look.
        Curr Surg. 2006; 63: 418-425
        • Williams R.G.
        • Roberts N.K.
        • Schwind C.J.
        • Dunnington G.L.
        The nature of general surgery resident performance problems.
        Surgery. 2009; 145: 651-658
        • Adams K.E.
        • Emmons S.
        • Romm J.
        How resident unprofessional behavior is identified and managed: a program director survey.
        Am J Obstet Gynecol. 2008; 198 (e1-5): 692
        • Hauer K.E.
        • Ciccone A.
        • Henzel T.R.
        • et al.
        Remediation of the deficiencies of physicians across the continuum from medical school to practice: a thematic review of the literature.
        Acad Med. 2009; 84: 1822-1832
        • Papadakis M.A.
        • Arnold G.K.
        • Blank L.L.
        • et al.
        Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards.
        Ann Intern Med. 2008; 148: 869-876
        • Papadakis M.A.
        • Hodgson C.S.
        • Teherani A.
        • Kohatsu N.D.
        Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board.
        Acad Med. 2004; 79: 244-249
        • Davis D.A.
        • Mazmanian P.E.
        • Fordis M.
        • et al.
        Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
        JAMA. 2006; 296: 1094-1102
        • Kruger J.
        • Dunning D.
        Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments.
        J Pers Soc Psychol. 1999; 77: 1121-1134
        • Holmboe E.S.
        • Hawkins R.E.
        Methods for evaluating the clinical competence of residents in internal medicine: a review.
        Ann Intern Med. 1998; 129: 42-48
        • Torbeck L.
        • Canal D.F.
        Remediation practices for surgery residents.
        Am J Surg. 2009; 197: 397-402
        • Ferguson E.
        • James D.
        • Madeley L.
        Factors associated with success in medical school: systematic review of the literature.
        BMJ. 2002; 324: 952-957
        • Wagoner N.E.
        • Suriano J.R.
        A new approach to standardizing the dean's letter.
        Acad Med. 1989; 64: 688-689
        • Lurie S.J.
        • Lambert D.R.
        • Grady-Weliky T.A.
        Relationship between dean's letter rankings and later evaluations by residency program directors.
        Teach Learn Med. 2007; 19: 251-256
        • Cullen M.W.
        • Reed D.A.
        • Halvorsen A.J.
        • et al.
        Selection criteria for internal medicine residency applicants and professionalism ratings during internship.
        Mayo Clin Proc. 2011; 86: 197-202
        • Brenner A.M.
        • Mathai S.
        • Jain S.
        • Mohl P.C.
        Can we predict “problem residents”?.
        Acad Med. 2010; 85: 1147-1151
        • Edson R.S.
        • Beckman T.J.
        • West C.P.
        • et al.
        A multi-institutional survey of internal medicine residents' learning habits.
        Med Teach. 2010; 32: 773-775
        • Tabby D.S.
        • Majeed M.H.
        • Schwartzman R.J.
        Problem neurology residents: a national survey.
        Neurology. 2011; 76: 2119-2123