Advertisement

Grade Appeals in the Internal Medicine Clerkship: A National Survey and Recommendations for Improvement

      Perspectives Viewpoints
      • According to internal medicine clerkship directors, “concerns about matching for residency” is the commonest reason students give for appealing their grades.
      • Only 17% of internal medicine clerkship directors have received training in how to adjudicate grade appeals.
      • Eight percent of students requested grade clarifications, and 3% pursued formal grade appeals. Only 14% of grade appeals led to a grade change.
      • Clerkship directors recommend clear criteria for appeals, standardized procedures, use of external faculty reviewers, and electronic tracking of grade appeals.

      Introduction

      Changes in social norms over many decades have led students to expect a more transparent educational experience.
      • Lisby GC
      College student grade disputes: adjudicative vs. mediative models of conflict resolution.
      Medical schools have become more open and accountable, school administrators are more available, and the hierarchies of knowledge have leveled off in the internet age.
      • Dinkins S
      The dreaded grade appeal.
      With this increased transparency, medical student grade appeals and the factors that drive them have come under discussion. These factors may include grade inflation,
      • Fazio SB
      • Papp KK
      • Torre DM
      • Defer TM
      Grade inflation in the internal medicine clerkship: a national survey.
      ,
      • Schiel KZ
      • Everard KM
      Grade inflation in the family medicine clerkship.
      variation and imprecision of clerkship grading,
      • Alexander EK
      • Osman NY
      • Walling JL
      • Mitchell VG
      Variation and imprecision of clerkship grading in US medical schools.
      ,
      • Fazio SB
      • Torre DM
      • DeFer TM
      Grading practices and distributions across internal medicine clerkships.
      inadequate feedback in the clinical workplace,
      • Bing-You R
      • Hayes V
      • Varaklis K
      • Trowbridge R
      • Kemp H
      • McKelvy D
      Feedback for learners in medical education: what is known? A scoping review.
      and the perceived need for clerkship honors to attain residency goals.
      • Thomas LA
      • Milburn N
      • Kay A
      • Hatch E
      Factors associated with grade appeals: a survey of psychiatry clerkship directors.
      A formal grade appeal allows students the opportunity to voice concerns if they feel that their grade does not accurately reflect their performance due to an error, discrimination, or bias. Liaison Committee on Medical Education Standard 9.9 requires that medical schools have procedures that “permit a medical student to review and to challenge the student's educational records … if the student considers the information contained therein to be inaccurate, misleading, or inappropriate.”

      Liaison Committee on Medical Education (LCME). Functions and structures of a medical school. Available at: http://lcme.org/publications/#Standards. Accessed October 27, 2020.

      Despite accreditation requirements for having a procedure to challenge grades, little is known about the prevalence of grade appeals, how schools handle the appeals process, and what specific factors may be driving appeals. Thomas et al8 surveyed North American psychiatry clerkship directors and found an average grade challenge rate of 4.5% (range 0-17%). Clerkships with an honors/pass/fail grading system had a significantly higher challenge rate than clerkships with other schemas, and clerkships where a grade appeal could result in a lower grade had significantly lower challenge rates. Three main pathways for adjudicating a grade appeal were by the clerkship director only, by the clerkship director followed by a committee, or by the clerkship director followed by a dean or department chair.
      Clerkship grade appeals are complex, given the numbers and variability of evaluators, assessments, and clinical contexts. Guidance on how to create a standardized, consistent, and transparent grade appeals process is lacking. To increase our understanding of grade appeals and propose best practices for adjudicating them, we surveyed US internal medicine core clerkship directors about the frequency of grade inquiries and appeals, their institutional processes and policies for adjudication of grade appeals, and perceived changes in appeals rates and underlying factors.

      Methods

      Clerkship Directors in Internal Medicine (CDIM) is a charter organization of the Alliance for Academic Internal Medicine (AAIM), a nonprofit professional association that includes academic faculty and leaders responsible for third- and fourth-year undergraduate medical education. CDIM has conducted annual research surveys on undergraduate medical education topics since 1999.

      Survey Development

      In Spring 2019, CDIM issued a call for thematic survey section proposals to its physician-faculty members. The 18-member CDIM Survey and Scholarship Committee reviewed all submissions for relevance to the third- and fourth-year medicine training and selected 3 topics to include in the 2019 survey. After pilot-testing by the CDIM Council and by 6 noncommittee CDIM members, final revisions were made. All section authors, committee members, and Council members had extensive experience in clerkship leadership and direct teaching.
      The section on internal medicine clerkship grade appeals consisted of 22 questions, including multiple-choice, 5-point Likert scale, numeric entry, and open-text response options, as well as logical skip and display patterns. Due to conditional logic or item-nonresponse, denominators for some questions do not sum to the total number of survey respondents (see Supplementary Data Appendix A, available online, for the survey instrument). The 2019 Annual Survey was sent to 138 CDIM members designated as “clerkship director” in the AAIM membership database at US medical schools with full Liaison Committee on Medical Education (LCME) accreditation as of October 2019 (ie, one designated clerkship director per school was invited to complete the survey). Unique participant URLs were sent by e-mail invitation to the survey population through Qualtrics Surveys (Seattle, Wash; Version 08-2019). The study (#19-AAIM-108) was deemed exempt by Pearl Institutional Review Board (US Department of Health and Human Services OHRP #IRB00007772) according to Food and Drug Administration 21 CFR 56.104 and 45CFR46.104(b)(2): (2).
      The survey was sent from October 2 through December 9, 2019, and included 4 e-mail reminders to nonrespondents. No participants were offered incentives for participation.
      Prior to de-identifying the results, respondents’ records were merged with the complete survey population file to include demographics and medical school characteristics. Descriptive statistics were used to report the summary results, and Pearson's chi-squared or Fisher's exact test (α = 0.05) were used to test for associations between categorical variables. Data analysis was conducted in Stata 16.1 SE (StataCorp LLC, College Station, Texas).
      We developed the following definitions for the survey.
      • “Informal grade clarification (or explanation)”: A first step where a student contacts the clerkship director or equivalent educational leader to ask for an informal explanation of their grade. The educational leader may resolve the matter by communicating with the student by e-mail or in person.
      • “Formal grade appeal”: The formal process where, if the grade remains in dispute after the initial grade clarification, a student can initiate a formal review of the grade and provide additional contextual information. This process can include several tiers of formal grade appeals, where a student can appeal if they disagree with outcome of the initial appeal.
      We used thematic analysis
      • Vaismoradi M
      • Turunen H
      • Bondas T
      Content analysis and thematic analysis: implications for conducting a qualitative descriptive study.
      to review the domains and themes that emerged from 3 questions: “Why do you think the percentage of informal grade clarifications has changed?”; “Why do you think the percentage of formal grade appeals has changed?”; and “How would you improve the grade appeals process for internal medicine clerkships?” Three investigators (ND, NI, CL) independently read all the responses and developed preliminary codes with definitions and created a final coding book after discussion. The investigators independently applied the final coding book to the comments, and for discrepant codes, discussed until they reached consensus.

      Results

      The survey response rate was 82% (113/138). There were no statistically significant differences between respondents and non-respondents in core demographic and institutional characteristics (see Supplementary Data Appendix B, available online). Of the responding clerkship directors, 52% were female and 48% were male; 56% were from public medical schools and 44% were from private medical schools. Nineteen percent of respondents were full professors, 45% were associate professors, and 36% were assistant professors. The mean number of years in the clerkship director role was 7.6 (SD 6.8).

      Clerkship Grading Systems

      Ninety-two percent of respondents reported 3 or more grading tiers (eg, “honors/high pass/pass/fail” or “A-B-C-F”), and 8% reported that their clerkships were pass/fail. The estimated mean grade distributions were 34% in the top tier, 44% in the second tier, 20% in the third tier, and 1% in the fourth or fifth tier.

      Procedures for Grade Appeals

      Ninety-two percent of clerkship directors reported that their schools had a codified or policy-based process for grade appeals. Eighty-four percent of clerkship directors were “clear” or “very clear” about how the grade appeal process works; however, only 17% of clerkship directors had received formal training on how to adjudicate grade appeals. Fifty-nine percent of clerkship directors thought that their school's grade appeal policy was “clear” or “very clear” to students; 18% of clerkship directors did not know whether there was a process for students who disagreed with their first formal appeal to pursue a second formal appeal (see Appendix C, online).
      Eighty-three percent of clerkships had the clerkship director conduct the first formal appeal; 42% also had a dean or associate dean involved in the first appeal. Thirty-four percent reported that the clerkship director alone was responsible for the final decision on the appeal, and 66% reported that additional faculty members, a committee, or departmental or school leadership made the final decision. Sixty-eight percent of clerkships reported allowing a second appeal (with 61% going to a dean or associate dean, and 35% to departmental non-clerkship leadership or other committees), and 39% of clerkships allowed a third appeal (with 79% going to a dean or assistant dean and 18% to other committees).

      Grade Clarification and Formal Grade Appeal Trends

      Clerkship directors estimated that 8% of students in their clerkship (SD 7.0, range 0-30%) had requested a grade clarification and 3% (SD 4.3, range 0-20%) had initiated a formal grade appeal within the past year. Fourteen percent of grade appeals were estimated to have a led to a grade change. When asked how informal grade clarifications had changed over the past 3 years, 6% of clerkship directors reported that they had decreased, 37% reported no change, and 40% reported an increase. In that same time frame regarding formal grade appeals, 12% reported a decrease, 45% reported no change, and 21% reported an increase.

      Reasons for Students’ Grade Appeals

      When queried about the reasons for students’ formal grade appeals, clerkship directors reported, in order of frequency (ranging from “sometimes” to “very”): students’ concern about their ability to match for residency based on grade (73%); believing their improvement was not reflected in their grade (71%); feeling their evaluator(s) was at the extreme of grading tendencies, that is, a “tough” grader (64%); students not understanding the grading criteria (45%); perception that their evaluator(s) was biased due to a student's status in an underrepresented population (22%) (see Appendix D, available online).

      Impact on Clerkship Directors

      Most clerkship directors completed informal grade clarifications in 1 hour or less, whereas formal grade appeals commonly took 2-3 hours. The majority (86%) of clerkship directors described the formal grade appeals process as being stressful to some degree (ranging from “somewhat” to “very”).

      Factors Leading to Grade Clarifications and Grade Appeals

      Forty-six percent (52/113) of respondents reported a change in informal grade clarifications over the past 3 years, and 94% (49/52) of those respondents commented about the causes for the change. Thirty-two percent (36/113) of respondents reported a change in formal grade appeals over the past 3 years, and 81% (29/36) of those respondents commented about the causes. We identified 5 major themes (student factors, clerkship grading process, school grading systems, appeal process, and residency pressure) as causes of both grade clarifications and formal grade appeals. The themes, descriptions, and representative quotations are displayed in Table 1.
      Table 1Domains and Themes from Qualitative Analysis of Clerkship Director Responses to the Questions “Why Do You Believe that the Percentage of Informal Grade Clarifications has Changed?” and “Why Do You Believe that the Percentage of Formal Grade Appeals has Changed?”
      DomainThemeDescriptionRepresentative Quotations
      Student factorsComfort with inquiryStudent comfort with asking about their grades has increased the number of grade appeals.“[There is a] cultural shift among medical students who feel more comfortable inquiring about grades.”
      GenerationalDifferences in student behavior across generations is responsible for changes in current grade appeal climate.“[This is a] different generation of students with different expectations.”
      Overestimation of performanceStudents may overestimate abilities and expect higher grades than what was earned.“[Students] have elevated perceptions of their clinical abilities.”
      Concerns regarding grade validityStudents do not feel that clinical evaluations are a valid measure of performance and feel that more objective measures should be used.“Students are becoming increasingly skeptical of the underlying rationale and rubric behind individual evaluations.”
      Previous successSuccess of previous students in appealing grades increases the number of grade appeals.“Prior students have probably told underclassmen that you can argue and get your grade changed.”
      Clerkship grading processTransparency and clarity of gradingThe impact of transparency and clarity of the grading process on the number of grade clarifications or appeals.“Need for increased transparency from the students about grading for subjective domains of evaluation.”
      Grading processInputs related to the grading process included but not limited to clinical evaluations and grading criteria.“The grading rubric was changed to be consistent across all clerkships, and cut-offs for clinical ratings and shelf exams for the medicine clerkship were changed.”
      Grade inflationGrade inflation has contributed to the number of increased appeals.“With more students getting honors, there are higher expectations among students to achieve this grade.”
      School grading systemsPass/fail pre-clerkshipPass/fail grading in the pre-clerkship years may cause an increase in appeals in the clerkship years.“Our school recently went to just pass/fail in the pre-clinical years, so the clerkships are the first time they can have other grades. I wonder if this has changed their satisfaction/acceptance of a final grade?”
      Pass/fail during clerkshipPass/fail grading during the clerkship may lower the number of appeals in the clerkship.“[Appeals decreased] because we changed to a pass/fail curriculum.”
      Class rankingA school's class ranking system may impact the number of grade appeals.“The stakes are high because only the clerkships inform the final class rank.”
      Appeal processClarity and transparencyThe transparency and clarity of the appeal process either increases or decreases the number of appeals.“[We] explained the process to the students and that they have the right to appeal.”
      Open-door policyAccessibility of faculty/school leaders or clerkship directors impacts the ease to initiate a grade appeal“We have had a pretty open-door policy on asking for clarification on their grades.”
      Residency pressureStudent match pressureStress and anxiety of related to the residency match causes an increased number of grade appeals.“Underlying anxiety regarding the residency match has probably increased.”

      Proposals to Improve the Grading Process

      Fifty-eight percent (66/113) of respondents commented about how they would improve the grade appeal process. We identified 3 domains (grading process, appeal process, and faculty development) and 14 themes that were nested in these overarching domains. The themes, descriptions, and representative quotations are displayed in Appendix E (available online).

      Discussion

      In this first survey of US internal medicine clerkship directors on clerkship grade appeals, we report the institutional processes and policies for adjudication of grade appeals, estimated frequency of grade clarifications and appeals, and the factors that may account for appeals. Our findings add to the only other published study of clerkship grade appeals, which analyzed North American psychiatry clerkships.
      • Thomas LA
      • Milburn N
      • Kay A
      • Hatch E
      Factors associated with grade appeals: a survey of psychiatry clerkship directors.
      The majority (92%) of internal medicine clerkships had a defined process for conducting a formal grade appeal, and in most cases this review was led by the clerkship director. Nearly two-thirds of clerkships reported a process for a second appeal, and one-third allowed for a third appeal. Most clerkship directors believed that both informal grade clarifications and formal grade appeals were constant or increased over the past 3 years. Clerkship director perceptions of the drivers of informal grade clarifications and formal appeals centered around 5 themes: student factors, clerkship grading process, school grading systems, appeal process, and residency selection.
      Our internal medicine clerkship directors’ estimate of a 3% rate of formal grade appeals is similar to the rate of 4.5% reported by Thomas et al8 in psychiatry clerkships. Our finding of a 14% rate of appeals leading to grade changes indicates that only about 0.4% of all internal medicine clerkship students had their grade changed based on an appeal. An estimated 8% of students made grade clarification inquiries, which means that more than 60% of these students did not pursue formal grade appeals after discussing their grade with the clerkship director. This shows the importance of designating the clerkship director as a first responder to grade inquiries, which is the universal practice in both internal medicine and psychiatry clerkships.
      Only 17% of clerkship directors had received training in how to adjudicate grade appeals. Improvised and impromptu responses to grade appeals will not fulfill our obligation to have a fair and consistent process that is followed for all students. Medical schools should ensure that all clerkship directors receive formal training on the institution's grade appeals procedures. This would include training on the stepwise process for managing appeals and best practices for communicating with students and other stakeholders. This instruction would ideally be organized and delivered by the school (eg, by the Dean's office or a program evaluation and assessment committee). A national set of recommendations on grade appeals from the Association of American Medical Colleges could relieve each school of the need to invent its own grade appeals procedures and help to standardize the adjudication process.
      The vast majority (86%) of clerkship directors described clerkship grade appeals as “somewhat” or “very” stressful and reported a significant commitment of time to resolve them. Internal medicine clerkship directors are dealing with increases in both the number and complexity of their responsibilities, often with inadequate resources and full-time equivalent support.
      • Glod SA
      • Alexandraki I
      • Harish J
      • et al.
      Clerkship roles and responsibilities in a rapidly changing landscape: a national survey of internal medicine clerkship directors.
      With standardized and transparent procedures in place for grade appeals as well as faculty development on how to manage appeals, the time burden and stress that clerkship directors report feeling when dealing with grade appeals might be mitigated.
      To improve the grade appeals process in the internal medicine clerkship, clerkship directors recommended clear criteria for grade inquiries and appeals, standardized grade appeals processes, committees that may include external reviewers (eg, faculty educators outside the formal clerkship leadership who are familiar with grading rubrics) as part of the grade appeals process, and electronic tracking of grade appeals over time across clerkships and school wide (Table 2). These suggestions are congruent with the accreditation requirements in LCME Standard 9.9, which requires that schools provide a fair, formal, and timely process that is followed for all students, with transparent disclosure of the evidence used to determine the action taken and an opportunity for students to appeal any adverse decision or challenge data in their educational record.

      Liaison Committee on Medical Education (LCME). Functions and structures of a medical school. Available at: http://lcme.org/publications/#Standards. Accessed October 27, 2020.

      Table 2Clerkship Directors’ Recommendations to Improve the Grade Appeals Process for Internal Medicine Clerkships
      Establish clear criteria for informal grade clarifications and formal grade appeals
      • The school should establish formal reasons that students can appeal their grade. This can include concerns regarding discrimination or bias.
      Standardize the grade appeals process, including establishing deadlines for submission
      • Schools should establish a standardized grade appeals process, which applies to all clerkships. For example, first formal appeal is to the department directly, and then second formal appeal is to the School.
      • The steps of this process should be clear and transparent to students and clerkship directors.
      • Create a standard electronic form (eg, include grading criteria, criteria for appeals, and provide space for reason for appeal)
      • Establish a time limit for filing a grade appeal after a grade is submitted.
      Establish a grade appeals committee that reviews each formal appeal
      • Depending on the step of the formal grade appeals process, schools should consider adding reviewers from outside the formal clerkship leadership. This might include other medicine faculty educators with experience with student education and familiarity with grading criteria, faculty with diversity/equity/inclusion expertise, or a leader from another clerkship or dean's office.
      Track appeals over time and across clerkships at the School-level
      • A tracking system can help identify systems issues with particular clerkships, which could eventually help clerkship directors identify gaps and make a change to the clerkship grading process.
      Our survey demonstrated that many clerkship directors are concerned that residency match pressure, with its emphasis on clerkship grades, is a cause of clerkship grade appeals, particularly with the increase in pass/fail grading in the pre-clerkship years and the United States Medical Licensing Examination Step 1 examination transitioning to pass/fail grading by 2022. Many clerkship directors suggested that switching from a tiered grading system to pass/fail could alleviate stress around clerkship grades and thus reduce grade appeals; some clerkship directors reported that grade appeals decreased after changing to a pass/fail system. However, as of 2019, only 8% of LCME medical schools used pass/fail grading in their clerkships,

      Association of American Medical Colleges. Grading systems use by US medical schools. Available at: https://www.aamc.org/data-reports/curriculum-reports/interactive-data/grading-systems-use-us-medical-schools. Accessed October 27, 2020.

      and the prospect of a rapid and large-scale transition to pass/fail clerkship grading seems unlikely. Ultimately, grade appeals might be seen as a “canary in the coal mine” with regard to the rising residency match pressure caused by a stressful and ultracompetitive undergraduate-to-graduate medical education transition process.
      We received responses from 138 of 142 LCME fully-accredited US medical schools with CDIM membership at the time of the survey. The high survey response rate suggests that our results were statistically representative of the survey-eligible population. Our major limitation is that the data we report represent clerkship director perceptions without corroboration or supplementation by clerkship or school data on grade appeals, which raises the possibility that recall bias and affective bias (clerkship directors overestimating the burden of grade appeals) could have affected our results. Future studies should use prospective tracking of clerkship or school records to characterize the appeals process more accurately. Finally, we surveyed only clerkship directors and did not obtain the perspectives of other important stakeholders such as students or medical school deans, who could be surveyed in future studies.

      APPENDIX A. SURVEY INSTRUMENT

      2019 CDIM Annual Survey of Core Clerkship Directors

      2019 CDIM Annual Survey of Core Clerkship Directors
      For over 20 years, CDIM has been surveying its membership to understand the clinical clerkship in internal medicine (IM) and to advance undergraduate medical education. The aggregated CDIM Survey summary results are presented at academic medicine professional conferences, in peer-reviewed journals, and at www.im.org/data/cdim-surveys. In addition to three thematic sections, this survey includes a brief section on demographics as well as characteristics of your medical school. Depending on your responses, the survey should take approximately 20-25 minutes to complete. Immediately after completing the survey, you will receive your results by email.
      At any point, you may exit and return later without losing your data. Please use the unique survey link in your email invitation; you will be returned to where you left off. DO NOT USE your browser's "Back" or "Forward" buttons to navigate the survey. Instead, you must use the survey "<<BACK" and "NEXT>>" buttons at the bottom of each page. Please complete this survey by the deadline provided in your email invitation. After that date, you will not be able to submit your responses.
      This study (Number: 19-AAIM-108) is exempt by Pearl IRB (U.S. DHHS OHRP #IRB00007772) under Food and Drug Administration 21 CFR 56.104 and 45CFR46.104(b)(2). You have been invited to participate because you are an IM core clerkship director (or associate/co-clerkship director) whose medical school is LCME-accredited (full or preliminary) and a CDIM member as of October 2019. Participation is voluntary. Refusal to participate will not affect your or your institution's CDIM membership. The survey software will alert you if you leave certain questions empty, but you may skip any that you do not wish to answer. Before the CDIM Survey and Scholarship Committee accesses the survey dataset, all personal and institutional identifiers will be removed by AAIM Surveys staff, who serve as principal investigators, hold valid human subjects research training certificates, and will manage data collection.
      If you encounter technical problems or are not the most appropriate person to complete this survey, please contact AAIM Surveys staff at [email protected] or 703-341-4540. Questions about survey content may be directed to the CDIM Survey Committee Chair via [email protected] as well. If you feel that your rights as a participant have not been upheld, please contact Pearl IRB at [email protected] or 317-602-5917.
      *****ESSENTIAL FOR NAVIGATING THE SURVEY*****
      • 1.
        This survey is compatible with most tablet devices, but if you encounter technical problems please check that your device's operating system is updated. Use of smartphones is discouraged, due to programming that might cause unexpected errors or survey navigation problems. Regardless of the device you use, your data will be collected using Secure Socket Layer encryption.
      • 2.
        For further technical assistance and support FAQs about navigating this survey, please click here (a separate browser tab/window will open).
      Thank you for helping to advance your profession!

      Section II. Challenges and Complexities of Clerkship Grade Appeals

      For this section, please refer to the following definitions.
      “Informal grade clarification:” This is usually the first step where a student reaches out to the clerkship director or equivalent educational leader to ask for an informal explanation of their grade. The educational leader may resolve the matter by communicating with the student by email or in person.
      “Formal grade appeal:” The formal process where, if the grade remains in dispute after the initial grade clarification, a student can initiate a formal review of the grade and provide additional contextual information. This process can include several tiers of formal grade appeals, where a student can appeal if they disagree with the outcome of the initial appeal.
      Q18 Which of the following best describes the clerkship grading system at your medical school?
      A, B, C, F
      • Honors/Pass/Fail
      • Honors/High-Pass/Pass/Fail
      • Honors/High-Pass/Pass/Low-Pass/Fail
      • Pass/Fail
      • Other (please specify): ___________
      Q19 The following question requires the total percent to equal "100." If you cannot answer this question completely or do not wish to answer, click below to bypass the validation and proceed in the survey.
      • Click here if you cannot or do not wish to answer this question
      Q20 In a typical academic year, approximately what percentage of your students receives the following?
      Enter numerals only.
      • Top grade designation: _______
      • Second-tier grade designation: _______
      • Third-tier grade designation: _______
      • Fourth-tier grade designation: _______
      • Fifth-tier grade designation: _______
      • Total: ________
      Q21 Is there a codified (or policy-based) process for conducting a formal grade appeal in your clerkship?
      Answer carefully: your response will be used for subsequent questions.
      • No
      • Yes
      Skip To: Q29 If Is there a codified (or policy-based) process for conducting a formal grade appeal in your clerks... = No
      Q22 Who is involved in conducting the review for the initial formal grade appeal? Check all that apply.
      • Internal medicine clerkship leadership
      • Departmental non-clerkship leadership
      • School/College of Medicine/Office of Medical Education leadership (i.e., dean or associate dean)
      • Other (please specify): ___________
      • Don't know/Unsure
      Q23 For the initial formal grade appeal, who is ultimately responsible for making the final decision about whether the grade will be changed?
      • Internal medicine clerkship director only
      • Clerkship director and another faculty member
      • Clerkship grading committee
      • Committee of clerkship leaders and/or other medical educators
      • Departmental non-clerkship leadership (e.g., department chair, vice chair of education)
      • School/College of Medicine/Office of Medical Education
      • Other (please specify): ___________
      • Don't know/Unsure
      Q24 Is there a process in place for students who disagree with the outcome of a formal grade appeal to pursue another formal grade appeal (i.e., a second formal grade appeal)?
      • No
      • Yes
      • Don't know/Unsure
      Q25 To whom do they appeal in this second formal appeal?
      • Internal medicine clerkship leadership
      • Departmental non-clerkship leadership
      • School/College of Medicine/Office of Medical Education leadership (i.e., dean or associate dean)
      • Other committee (please specify): ___________
      • Don't know/Unsure
      Q26 Is there a process in place for students who disagree with the outcome of a formal grade appeal to pursue a third formal grade appeal?
      • No
      • Yes
      • Don't know/Unsure
      Q27 To whom do they appeal in this third formal grade appeal?
      • Internal medicine clerkship leadership
      • Departmental non-clerkship leadership
      • School/College of Medicine/Office of Medical Education leadership (i.e., dean or associate dean)
      • Other committee (please specify): ___________
      • Don't know/Unsure
      Q28 Regarding your clerkship's formal grades appeal process…
      Tabled 1
      Very UnclearUnclearNeutralClearVery Clear
      How clear are you about each step (from initial grade appeal to final, non-appealable decision)?
      How clear do you think the process is to students?
      Q29 Have you received training from the School of Medicine or Department of Medicine on how to adjudicate formal clerkship grade appeals?
      • No
      • Yes
      Q30 As a reminder:
      • “Informal grade clarification (or explanation)”: This is usually the first step where a student reaches out to the clerkship director or equivalent educational leader to ask for an informal explanation of their grade. The educational leader may resolve the matter by communicating with the student by email or in person.
      • “Formal grade appeal”: The formal process where, if the grade remains in dispute after the initial grade clarification, a student can initiate a formal review of the grade and provide additional contextual information. This process can include several tiers of formal grade appeals, where a student can appeal if they disagree with the outcome of the initial appeal.
      Q31 For a typical block of students in the past year, approximately what percent of those students requested an informal grade clarification?
      • Enter a whole number without decimal points: ___________
      • Don't know/Unsure
      Q32 In the past year, approximately what percent of students initiated a formal grade appeal with your clerkship?
      • Enter a whole number without decimal points: ___________
      • Don't know/Unsure
      Q33 Approximately what percent of the formal grade appeals led to a changed grade?
      • Enter a whole number without decimal points: ___________
      • Don't know/Unsure
      Q34 Over the past three years, how has the percentage of the following changed?
      Tabled 1
      DecreasedRemained ConstantIncreasedDon't Know / Unsure
      Informal grade clarifications (informal explanations of grade)
      Formal grade appeals
      Q35 Why do you believe that the percentage of informal grade clarifications has changed?
      Q36 Why do you believe that the percentage of formal grade appeals has changed?
      Q37 On average, how much time do you spend per student on the following processes (including reviewing files, facilitating and participating in the review process, responding and meeting with student)?
      Tabled 1
      30 minutes or less31 to 59 minutes1 hour>1 to 2 hours>2 to 3 hours>3 to 4 hours>4 to 5 hours>5 to 6 hours>6 to 7 hours>7 hoursN/A
      One typical informal grade clarification
      One typical formal grade appeal
      Q38 How stressful is the filing of a formal grade appeal for you?
      • Not at all stressful
      • Somewhat stressful
      • Stressful
      • Very stressful
      • Don't know/Unsure
      Q39 How frequently do students cite the following reasons for appealing their grade?
      Tabled 1
      Very InfrequentlyInfrequentlySometimesFrequentlyVery FrequentlyDon't know / Unsure
      Concern about ability to match for residency based on grade
      Feeling evaluator(s) was biased due to student's status as underrepresented in medicine
      Feeling evaluator(s) was at the extreme of grading tendencies (i.e., “tough grader”)
      Feeling their summary narrative for the MSPE should include different information
      Not understanding the grading criteria
      Perceiving discordance between verbal feedback and written evaluations
      Believing their improvement was not reflected in grade
      Believing their level of effort was not reflected in grade
      Perceived their performance was superior compared to peers
      Other (please explain):
      Q40 For your clerkship, to what extent do you think that grade inflation has affected the frequency of formal grade appeals?
      • To no extent
      • To a small extent
      • To a moderate extent
      • To a great extent
      • Don't know/Unsure
      Q41 How would you improve the grade appeals process for your clerkship?

      APPENDIX B

      Tabled 1Core Characteristics of Respondents and Nonrespondents: 2019 CDIM Annual Survey of Internal Medicine Core Clinical Clerkship Directors
      Respondents (n = 113)Nonrespondents (n = 25)
      n (Column %)n (Column %)P Value
      Bivariate test between “Respondents” and “Nonrespondents.” Pearson chi-squared with one degree of freedom: α ≤ 0.05; Fisher's exact test used when anticipated cells sizes were 5 or less.
      Medical school type
       Public63 (55.8)19 (76.0).074
       Private50 (44.3)6 (24.0)
      US Census region
      Suppresses one US territory-based medical school, due to small cell sizes/data confidentiality.
       Northeast29 (25.7)5 (20.0).619
       Midwest28 (24.8)5 (20.0).797
       South44 (38.9)8 (32.0).650
       West12 (10.6)6 (24.0).181
      Gender
      Self-reported, from Alliance for Academic Internal Medicine institutional membership database in October 2019.
       Female59 (52.5)12 (48.0).703
       Male54 (47.8)13 (52.0)
      Medical school size
      Self-reported, from Alliance for Academic Internal Medicine institutional membership database in October 2019.
       100 or more students43 (38.1)11 (44.0).653
       65-99 students37 (32.7)7 (28.0).813
       Less than 65 students30 (26.6)6 (24.0).990
       Medical School: No Internal Medicine Residency3 (2.7)1 (4.0).555
      low asterisk Bivariate test between “Respondents” and “Nonrespondents.” Pearson chi-squared with one degree of freedom: α ≤ 0.05; Fisher's exact test used when anticipated cells sizes were 5 or less.
      Suppresses one US territory-based medical school, due to small cell sizes/data confidentiality.
      Self-reported, from Alliance for Academic Internal Medicine institutional membership database in October 2019.

      APPENDIX C. PROCEDURES FOR GRADE APPEALS

      APPENDIX D. REASONS FOR STUDENTS’ GRADE APPEALS

      APPENDIX E

      Tabled 1Domains and Themes from Qualitative Analysis of Clerkship Directors’ Responses to the Question, “How Would You Improve the Grade Appeal Process for Your Clerkship?”
      DomainThemeDescriptionRepresentative Quotations
      Grading processPass/failShifting from a tiered grading system to pass/fail.“Our grading formula could (and will) be more straightforward starting next academic year, which I think will make the grades appeals process clearer as well.”
      Clarity and transparency of gradingIncreasing clarity and transparency of the grading process for students.“I feel like our algorithm for how grades happen keeps informal appeals to an absolute minimum—they either made it or they didn't.”
      Reliable dataThe need for more reliable data to determine grades.“I think good grade determination starts with good data. The evaluation process (which should always be scrutinized and improved, as continuous quality improvement) needs to be closer to the mark.”
      Grading committeeThe use of a grading committee, rather than an individual grader, to improve validity of grades.“Since we do not use a formulaic grading system, but rather a more holistic process with specialty-specific grading committees, the University policy covers us quite nicely.”
      Appeal processOutside reviewerAn outside reviewer, rather than a clerkship director, to review grade appeals.“Adding an outside reviewer in addition to the grading committee members for the initial formal appeal review [improves the process].”
      Standardized reason for appealAs part of a standardized process, students may only appeal for specific reasons.“Have very well delineated reasons for appeals and stick to them.”
      Time limitAs part of a standardized process, students may appeal only within the specified time frame.“After attending CDIM, we put in place a 14-day limit on grade appeals. We are hoping that will help.”
      Clarity and transparency of appeal processIncreasing clarity and transparency of the appeal process for students and clerkship directors.“I think a streamlined policy for each clerkship is needed … it should be transparent to student and clerkship director.”
      Faculty developmentClerkship director faculty trainingSpecific grade appeals training for the clerkship director.“It would also be great to have formal faculty development facilitated at the school level, since grade appeals procedures and best practices are issues that affect all clerkships, not just internal medicine.”
      Faculty and resident trainingSpecific faculty training for evaluators, including residents and faculty.“We have annual faculty and resident development sessions on RIME and effective narrative evaluation that have transformed how faculty and resident narrative evaluation is completed.”

      References

        • Lisby GC
        College student grade disputes: adjudicative vs. mediative models of conflict resolution.
        in: CNCR Hewlett Foundation Seed Grant White Papers. 2000 (Available at)
        • Dinkins S
        The dreaded grade appeal.
        in: Inside Higher Ed. 2009 (May 8Available at)
        • Fazio SB
        • Papp KK
        • Torre DM
        • Defer TM
        Grade inflation in the internal medicine clerkship: a national survey.
        Teach Learn Med. 2013; 25: 71-76
        • Schiel KZ
        • Everard KM
        Grade inflation in the family medicine clerkship.
        Fam Med. 2019; 51: 806-810
        • Alexander EK
        • Osman NY
        • Walling JL
        • Mitchell VG
        Variation and imprecision of clerkship grading in US medical schools.
        Acad Med. 2012; 87: 1070-1076
        • Fazio SB
        • Torre DM
        • DeFer TM
        Grading practices and distributions across internal medicine clerkships.
        Teach Learn Med. 2016; 28: 286-292
        • Bing-You R
        • Hayes V
        • Varaklis K
        • Trowbridge R
        • Kemp H
        • McKelvy D
        Feedback for learners in medical education: what is known? A scoping review.
        Acad Med. 2017; 92: 1346-1354
        • Thomas LA
        • Milburn N
        • Kay A
        • Hatch E
        Factors associated with grade appeals: a survey of psychiatry clerkship directors.
        Acad Psychiatry. 2018; 42: 354-356
      1. Liaison Committee on Medical Education (LCME). Functions and structures of a medical school. Available at: http://lcme.org/publications/#Standards. Accessed October 27, 2020.

        • Vaismoradi M
        • Turunen H
        • Bondas T
        Content analysis and thematic analysis: implications for conducting a qualitative descriptive study.
        Nurs Health Sci. 2013; 15: 398-405
        • Glod SA
        • Alexandraki I
        • Harish J
        • et al.
        Clerkship roles and responsibilities in a rapidly changing landscape: a national survey of internal medicine clerkship directors.
        J Gen Intern Med. 2020; 35: 1375-1381
      2. Association of American Medical Colleges. Grading systems use by US medical schools. Available at: https://www.aamc.org/data-reports/curriculum-reports/interactive-data/grading-systems-use-us-medical-schools. Accessed October 27, 2020.