Overconfidence as a Cause of Diagnostic Error in Medicine

  • Eta S. Berner
    Correspondence
    Requests for reprints should be addressed to Eta S. Berner, EdD, Department of Health Services Administration, School of Health Professions, University of Alabama at Birmingham, 1675 University Boulevard, Room 544, Birmingham, Alabama 35294-3361.
    Affiliations
    Department of Health Services Administration, School of Health Professions, University of Alabama at Birmingham, Birmingham, Alabama, USA
    Search for articles by this author
  • Mark L. Graber
    Affiliations
    VA Medical Center, Northport, New York and Department of Medicine, State University of New York at Stony Brook, Stony Brook, New York, USA
    Search for articles by this author

      Abstract

      The great majority of medical diagnoses are made using automatic, efficient cognitive processes, and these diagnoses are correct most of the time. This analytic review concerns the exceptions: the times when these cognitive processes fail and the final diagnosis is missed or wrong. We argue that physicians in general underappreciate the likelihood that their diagnoses are wrong and that this tendency to overconfidence is related to both intrinsic and systemically reinforced factors. We present a comprehensive review of the available literature and current thinking related to these issues. The review covers the incidence and impact of diagnostic error, data on physician overconfidence as a contributing cause of errors, strategies to improve the accuracy of diagnostic decision making, and recommendations for future research.

      Keywords

      To read this article in full you will need to make a payment

      Subscribe:

      Subscribe to The American Journal of Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Lowry F.
        Failure to perform autopsies means some MDs “walking in a fog of misplaced optimism.”.
        CMAJ. 1995; 153: 811-814
        • Mongerson P.
        A patient's perspective of medical informatics.
        J Am Med Inform Assoc. 1995; 2: 79-84
        • Blendon R.J.
        • DesRoches C.M.
        • Brodie M.
        • et al.
        Views of practicing physicians and the public on medical errors.
        N Engl J Med. 2002; 347: 1933-1940
      1. YouGov survey of medical misdiagnosis. Isabel Healthcare–Clinical Decision Support System, 2005. Available at: http://www.isabelhealthcare.com. Accessed April 3, 2006.

        • Burroughs T.E.
        • Waterman A.D.
        • Gallagher T.H.
        • et al.
        Patient concerns about medical errors in emergency departments.
        Acad Emerg Med. 2005; 23: 57-64
        • Tierney W.M.
        Adverse outpatient drug events—a problem and an opportunity.
        N Engl J Med. 2003; 348: 1587-1589
        • Norman G.R.
        • Coblentz C.L.
        • Brooks L.R.
        • Babcook C.J.
        Expertise in visual diagnosis: a review of the literature.
        Acad Med. 1992; 67: S78-S83
      2. Foucar E, Foucar MK. Medical error. In: Foucar MK, ed. Bone Marrow Patholog, 2nd ed. Chicago: ASCP Press, 2001:76–82.

        • Fitzgerald R.
        Error in radiology.
        Clin Radiol. 2001; 56: 938-946
        • Kronz J.D.
        • Westra W.H.
        • Epstein J.I.
        Mandatory second opinion surgical pathology at a large referral hospital.
        Cancer. 1999; 86: 2426-2435
        • Berlin L.
        • Hendrix R.W.
        Perceptual errors and negligence.
        Am J Radiol. 1998; 170: 863-867
      3. Kripalani S, Williams MV, Rask K. Reducing errors in the interpretation of plain radiographs and computed tomography scans. In: Shojania KG, Duncan BW, McDonald KM, Wachter RM, eds. Making Health Care Safer. A Critical Analysis of Patient Safety Practices. Rockville, MD: Agency for Healthcare Research and Quality, 2001.

        • Neale G.
        • Woloschynowych J.
        • Vincent C.
        Exploring the causes of adverse events in NHS hospital practice.
        J R Soc Med. 2001; 94: 322-330
        • O'Connor P.M.
        • Dowey K.E.
        • Bell P.M.
        • Irwin S.T.
        • Dearden C.H.
        Unnecessary delays in accident and emergency departments: do medical and surgical senior house officers need to vet admissions?.
        Acad Emerg Med. 1995; 12: 251-254
        • Chellis M.
        • Olson J.E.
        • Augustine J.
        • Hamilton G.C.
        Evaluation of missed diagnoses for patients admitted from the emergency department.
        Acad Emerg Med. 2001; 8: 125-130
        • Elstein A.S.
        Clinical reasoning in medicine.
        in: Higgs J.J.M. Clinical Reasoning in the Health Professions. Butterworth-Heinemann Ltd, Oxford, England1995: 49-59
        • Kedar I.
        • Ternullo J.L.
        • Weinrib C.E.
        • Kelleher K.M.
        • Brandling-Bennett H.
        • Kvedar J.C.
        Internet based consultations to transfer knowledge for patients requiring specialised care: retrospective case review.
        BMJ. 2003; 326: 696-699
        • McGinnis K.S.
        • Lessin S.R.
        • Elder D.E.
        Pathology review of cases presenting to a multidisciplinary pigmented lesion clinic.
        Arch Dermatol. 2002; 138: 617-621
        • Zarbo R.J.
        • Meier F.A.
        • Raab S.S.
        Error detection in anatomic pathology.
        Arch Pathol Lab Med. 2005; 129: 1237-1245
        • Tomaszewski J.E.
        • Bear H.D.
        • Connally J.A.
        • et al.
        Consensus conference on second opinions in diagnostic anatomic pathology: who, what, and when.
        Am J Clin Pathol. 2000; 114: 329-335
        • Harris M.
        • Hartley A.L.
        • Blair V.
        • et al.
        Sarcomas in north west England.
        Br J Cancer. 1991; 64: 315-320
        • Kim J.
        • Zelman R.J.
        • Fox M.A.
        • et al.
        Pathology Panel for Lymphoma Clinical Studies: a comprehensive analysis of cases accumulated since its inception.
        J Natl Cancer Inst. 1982; 68: 43-67
        • Goddard P.
        • Leslie A.
        • Jones A.
        • Wakeley C.
        • Kabala J.
        Error in radiology.
        Br J Radiol. 2001; 74: 949-951
        • Berlin L.
        Defending the ”missed” radiographic diagnosis.
        Am J Radiol. 2001; 176: 317-322
        • Espinosa J.A.
        • Nolan T.W.
        Reducing errors made by emergency physicians in interpreting radiographs: longitudinal study.
        BMJ. 2000; 320: 737-740
        • Arenson R.L.
        The wet read. AHRQ [Agency for Heathcare Research and Quality] Web M&M, March 2006.
        (Accessed November 28)
        • Beam C.A.
        • Layde P.M.
        • Sullivan D.C.
        Variability in the interpretation of screening mammograms by US radiologists: findings from a national sample.
        Arch Intern Med. 1996; 156: 209-213
        • Majid A.S.
        • de Paredes E.S.
        • Doherty R.D.
        • Sharma N.R.
        • Salvador X.
        Missed breast carcinoma: pitfalls and pearls.
        Radiographics. 2003; 23: 881-895
        • Goodson III, W.H.
        • Moore II, D.H.
        Causes of physician delay in the diagnosis of breast cancer.
        Arch Intern Med. 2002; 162: 1343-1348
        • Smith-Bindman R.
        • Chu P.W.
        • Miglioretti D.L.
        • et al.
        Comparison of screening mammography in the United States and the United Kingdom.
        JAMA. 2003; 290: 2129-2137
        • Schiff G.D.
        • Kim S.
        • Abrams R.
        • et al.
        Diagnosing diagnosis errors: lessons from a multi-institutional collaborative project.
        Advances in Patient Safety: From Research to Implementation. Agency for Healthcare Research and Quality, Rockville: MD2007 (vol 2. February 2005. AHRQ Publication No. 050021. Available at: http://www.ahrq.gov/downloads/pub/advances/vol2/schiff.pdf./. Accessed December 3)
      4. Shojania K, Burton E, McDonald K, et al. The autopsy as an outcome and performance measure: evidence report/technology assessment #58. Rockville, MD: Agency for Healthcare Research and Quality, October 2002. AHRQ Publication No. 03-E002.

        • Pidenda L.A.
        • Hathwar V.S.
        • Grand B.J.
        Clinical suspicion of fatal pulmonary embolism.
        Chest. 2001; 120: 791-795
        • Lederle F.A.
        • Parenti C.M.
        • Chute E.P.
        Ruptured abdominal aortic aneurysm: the internist as diagnostician.
        Am J Med. 1994; 96: 163-167
        • von Kodolitsch Y.
        • Schwartz A.G.
        • Nienaber C.A.
        Clinical prediction of acute aortic dissection.
        Arch Intern Med. 2000; 160: 2977-2982
        • Edlow J.A.
        Diagnosis of subarachnoid hemorrhage.
        Neurocrit Care. 2005; 2: 99-109
        • Burton E.C.
        • Troxclair D.A.
        • Newman III, W.P.
        Autopsy diagnoses of malignant neoplasms: how often are clinical diagnoses incorrect?.
        JAMA. 1998; 280: 1245-1248
        • Perlis R.H.
        Misdiagnosis of bipolar disorder.
        Am J Manag Care. 2005; 11: S271-S274
        • Graff L.
        • Russell J.
        • Seashore J.
        • et al.
        False-negative and false-positive errors in abdominal pain evaluation: failure to diagnose acute appendicitis and unneccessary surgery.
        Acad Emerg Med. 2000; 7: 1244-1255
        • Raab S.S.
        • Grzybicki D.M.
        • Janosky J.E.
        • et al.
        Clinical impact and frequency of anatomic pathology errors in cancer diagnoses.
        Cancer. 2005; 104: 2205-2213
        • Buchweitz O.
        • Wulfing P.
        • Malik E.
        Interobserver variability in the diagnosis of minimal and mild endometriosis.
        Eur J Obstet Gynecol Reprod Biol. 2005; 122: 213-217
        • Gorter S.
        • van der Heijde D.M.
        • van der Linden S.
        • et al.
        Psoriatic arthritis: performance of rheumatologists in daily practice.
        Ann Rheum Dis. 2002; 61: 219-224
        • Bogun F.
        • Anh D.
        • Kalahasty G.
        • et al.
        Misdiagnosis of atrial fibrillation and its clinical consequences.
        Am J Med. 2004; 117: 636-642
        • Arnon S.S.
        • Schecter R.
        • Maslanka S.E.
        • Jewell N.P.
        • Hatheway C.L.
        Human botulism immune globulin for the treatment of infant botulism.
        N Engl J Med. 2006; 354: 462-472
        • Edelman D.
        Outpatient diagnostic errors: unrecognized hyperglycemia.
        Eff Clin Pract. 2002; 5: 11-16
        • Russell N.J.
        • Pantin C.F.
        • Emerson P.A.
        • Crichton N.J.
        The role of chest radiography in patients presenting with anterior chest pain to the Accident & Emergency Department.
        J R Soc Med. 1988; 81: 626-628
      5. Dobbs D. Buried answers. New York Times Magazine. April 24, 2005:40–45.

        • Cabot R.C.
        Diagnostic pitfalls identified during a study of three thousand autopsies.
        JAMA. 1912; 59: 2295-2298
        • Cabot R.C.
        A study of mistaken diagnosis: based on the analysis of 1000 autopsies and a comparison with the clinical findings.
        JAMA. 1910; 55: 1343-1350
        • Aalten C.M.
        • Samsom M.M.
        • Jansen P.A.
        Diagnostic errors: the need to have autopsies.
        Neth J Med. 2006; 64: 186-190
        • Shojania K.G.
        Autopsy revelation. AHRQ [Agency for Heathcare Research and Quality] Web M&M, March 2004.
        (Accessed November 28)
        • Tamblyn R.M.
        Use of standardized patients in the assessment of medical practice.
        CMAJ. 1998; 158: 205-207
        • Berner E.S.
        • Houston T.K.
        • Ray M.N.
        • et al.
        Improving ambulatory prescribing safety with a handheld decision support system: a randomized controlled trial.
        J Am Med Inform Assoc. 2006; 13: 171-179
        • Christensen-Szalinski J.J.
        • Bushyhead J.B.
        Physician's use of probabalistic information in a real clinical setting.
        J Exp Psychol Hum Percept Perform. 1981; 7: 928-935
        • Peabody J.W.
        • Luck J.
        • Jain S.
        • Bertenthal D.
        • Glassman P.
        Assessing the accuracy of administrative data in health information systems.
        Med Care. 2004; 42: 1066-1072
        • Margo C.E.
        A pilot study in ophthalmology of inter-rater reliability in classifying diagnostic errors: an underinvestigated area of medical error.
        Qual Saf Health Care. 2003; 12: 416-420
        • Hoffman P.J.
        • Slovic P.
        • Rorer L.G.
        An analysis-of-variance model for the assessment of configural cue utilization in clinical judgment.
        Psychol Bull. 1968; 69: 338-349
        • Kohn L.
        • Corrigan J.M.
        • Donaldson M.
        To Err Is Human: Building a Safer Health System.
        National Academy Press, Washington, DC1999
        • Leape L.
        • Brennan T.A.
        • Laird N.
        • et al.
        The nature of adverse events in hospitalized patients: results of the Harvard Medical Practice Study II.
        N Engl J Med. 1991; 324: 377-384
        • Thomas E.J.
        • Studdert D.M.
        • Burstin H.R.
        • et al.
        Incidence and types of adverse events and negligent care in Utah and Colorado.
        Med Care. 2000; 38: 261-271
        • Baker G.R.
        • Norton P.G.
        • Flintoft V.
        • et al.
        The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada.
        CMAJ. 2004; 170: 1678-1686
        • Wilson R.M.
        • Harrison B.T.
        • Gibberd R.W.
        • Hamilton J.D.
        An analysis of the causes of adverse events from the Quality in Australian Health Care Study.
        Med J Aust. 1999; 170: 411-415
        • Davis P.
        • Lay-Yee R.
        • Briant R.
        • Ali W.
        • Scott A.
        • Schug S.
        Adverse events in New Zealand public hospitals II: preventability and clinical context.
        N Z Med J. 2003; 116: U624
        • Bhasale A.
        • Miller G.
        • Reid S.
        • Britt H.C.
        Analyzing potential harm in Australian general practice: an incident-monitoring study.
        Med J Aust. 1998; 169: 73-76
        • Makeham M.
        • Dovey S.
        • County M.
        • Kidd M.R.
        An international taxonomy for errors in general practice: a pilot study.
        Med J Aust. 2002; 177: 68-72
        • Fischer G.
        • Fetters M.D.
        • Munro A.P.
        • Goldman E.B.
        Adverse events in primary care identified from a risk-management database.
        J Fam Pract. 1997; 45: 40-46
        • Wu A.W.
        • Folkman S.
        • McPhee S.J.
        • Lo B.
        Do house officers learn from their mistakes?.
        JAMA. 1991; 265: 2089-2094
        • Weingart S.
        • Ship A.
        • Aronson M.
        Confidential clinical-reported surveillance of adverse events among medical inpatients.
        J Gen Intern Med. 2000; 15: 470-477
        • Balsamo R.R.
        • Brown M.D.
        Risk management.
        in: Sanbar S.S. Gibofsky A. Firestone M.H. LeBlang T.R. Legal Medicine, 4th ed. Mosby, St Louis, MO1998: 223-244
        • Failure to diagnose
        Midiagnosis of conditions and diseaes. Medical Malpractice Lawyers and Attorneys Online, 2006.
        (Accessed November 28)
      6. General and Family Practice Claim Summary. Physician Insurers Association of America, Rockville, MD, 2002.

        • Berlin L.
        Fear of cancer.
        AJR Am J Roentgenol. 2004; 183: 267-272
      7. Missed or failed diagnosis: what the UNITED claims history can tell us. United GP Registrar's Toolkit, 2005.
        (Accessed November 28)
        • Studdert D.M.
        • Mello M.M.
        • Gawande A.A.
        • et al.
        Claims, errors, and compensation payments in medical malpractice litigation.
        N Engl J Med. 2006; 354: 2024-2033
        • Schiff G.D.
        Commentary: diagnosis tracking and health reform.
        Am J Med Qual. 1994; 9: 149-152
        • Redelmeier D.A.
        Improving patient care: the cognitive psychology of missed diagnoses.
        Ann Intern Med. 2005; 142: 115-120
        • Gandhi T.K.
        • Kachalia A.
        • Thomas E.J.
        • et al.
        Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims.
        Ann Intern Med. 2006; 145: 488-496
        • Kirch W.
        • Schafii C.
        Misdiagnosis at a university hospital in 4 medical eras.
        Medicine (Baltimore). 1996; 75: 29-40
        • Goldman L.
        • Sayson R.
        • Robbins S.
        • Cohn L.H.
        • Bettmann M.
        • Weisberg M.
        The value of the autopsy in three different eras.
        N Engl J Med. 1983; 308: 1000-1005
        • Shojania K.G.
        • Burton E.C.
        • McDonald K.M.
        • Goldman L.
        Changes in rates of autopsy-detected diagnostic errors over time: a systematic review.
        JAMA. 2003; 289: 2849-2856
        • Sonderegger-Iseli K.
        • Burger S.
        • Muntwyler J.
        • Salomon F.
        Diagnostic errors in three medical eras: a necropsy study.
        Lancet. 2000; 355: 2027-2031
        • Berner E.S.
        • Miller R.A.
        • Graber M.L.
        Missed and delayed diagnoses in the ambulatory setting.
        Ann Intern Med. 2007; 146: 470-471
      8. Gawande A. Final cut. Medical arrogance and the decline of the autopsy. The New Yorker. March 19, 2001:94–99.

        • Kruger J.
        • Dunning D.
        Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments.
        J Pers Soc Psychol. 1999; 77: 1121-1134
      9. LaFee S. Well news: all the news that's fit. The San Diego Union-Tribune. March 7, 2006. Available at: http://www.quotegarden.com/medical.html. Accessed February 6, 2008.

        • Graber M.L.
        Diagnostic error in medicine: a case of neglect.
        Jt Comm J Qual Patient Saf. 2005; 31: 112-119
        • Covell D.G.
        • Uman G.C.
        • Manning P.R.
        Information needs in office practice: are they being met?.
        Ann Intern Med. 1985; 103: 596-599
        • Gorman P.N.
        • Helfand M.
        Information seeking in primary care: how physicians choose which clinical questions to pursue and which to leave unanswered.
        Med Decis Making. 1995; 15: 113-119
        • Osheroff J.A.
        • Bankowitz R.A.
        Physicians' use of computer software in answering clinical questions.
        Bull Med Libr Assoc. 1993; 81: 11-19
        • Rosenbloom S.T.
        • Geissbuhler A.J.
        • Dupont W.D.
        • et al.
        Effect of CPOE user interface design on user-initiated access to educational and patient information during clinical care.
        J Am Med Inform Assoc. 2005; 12: 458-473
        • McGlynn E.A.
        • Asch S.M.
        • Adams J.
        • et al.
        The quality of health care delivered to adults in the United States.
        N Engl J Med. 2003; 348: 2635-2645
        • Cabana M.D.
        • Rand C.S.
        • Powe N.R.
        • et al.
        Why don't physicians follow clinical practice guidelines?.
        JAMA. 1999; 282: 1458-1465
        • Eccles M.P.
        • Grimshaw J.M.
        Selecting, presenting and delivering clinical guidelines: are there any ”magic bullets”?.
        Med J Aust. 2004; 180: S52-S54
        • Pearson T.A.
        • Laurora I.
        • Chu H.
        • Kafonek S.
        The lipid treatment assessment project (L-TAP): a multicenter survey to evaluate the percentages of dyslipidemic patients receiving lipid-lowering therapy and achieving low-density lipoprotein cholesterol goals.
        Arch Intern Med. 2000; 160: 459-467
        • Eccles M.
        • McColl E.
        • Steen N.
        • et al.
        Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial [primary care].
        BMJ. 2002; 325: 941
        • Smith W.R.
        Evidence for the effectiveness of techniques to change physician behavior.
        Chest. 2000; 118: 8S-17S
      10. Militello L, Patterson ES, Tripp-Reimer T, et al. Clinical reminders: why don't people use them? In: Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting, New Orleans LA, 2004:1651–1655.

        • Patterson E.S.
        • Doebbeling B.N.
        • Fung C.H.
        • Militello L.
        • Anders S.
        • Asch S.M.
        Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods.
        J Biomed Inform. 2005; 38: 189-199
        • Berner E.S.
        • Maisiak R.S.
        • Heudebert G.R.
        • Young Jr, K.R.
        Clinician performance and prominence of diagnoses displayed by a clinical diagnostic decision support system.
        AMIA Annu Symp Proc. 2003; 2003: 76-80
        • Steinman M.A.
        • Fischer M.A.
        • Shlipak M.G.
        • et al.
        Clinician awareness of adherence to hypertension guidelines.
        Am J Med. 2004; 117: 747-754
        • Tierney W.M.
        • Overhage J.M.
        • Murray M.D.
        • et al.
        Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease.
        Health Serv Res. 2005; 40: 477-497
        • Timmermans S.
        • Mauck A.
        The promises and pitfalls of evidence-based medicine.
        Health Aff (Millwood). 2005; 24: 18-28
        • Tanenbaum S.J.
        Evidence and expertise: the challenge of the outcomes movement to medical professionalism.
        Acad Med. 1999; 74: 757-763
        • van der Sijs H.
        • Aarts J.
        • Vulto A.
        • Berg M.
        Overriding of drug safety alerts in computerized physician order entry.
        J Am Med Inform Assoc. 2006; 13: 138-147
        • Katz J.
        Why doctors don't disclose uncertainty.
        Hastings Cent Rep. 1984; 14: 35-44
        • Graber M.L.
        • Franklin N.
        • Gordon R.R.
        Diagnostic error in internal medicine.
        Arch Intern Med. 2005; 165: 1493-1499
        • Croskerry P.
        Achieving quality in clinical decision making: cognitive strategies and detection of bias.
        Acad Emerg Med. 2002; 9: 1184-1204
        • Friedman C.P.
        • Gatti G.G.
        • Franz T.M.
        • et al.
        Do physicians know when their diagnoses are correct.
        J Gen Intern Med. 2005; 20: 334-339
        • Dreiseitl S.
        • Binder M.
        Do physicians value decision support.
        Artif Intell Med. 2005; 33: 25-30
        • Baumann A.O.
        • Deber R.B.
        • Thompson G.G.
        Overconfidence among physicians and nurses: the 'micro-certainty, macro-uncertainty' phenomenon.
        Soc Sci Med. 1991; 32: 167-174
        • Podbregar M.
        • Voga G.
        • Krivec B.
        • Skale R.
        • Pareznik R.
        • Gabrscek L.
        Should we confirm our clinical diagnostic certainty by autopsies?.
        Intensive Care Med. 2001; 27: 1750-1755
        • Landefeld C.S.
        • Chren M.M.
        • Myers A.
        • Geller R.
        • Robbins S.
        • Goldman L.
        Diagnostic yield of the autopsy in a university hospital and a community hospital.
        N Engl J Med. 1988; 318: 1249-1254
        • Potchen E.J.
        Measuring observer performance in chest radiology: some experiences.
        J Am Coll Radiol. 2006; 3: 423-432
        • Kachalia A.
        • Gandhi T.K.
        • Puopolo A.L.
        • et al.
        Missed and delayed diagnoses in the emergency department: a study of closed malpractice claims from 4 liability insurers.
        Ann Emerg Med. 2007; 49: 196-205
        • Croskerry P.
        The importance of cognitive errors in diagnosis and strategies to minimize them.
        Acad Med. 2003; 78: 775-780
        • Bornstein B.H.
        • Emler A.C.
        Rationality in medical decision making: a review of the literature on doctors' decision-making biases.
        J Eval Clin Pract. 2001; 7: 97-107
        • McSherry D.
        Avoiding premature closure in sequential diagnosis.
        Artif Intell Med. 1997; 10: 269-283
        • Dubeau C.E.
        • Voytovich A.E.
        • Rippey R.M.
        Premature conclusions in the diagnosis of iron-deficiency anemia: cause and effect.
        Med Decis Making. 1986; 6: 169-173
        • Voytovich A.E.
        • Rippey R.M.
        • Suffredini A.
        Premature conclusions in diagnostic reasoning.
        J Med Educn. 1985; 60: 302-307
        • Simon H.A.
        The Sciences of the Artificial. 3rd ed. MIT Press, Cambridge, MA1996
        • Elstein A.S.
        • Shulman L.S.
        • Sprafka S.A.
        Medical Problem Solving. Harvard University Press, Cambridge, MA1978
        • Barrows H.S.
        • Norman G.R.
        • Neufeld V.R.
        • Feightner J.W.
        The clinical reasoning of randomly selected physicians in general medical practice.
        Clin Invest Med. 1982; 5: 49-55
        • Barrows H.S.
        • Feltovich P.J.
        The clinical reasoning process.
        Med Educ. 1987; 21: 86-91
        • Neufeld V.R.
        • Norman G.R.
        • Feightner J.W.
        • Barrows H.S.
        Clinical problem-solving by medical students: a cross-sectional and longitudinal analysis.
        Med Educ. 1981; 15: 315-322
        • Norman G.R.
        The epistemology of clinical reasoning: perspectives from philosophy, psychology, and neuroscience.
        Acad Med. 2000; 75: S127-S135
        • Schmidt H.G.
        • Norman G.R.
        • Boshuizen H.P.A.
        A cognitive perspective on medical expertise: theory and implications.
        Acad Med. 1990; 65: 611-621
        • Gladwell M.
        Blink: The Power of Thinking Without Thinking.
        Little Brown and Company, Boston2005
        • Klein G.
        Sources of Power: How People Make Decisions. MIT Press, Cambridge, MA1998
        • Rosch E.
        • Mervis C.B.
        Family resemblances: studies in the internal structure of categories.
        Cognit Psychol. 1975; 7: 573-605
        • Eva K.W.
        • Norman G.R.
        Heuristics and biases—a biased perspective on clinical reasoning.
        Med Educ. 2005; 39: 870-872
        • Sieck W.R.
        • Arkes H.R.
        The recalcitrance of overconfidence and its contribution to decision aid neglect.
        J Behav Decis Making. 2005; 18: 29-53
        • Kruger J.
        • Dunning D.
        Unskilled and unaware—but why.
        J Pers Soc Psychol. 2002; 82: 189-192
        • Krueger J.
        • Mueller R.A.
        Unskilled, unaware, or both.
        J Pers Soc Psychol. 2002; 82: 180-188
        • Mele A.R.
        Real self-deception.
        Behav Brain Sci. 1997; 20: 91-102
        • Reason J.T.
        • Manstead A.S.R.
        • Stradling S.G.
        Errors and violation on the roads: a real distinction.
        Ergonomics. 1990; 33: 1315-1332
        • Gigerenzer G.
        • Goldstein D.G.
        Reasoning the fast and frugal way: models of bounded rationality.
        Psychol Rev. 1996; 103: 650-669
        • Hamm R.M.
        Clinical intuition and clinical analysis: expertise and the cognitive continuum.
        in: Elstein A. Dowie J. Professional Judgment: A Reader in Clinical Decision Making. Cambridge University Press, Cambridge, UK1988: 78-105
        • Gigerenzer G.
        Adaptive Thinking. Oxford University Press, New York2000
        • Berner E.S.
        • Webster G.D.
        • Shugerman A.A.
        • et al.
        Performance of four computer-based diagnostic systems.
        N Engl J Med. 1994; 330: 1792-1796
        • Ely J.W.
        • Levinson W.
        • Elder N.C.
        • Mainous III, A.G.
        • Vinson D.C.
        Perceived causes of family physicians' errors.
        J Fam Pract. 1995; 40: 337-344
        • Studdert D.M.
        • Mello M.M.
        • Sage W.M.
        • et al.
        Defensive medicine among high-risk specialist physicians in a volatile malpractice environment.
        JAMA. 2005; 293: 2609-2617
        • Anderson R.E.
        Billions for defense: the pervasive nature of defensive medicine.
        Arch Intern Med. 1999; 159: 2399-2402
        • Trivers R.
        The elements of a scientific theory of self-deception.
        Ann N Y Acad Sci. 2000; 907: 114-131
        • Lundberg G.D.
        Low-tech autopsies in the era of high-tech medicine: continued value for quality assurance and patient safety.
        JAMA. 1998; 280: 1273-1274
        • Darwin C.
        The Descent of Man. Project Gutenberg, August 1, 2000.
        (Accessed November 28)
        • Leonhardt D.
        Why doctors so often get it wrong. The New York Times. February 22, 2006 [published correction appears in The New York Times, February 28, 2006].
        (Accessed November 28)
        • Hodges B.
        • Regehr G.
        • Martin D.
        Difficulties in recognizing one's own incompetence: novice physicians who are unskilled and unaware of it.
        Acad Med. 2001; 76: S87-S89
        • Davis D.A.
        • Mazmanian P.E.
        • Fordis M.
        • Van H.R.
        • Thorpe K.E.
        • Perrier L.
        Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
        JAMA. 2006; 296: 1094-1102
        • Davis D.
        • O'Brien M.A.
        • Freemantle N.
        • Wolf F.M.
        • Mazmanian P.
        • Taylor-Vaisey A.
        Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?.
        JAMA. 1999; 282: 867-874
        • Bowen J.L.
        Educational strategies to promote clinical diagnostic reasoning.
        N Engl J Med. 2006; 355: 2217-2225
        • Norman G.
        Building on experience—the development of clinical reasoning.
        N Engl J Med. 2006; 355: 2251-2252
        • Bordage G.
        Why did I miss the diagnosis.
        Acad Med. 1999; 74: S128-S143
        • Norman G.
        Research in clinical reasoning: past history and current trends.
        Med Educ. 2005; 39: 418-427
        • Singh H.
        • Petersen L.A.
        • Thomas E.J.
        Understanding diagnostic errors in medicine: a lesson from aviation.
        Qual Saf Health Care. 2006; 15: 159-164
      11. Croskerry P. When diagnoses fail: new insights, old thinking. Canadian Journal of CME. 2003;Nov:79–87.

        • Hall K.H.
        Reviewing intuitive decision making and uncertainty: the implications for medical education.
        Med Educ. 2002; 36: 216-224
        • Croskerry P.
        Cognitive forcing strategies in clinical decision making.
        Ann Emerg Med. 2003; 41: 110-120
        • Mitchell D.J.
        • Russo J.E.
        • Pennington N.
        Back to the future: temporal perspective in the explanation of events.
        J Behav Decis Making. 1989; 2: 25-38
        • Schon D.A.
        Educating the Reflective Practitioner.
        Jossey-Bass, San Francisco1987
        • Mamede S.
        • Schmidt H.G.
        The structure of reflective practice in medicine.
        Med Educ. 2004; 38: 1302-1308
      12. Soares SMS. Reflective practice in medicine (PhD thesis). Erasmus Universiteit, Rotterdam, Rotterdam, the Netherlands, 2006. 30552B 6000.

        • Mamede S.
        • Schmidt H.G.
        • Rikers R.
        Diagnostic errors and reflective practice in medicine.
        J Eval Clin Pract. 2007; 13: 138-145
        • Reason J.
        Human error: models and management.
        BMJ. 2000; 320: 768-770
        • Nolan T.W.
        System changes to improve patient safety.
        BMJ. 2000; 320: 771-773
        • Miller R.A.
        Medical diagnostic decision support systems—past, present, and future: a threaded bibliography and brief commentary.
        J Am Med Inform Assoc. 1994; 1: 8-27
        • Kassirer J.P.
        A report card on computer-assisted diagnosis—the grade: C.
        N Engl J Med. 1994; 330: 1824-1825
        • Berner E.S.
        • Maisiak R.S.
        Influence of case and physician characteristics on perceptions of decision support systems.
        J Am Med Inform Assoc. 1999; 6: 428-434
        • Friedman C.P.
        • Elstein A.S.
        • Wolf F.M.
        • et al.
        Enhancement of clinicians' diagnostic reasoning by computer-based consultation: a multisite study of 2 systems.
        JAMA. 1999; 282: 1851-1856
      13. Lincoln MJ, Turner CW, Haug PJ, et al. Iliad's role in the generalization of learning across a medical domain. Proc Annu Symp Comput Appl Med Care. 1992; 174–178.

        • Lincoln M.J.
        • Turner C.W.
        • Haug P.J.
        • et al.
        Iliad training enhances medical students' diagnostic skills.
        J Med Syst. 1991; 15: 93-110
        • Turner C.W.
        • Lincoln M.J.
        • Haug P.J.
        • et al.
        Iliad training effects: a cognitive model and empirical findings.
        Proc Annu Symp Comput Appl Med Care. 1991; : 68-72
        • Arene I.
        • Ahmed W.
        • Fox M.
        • Barr C.E.
        • Fisher K.
        Evaluation of quick medical reference (QMR) as a teaching tool.
        MD Comput. 1998; 15: 323-326
        • Apkon M.
        • Mattera J.A.
        • Lin Z.
        • et al.
        A randomized outpatient trial of a decision-support information technology tool.
        Arch Intern Med. 2005; 165: 2388-2394
        • Ramnarayan P.
        • Roberts G.C.
        • Coren M.
        • et al.
        Assessment of the potential impact of a reminder system on the reduction of diagnostic errors: a quasi-experimental study.
        BMC Med Inform Decis Mak. 2006; 6: 22
        • Maffei F.A.
        • Nazarian E.B.
        • Ramnarayan P.
        • Thomas N.J.
        • Rubenstein J.S.
        Use of a web-based tool to enhance medical student learning in the pediatric ICU and inpatient wards.
        Pediatr Crit Care Med. 2005; 6: 109
        • Ramnarayan P.
        • Tomlinson A.
        • Kularni G.
        • Rao A.
        • Britto J.
        A novel diagnostic aid (ISABEL): development and preliminary evaluation of clinical performance.
        Medinfo. 2004; 11: 1091-1095
        • Ramnarayan P.
        • Kapoor R.R.
        • Coren J.
        • et al.
        Measuring the impact of diagnostic decision support on the quality of clinical decision making: development of a reliable and valid composite score.
        J Am Med Inform Assoc. 2003; 10: 563-572
        • Ramnarayan P.
        • Tomlinson A.
        • Rao A.
        • Coren M.
        • Winrow A.
        • Britto J.
        ISABEL: a web-based differential diagnosis aid for paediatrics: results from an initial performance evaluation.
        Arch Dis Child. 2003; 88: 408-413
        • Miller R.A.
        Evaluating evaluations of medical diagnostic systems.
        J Am Med Inform Assoc. 1996; 3: 429-431
        • Berner E.S.
        Diagnostic decision support systems: how to determine the gold standard?.
        J Am Med Inform Assoc. 2003; 10: 608-610
        • Bauer B.A.
        • Lee M.
        • Bergstrom L.
        • et al.
        Internal medicine resident satisfaction with a diagnostic decision support system (DXplain) introduced on a teaching hospital service.
        Proc AMIA Symp. 2003; : 608-610
        • Teich J.M.
        • Merchia P.R.
        • Schmiz J.L.
        • Kuperman G.J.
        • Spurr C.D.
        • Bates D.W.
        Effects of computerized physician order entry on prescribing practices.
        Arch Intern Med. 2000; 160: 2741-2747
        • Croskerry P.
        The feedback sanction.
        Acad Emerg Med. 2000; 7: 1232-1238
        • Jamtvedt G.
        • Young J.M.
        • Kristoffersen D.T.
        • O'Brien M.A.
        • Oxman A.D.
        Does telling people what they have been doing change what they do?.
        Qual Saf Health Care. 2006; 15: 433-436
        • Papa F.J.
        • Aldrich D.
        • Schumacker R.E.
        The effects of immediate online feedback upon diagnostic performance.
        Acad Med. 1999; 74: S16-S18
        • Stone E.R.
        • Opel R.B.
        Training to improve calibration and discrimination: the effects of performance and environment feedback.
        Organ Behav Hum Decis Process. 2000; 83: 282-309
        • Duffy F.D.
        • Holmboe E.S.
        Self-assessment in lifelong learning and improving performance in practice: physician know thyself.
        JAMA. 2006; 296: 1137-1139
        • Borgstede J.P.
        • Zinninger M.D.
        Radiology and patient safety.
        Acad Radiol. 2004; 11: 322-332
        • Frable W.J.
        ”Litigation cells” in the Papanicolaou smear: extramural review of smears by ”experts.”.
        Arch Pathol Lab Med. 1997; 121: 292-295
        • Wilbur D.C.
        False negatives in focused rescreening of Papanicolaou smears: how frequently are ”abnormal” cells detected in retrospective review of smears preceding cancer or high grade intraepithelial neoplasia?.
        Arch Pathol Lab Med. 1997; 121: 273-276
      14. College of American Pathologists. Available at: http://www.cap.org/apps/cap.portal.

        • Raab S.S.
        Improving patient safety by examining pathology errors.
        Clin Lab Med. 2004; 24: 863
        • Nieminen P.
        • Kotaniemi L.
        • Hakama M.
        • et al.
        A randomised public-health trial on automation-assisted screening for cervical cancer in Finland: performance with 470,000 invitations.
        Int J Cancer. 2005; 115: 307-311
        • Nieminen P.
        • Kotaniemi-Talonen L.
        • Hakama M.
        • et al.
        Randomized evaluation trial on automation-assisted screening for cervical cancer: results after 777,000 invitations.
        J Med Screen. 2007; 14: 23-28
        • Fenton J.J.
        • Taplin S.H.
        • Carney P.A.
        • et al.
        Influence of computer-aided detection on performance of screening mammography.
        N Engl J Med. 2007; 356: 1399-1409
        • Hall F.M.
        Breast imaging and computer-aided detection.
        N Engl J Med. 2007; 356: 1464-1466
        • Hill R.B.
        • Anderson R.E.
        Autopsy: Medical Practice and Public Policy.
        Butterworth-Heinemann, Boston1988
        • Cases and commentaries
        AHRQ Web M&M, October 2007.
        (Accessed December 12)
        • Fischer M.A.
        • Mazor K.M.
        • Baril J.
        • Alper E.
        • DeMarco D.
        • Pugnaire M.
        Learning from mistakes: factors that influence how students and residents learn from medical errors.
        J Gen Intern Med. 2006; 21: 419-423
        • Patriquin L.
        • Kassarjian A.
        • Barish M.
        • et al.
        Postmortem whole-body magnetic resonance imaging as an adjunct to autopsy: preliminary clinical experience.
        J Magn Reson Imaging. 2001; 13: 277-287
      15. Umpire Information System (UIS). Available at: http://www.questec.com/q2001/prod_uis.htm. Accessed April 10, 2008.

        • Graber M.L.
        • Franklin N.
        • Gordon R.
        Reducing diagnostic error in medicine: what's the goal?.
        Acad Med. 2002; 77: 981-992
        • Deyo R.A.
        Cascade effects of medical technology.
        Annu Rev Public Health. 2002; 23: 23-44
        • Dijksterhuis A.
        • Bos M.W.
        • Nordgren L.F.
        • van Baaren R.B.
        On making the right choice: the deliberation-without-attention effect.
        Science. 2006; 311: 1005-1007
        • Redelmeier D.A.
        • Shafir E.
        Medical decision making in situations that offer multiple alternatives.
        JAMA. 1995; 273: 302-305
        • Miller R.A.
        • Masarie Jr, F.E.
        The demise of the ”Greek Oracle” model for medical diagnostic systems.
        Methods Inf Med. 1990; 29: 1-2
        • Tsai T.L.
        • Fridsma D.B.
        • Gatti G.
        Computer decision support as a source of interpretation error: the case of electrocardiograms.
        J Am Med Inform Assoc. 2003; 10: 478-483
        • Galletta D.F.
        • Durcikova A.
        • Everard A.
        • Jones B.M.
        Does spell-checking software need a warning label.
        Communications of the ACM. 2005; 48: 82-86
        • Tolstoy L.
        Anna Karenina. Project Gutenberg, July 1, 1998.
        (Accessed December 11)
      16. Committee on Quality of Health Care in America, Institute of Medicine Report. Washington, DC: The National Academy Press, 2001.

      17. Gladwell M. Million-dollar Murray. The New Yorker. February 13, 2006:96–107.