Advertisement

Scores to Predict Major Bleeding Risk During Oral Anticoagulation Therapy: A Prospective Validation Study

      Abstract

      Background

      Clinical scores may help physicians to better assess the individual risk/benefit of oral anticoagulant therapy. We aimed to externally validate and compare the prognostic performance of 7 clinical prediction scores for major bleeding events during oral anticoagulation therapy.

      Methods

      We followed 515 adult patients taking oral anticoagulants to measure the first major bleeding event over a 12-month follow-up period. The performance of each score to predict the risk of major bleeding and the physician's subjective assessment of bleeding risk were compared with the C statistic.

      Results

      The cumulative incidence of a first major bleeding event during follow-up was 6.8% (35/515). According to the 7 scoring systems, the proportions of major bleeding ranged from 3.0% to 5.7% for low-risk, 6.7% to 9.9% for intermediate-risk, and 7.4% to 15.4% for high-risk patients. The overall predictive accuracy of the scores was poor, with the C statistic ranging from 0.54 to 0.61 and not significantly different from each other (P=.84). Only the Anticoagulation and Risk Factors in Atrial Fibrillation score performed slightly better than would be expected by chance (C statistic, 0.61; 95% confidence interval, 0.52-0.70). The performance of the scores was not statistically better than physicians' subjective risk assessments (C statistic, 0.55; P=.94).

      Conclusion

      The performance of 7 clinical scoring systems in predicting major bleeding events in patients receiving oral anticoagulation therapy was poor and not better than physicians' subjective assessments.

      Keywords

      Although the number of outpatient oral anticoagulant prescriptions in the United States increased 1.45-fold between 1998 and 2004 (from 21 to 31 million),
      • Wysowski D.K.
      • Nourjah P.
      • Swartz L.
      Bleeding complications with warfarin use: a prevalent adverse effect resulting in regulatory action.
      the risk/benefit evaluation of oral anticoagulant therapy remains challenging. To help physicians assess the bleeding risk of their patients taking oral anticoagulant therapy, 7 clinical prediction scoring systems have been derived since 1998 (Table 1). All 7 scoring systems stratify patients into 3 categories of bleeding risk (low, intermediate, and high) and show major bleeding rates varying from 2.8% to 5.7% in the low-risk category to 7.4% to 15.4% in the high-risk category.
      • Clinical scores did not accurately predict the risk of major bleeding in unselected patients receiving oral anticoagulants.
      • Clinical scores were not superior to subjective clinical judgment in predicting the risk of major bleeding.
      Table 1Characteristics of the Derivation Studies and Scores
      • a
        Study design
      • b
        Sample size
      • c
        Length of follow-up
      • d
        Indication of anticoagulation
      Major Bleeds DefinitionRisk Factors

      (Attributed Risk Points)
      Risk Categories
      OBRI
      • Beyth R.J.
      • Quinn L.M.
      • Landefeld C.S.
      Prospective evaluation of an index for predicting the risk of major bleeding in outpatients treated with warfarin.
      • a
        Retrospective
      • b
        556
      • c
        48 mo
      • d
        Mixed
      Overt bleed that led to the loss of2 units in7 d or was otherwise life-threateningAge65 y (1)

      Stroke (1)

      Gastrointestinal bleed (1)

      Recent myocardial infarction, anemia,
      Anemia was defined as a hematocrit<30% in OBRI and Shireman et al;4 as a hemoglobin level<13 g/L in men and <12 g/L in women in RIETE and ATRIA; and according to International Classification of Diseases, 9th Revision codes in HEMORR2HAGES.
      diabetes or creatinine>1.5 mg/dL (1)
      Low: 0

      Intermediate: 1-2

      High: 3-4
      Kuijer et al
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      • a
        Retrospective
      • b
        241
      • c
        3 mo
      • d
        Venous Thromboembolism
      Overt bleed and a decline in hemoglobin concentration of20 g/L if need for transfusion of2 U of red blood cells, if it was retroperitoneal or intracranial, or if it warranted permanent discontinuation of treatmentAge60 y (1.6)

      Female (1.3)

      Cancer (2.2)
      Low: 0

      Intermediate: 1-2

      High: ≥3
      Shireman et al
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      • a
        Retrospective
      • b
        19,875
      • c
        3 mo
      • d
        Atrial fibrillation
      Hospitalizations due to a gastrointestinal bleed or an intracranial bleed according to DRG and ICD-9 CM codesAge70 y (0.49)

      Female (0.32)

      Gastrointestinal bleed>10 d (0.58)

      Gastrointestinal bleed<10 d (0.62)

      Anemia
      Anemia was defined as a hematocrit<30% in OBRI and Shireman et al;4 as a hemoglobin level<13 g/L in men and <12 g/L in women in RIETE and ATRIA; and according to International Classification of Diseases, 9th Revision codes in HEMORR2HAGES.
      (0.86)

      Diabetes (0.27)

      Alcohol or drug abuse (0.71)

      Antiplatelet (0.32)
      Low: ≤1.07

      Intermediate: >1.07, <2.19

      High: ≥2.19
      HEMORR2HAGES
      • Gage B.F.
      • Yan Y.
      • Milligan P.E.
      • et al.
      Clinical classification schemes for predicting hemorrhage: results from the National Registry of Atrial Fibrillation (NRAF).
      • a
        Retrospective
      • b
        1604
      • c
        36 mo
      • d
        Atrial fibrillation
      Any hospitalization for hemorrhage determined by Medicare claimsAge75 y (1)

      Gastrointestinal bleed (2)

      Anemia
      Anemia was defined as a hematocrit<30% in OBRI and Shireman et al;4 as a hemoglobin level<13 g/L in men and <12 g/L in women in RIETE and ATRIA; and according to International Classification of Diseases, 9th Revision codes in HEMORR2HAGES.
      (1)

      Glomerular filtration rate<30 mL/min or hepatic disease (1)

      Cancer (1)

      Hypertension (1)

      CYP2C9 mutation (1)

      Alcohol abuse (1)

      Low platelet
      Low platelet count or platelet dysfunction based on chart review for aspirin use or thrombocytopenia.
      (1)

      Fall risk
      Based on chart review for high risk of falling, dementia, Parkinson's disease, or psychiatric disease.
      (1)
      Low: 0-1

      Intermediate: 2-3

      High: ≥4
      RIETE
      • Ruiz-Gimenez N.
      • Suarez C.
      • Gonzalez R.
      • et al.
      Predictive variables for major bleeding events in patients presenting with documented acute venous thromboembolism Findings from the RIETE Registry.
      • a
        Prospective
      • b
        13,057
      • c
        12 mo
      • d
        Atrial fibrillation
      Overt bleed that required a transfusion of2 U of blood, or was retroperitoneal, spinal or intracranial, or was fatalAge75 y (1)

      Major bleed<15 d (2)

      Anemia
      Anemia was defined as a hematocrit<30% in OBRI and Shireman et al;4 as a hemoglobin level<13 g/L in men and <12 g/L in women in RIETE and ATRIA; and according to International Classification of Diseases, 9th Revision codes in HEMORR2HAGES.
      (1.5)

      Creatinine>1.2 mg/dl (1.5)

      Cancer (1)

      Pulmonary embolism (1)
      Low: 0

      Intermediate: 1-4

      High: ≥5
      HAS-BLED
      • Pisters R.
      • Lane D.A.
      • Nieuwlaat R.
      • et al.
      A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey.
      • a
        Prospective
      • b
        2115
      • c
        12 mo
      • d
        Atrial fibrillation
      Any bleed requiring hospitalization or causing a decrease in hemoglobin level of>20 g/L or requiring blood transfusion that was not a hemorrhagic strokeAge65 y (1)

      Stroke (1)

      Gastrointestinal bleed (1)

      Creatinine2.3 mg/dL (1)

      Hepatic disease (1)

      Hypertension (1)

      Alcohol abuse (1)

      Drug abuse (1)

      Labile INR
      Defined as the time in therapeutic range<60%.
      (1)
      Low: 0

      Intermediate: 1-2

      High: ≥3
      ATRIA
      • Fang M.C.
      • Go A.S.
      • Chang Y.
      • et al.
      A new risk scheme to predict warfarin-associated hemorrhage The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study.
      • a
        Retrospective
      • b
        6123
      • c
        6 y
      • d
        Atrial fibrillation
      Fatal bleed, requiring transfusion of2 U of blood, or into a critical anatomic site (according to ICD-9 CM codes)Anemia (3)

      Glomerular filtration rate<30 mL/min (3)

      Age75 y (2)

      Prior bleed (1)

      Hypertension (1)
      Low: 0-3

      Intermediate: 4

      High: 5-10
      INR=international normalized ratio; OBRI=Outpatient Bleeding Risk Index; DRG=diagnosis-related group; ICD-9 CM=International Classification of Diseases, 9th Revision, Clinical Modification; HEMORR2HAGES = Hepatic or renal disease, Ethanol abuse, Malignancy, Older, Reduced platelet count or function, Rebleeding risk, Hypertension, Anemia, Genetic factors, Excessive fall risk, Stroke; RIETE=Registry of Patients with Venous Thromboembolism; HAS-BLED=Hypertension, Abnormal renal-liver function, Stroke, Bleeding history, Labile INR, Elderly, Drugs/alcohol; ATRIA=Anticoagulation and Risk Factors in Atrial Fibrillation.
      low asterisk Anemia was defined as a hematocrit<30% in OBRI and Shireman et al;
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      as a hemoglobin level<13 g/L in men and <12 g/L in women in RIETE and ATRIA; and according to International Classification of Diseases, 9th Revision codes in HEMORR2HAGES.
      Low platelet count or platelet dysfunction based on chart review for aspirin use or thrombocytopenia.
      Based on chart review for high risk of falling, dementia, Parkinson's disease, or psychiatric disease.
      § Defined as the time in therapeutic range<60%.
      Most of these scoring systems, however, are limited by a retrospective derivation study design,
      • Beyth R.J.
      • Quinn L.M.
      • Landefeld C.S.
      Prospective evaluation of an index for predicting the risk of major bleeding in outpatients treated with warfarin.
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      • Gage B.F.
      • Yan Y.
      • Milligan P.E.
      • et al.
      Clinical classification schemes for predicting hemorrhage: results from the National Registry of Atrial Fibrillation (NRAF).
      • Fang M.C.
      • Go A.S.
      • Chang Y.
      • et al.
      A new risk scheme to predict warfarin-associated hemorrhage The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study.
      applicability to patients with a specific indication for anticoagulation only (ie, atrial fibrillation or venous thromboembolism),
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      • Gage B.F.
      • Yan Y.
      • Milligan P.E.
      • et al.
      Clinical classification schemes for predicting hemorrhage: results from the National Registry of Atrial Fibrillation (NRAF).
      • Fang M.C.
      • Go A.S.
      • Chang Y.
      • et al.
      A new risk scheme to predict warfarin-associated hemorrhage The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study.
      • Ruiz-Gimenez N.
      • Suarez C.
      • Gonzalez R.
      • et al.
      Predictive variables for major bleeding events in patients presenting with documented acute venous thromboembolism Findings from the RIETE Registry.
      • Pisters R.
      • Lane D.A.
      • Nieuwlaat R.
      • et al.
      A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey.
      lack of external validation
      • Fang M.C.
      • Go A.S.
      • Chang Y.
      • et al.
      A new risk scheme to predict warfarin-associated hemorrhage The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study.
      • Pisters R.
      • Lane D.A.
      • Nieuwlaat R.
      • et al.
      A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey.
      or prospective validation,
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      • Ruiz-Gimenez N.
      • Suarez C.
      • Gonzalez R.
      • et al.
      Predictive variables for major bleeding events in patients presenting with documented acute venous thromboembolism Findings from the RIETE Registry.
      and lack of formal comparison with subjective physician judgment.
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      • Gage B.F.
      • Yan Y.
      • Milligan P.E.
      • et al.
      Clinical classification schemes for predicting hemorrhage: results from the National Registry of Atrial Fibrillation (NRAF).
      • Fang M.C.
      • Go A.S.
      • Chang Y.
      • et al.
      A new risk scheme to predict warfarin-associated hemorrhage The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study.
      • Ruiz-Gimenez N.
      • Suarez C.
      • Gonzalez R.
      • et al.
      Predictive variables for major bleeding events in patients presenting with documented acute venous thromboembolism Findings from the RIETE Registry.
      • Pisters R.
      • Lane D.A.
      • Nieuwlaat R.
      • et al.
      A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey.
      Thus, it is uncertain how these scores perform in a real-world setting of patients regardless of their anticoagulation indication and whether the scores are more accurate than physicians' subjective assessments. External validation of these scoring systems could help physicians choose the most accurate instrument to predict bleeding risk during oral anticoagulant therapy and to better weigh the individual risk-benefit for each patient. Our primary aim was to prospectively compare the performance of the 7 scoring systems to predict the risk of major bleeds in a real-world setting of internal medicine patients receiving an oral anticoagulant therapy, regardless of their anticoagulation indication. Our secondary aim was to compare the predictive performance of the scores with physicians' subjective assessments.

      Material and Methods

      Study Design and Setting

      This prospective cohort study was conducted in the Internal Medicine and Ambulatory Care and Community Medicine Departments of Lausanne University Hospital, Switzerland. The protocol was approved by the institutional review board, and all participants gave their informed consent.

      Study Population

      We included all consecutive patients aged ≥ 18 years who were receiving oral anticoagulant therapy at the time of hospital discharge or at presentation in the ambulatory clinic from January 1, 2008, to March 31, 2009. We identified patients receiving oral anticoagulation using the hospital's computerized physician order entry system for inpatients or on the basis of physician notification of study staff for outpatients. Oral anticoagulant therapy was defined as treatment with 1 of the 2 vitamin K antagonists available in Switzerland (acenocoumarol or phenprocoumon, which are comparable to warfarin
      • Barcellona D.
      • Vannini M.L.
      • Fenu L.
      • et al.
      Warfarin or acenocoumarol: which is better in the management of oral anticoagulants?.
      • Oliva Berini E.
      • Galan Alvarez P.
      • Pacheco Onrubia A.M.
      [Comparison of quality and hemorrhagic risk of oral anticoagulant therapy using acenocoumarol versus warfarin].
      • Beinema M.
      • Brouwers J.R.
      • Schalekamp T.
      • Wilffert B.
      Pharmacogenetic differences between warfarin, acenocoumarol and phenprocoumon.
      ). We excluded patients who were unwilling or unable to give informed consent for study participation.

      Data Collection

      A trained research nurse used standardized forms to collect patient baseline information at the time of hospital discharge or presentation to the ambulatory clinic. Data points collected included age, sex, indication for and commencement date of oral anticoagulation, international normalized ratio target levels, concomitant antiplatelet therapy, number of medications being taken, risk of falls, comorbid conditions (history of stroke, prior major bleeding, arterial hypertension, diabetes mellitus, angina pectoris, myocardial infarction, heart failure, peripheral artery disease, cancer [active or in remission < 1 year], liver cirrhosis, and alcohol or drug abuse), and laboratory values (hemoglobin, hematocrit, creatinine, platelet count, and international normalized ratio). The risk of falls was assessed by asking patients 2 validated screening questions:
      • Ganz D.A.
      • Bao Y.
      • Shekelle P.G.
      • Rubenstein L.Z.
      Will my patient fall?.
      (1) Did you fall during the last year? and if not, then (2) Did you notice any problem with gait, balance, or mobility? To assess the physicians' subjective assessments of bleeding risk, treating physicians (residents and fellows) were asked to report at patient discharge their estimated annual risk of major bleeds in their anticoagulated patients as low, intermediate, or high and on a continuous scale from 0% to 100%. Physician experience was measured as years in clinical practice. We did not collect any information on cytochrome CYP 2C9, because this mutation is only rarely measured in standard clinical practice. Likewise, we did not collect data on international normalized ratio lability because this information is only known in patients who were taking oral anticoagulants before enrollment. For calculating the scores, both variables were considered to be absent, a strategy used in the derivation of the Hypertension, Abnormal renal-liver function, Stroke, Bleeding history, Labile INR, Elderly, Drugs/alcohol (HAS-BLED) score.
      • Pisters R.
      • Lane D.A.
      • Nieuwlaat R.
      • et al.
      A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey.

      Outcome Measures

      The primary outcome was the first major bleeding event within 12 months after study enrollment, defined as a fatal bleed; or a symptomatic bleed in a critical area or organ, such as intracranial, intraspinal, intraocular with threat to vision, retroperitoneal, intra-articular, pericardial, or intramuscular with compartment syndrome; or a bleed causing a decrease in hemoglobin level of ≥ 20 g/L or leading to transfusion of ≥ 2 units of whole blood or red cells.
      • Schulman S.
      • Kearon C.
      Definition of major bleeding in clinical investigations of antihemostatic medicinal products in non-surgical patients.
      • Seto A.C.
      • Kenyon K.
      • Wittkowsky A.K.
      Discrepancies in identification of major bleeding events in patients taking warfarin.
      A fatal bleed was defined as any death occurring within 7 days of a major bleeding episode in the absence of an alternative cause of death.
      • Nieto J.A.
      • Solano R.
      • Ruiz-Ribo M.D.
      • et al.
      Fatal bleeding in patients receiving anticoagulant therapy for venous thromboembolism: findings from the RIETE registry.
      To ascertain outcomes during follow-up, the study nurse contacted patients or caregivers via telephone at 4, 8, and 12 months after enrollment. Primary care physicians also were contacted, and hospital records were reviewed for all outcomes. We collected the following information at each follow-up interview: all international normalized ratio values, current anticoagulant and antiplatelet treatment, date and location of any bleeding, and date and cause of any death.
      On the basis of the best available evidence from patient and physician interviews and medical charts, fatal and nonfatal major bleeds were adjudicated by a panel of 3 independent consultant general physicians who were blinded to the presence or absence of patient risk factors for bleeding. Final diagnoses necessitated the full consensus of this physician panel.
      • Bergqvist D.
      • Clement D.
      Adjudication of end points in studies on substances influencing haemostasis—an example from vascular surgery.

      Statistical Analyses

      We determined the cumulative incidence of the first major bleeding event in all patients. To compare the discriminative power of the test scores and physicians' subjective assessments, we calculated the C statistic (or area under the receiver operating characteristics curve).
      • Bartfay E.
      • Bartfay W.J.
      Accuracy assessment of prediction in patient outcomes.
      • Rosman A.S.
      • Korsten M.A.
      Application of summary receiver operating characteristics (sROC) analysis to diagnostic clinical testing.
      To assess whether score performances improved when applied to the specific disease population for whom they were originally developed, we also compared the C statistic in the subgroups of patients with atrial fibrillation and venous thromboembolism as appropriate for each test. We calculated the C statistic according to the derivation studies' length of follow-up if ≤ 12 months. Although all possible effort was made to find the cause of death of all patients who died during follow-up, the cause of death in some remained unknown, and in these cases major bleeding as the cause could not be formally excluded (competing risk); we therefore also performed a sensitivity analysis using as primary outcome all deaths from any cause in addition to the first major bleeding event. A 2-sided P value < .05 was considered statistically significant. All analyses were performed using STATA 11.1 (StataCorp LP, College Station, Tex).

      Results

      Study Population

      Of 650 consecutive patients receiving oral anticoagulant therapy who were screened between January 1, 2008, and March 31, 2009, we excluded 132 patients (20.8%) because of refusal or inability to give informed consent. We further excluded 3 patients (0.5%) who withdrew consent within a few weeks of the start of the study, leaving a final sample of 515 patients. The 3 patients who withdrew consent did not have a bleeding event before withdrawal. Excluded patients were significantly older than enrolled patients (median 81 vs 71 years, P < .001) and somewhat more likely to be women (45.5% vs 36.2%, P = .06). Follow-up was complete for all 515 enrolled patients.
      Baseline characteristics of our patient sample are shown in Table 2. Median age of the patients was 71.2 years (interquartile range, 61.6-79.3 years). Most patients were hospitalized at the time of enrollment (96%). The majority of the patients started anticoagulant therapy ≥ 3 months before enrollment (64.1%, n = 330). The mean duration of anticoagulation was 273 days (standard deviation, 127).
      Table 2Baseline Characteristics
      CharacteristicsNo. of Patients (n=515)Percentage or Median (IQR 25%-75%)
      Age
       <65 y17233.4
       65-70 y6713.1
       70-74 y7614.8
       ≥75 y20038.8
      Men32963.9
      Serum creatinine (mg/dL)5150.98 (0.83-1.24)
      Anemia (based on WHO criteria)23445.4
      Platelet count<150 G/L5911.4
      Comorbid disease:
      Stroke or transitory ischemic attack8015.5
      History of gastrointestinal bleed265.0
      Recent major bleed (<10 d)265.0
      Remote major bleed (>10 d)214.1
      Recent myocardial infarction (<10 d)397.6
      Remote myocardial infarction (>10 d)8616.7
      Angina pectoris142.7
      Peripheral vascular disease6111.8
      Diabetes mellitus13025.2
      Arterial hypertension31961.9
      Alcohol abuse438.4
      Drug abuse00
      Active cancer
      Active cancer or cancer in remission of<1 year.
      295.6
      Heart failure15329.7
      Liver cirrhosis61.2
      Increased risk of falls (defined in “Materials and Methods”)30860.0
      Indication for anticoagulation
      Patients could have>1 reason for oral anticoagulation therapy.
       Atrial fibrillation31461.0
       Artificial heart valve438.3
       Aortic position356.8
       Mitral position122.3
       Deep venous thrombosis428.2
       Pulmonary embolism8516.5
       Intramyocardial thrombus or cardioembolism81.6
       Other thrombus localizations
      Thrombus: mesenteric vein thrombosis (n=3), cervical vein thrombosis (n=1), Budd-Chiari syndrome (n=1), and septic jugular vein thrombosis (n=1).
      61.2
       Akinetic myocardium356.8
       Peripheral or coronary artery bypass163.1
       Pulmonary arterial hypertension71.4
       Other indications
      Other anticoagulation indication: mitral stenosis (n=2), cardiac thermoablation (n=2), antiphospholipid syndrome (n=2), mitral valve surgery (n=1), cardiopulmonary bypass n=1, and retinal artery occlusion (n=1).
      91.7
       Atrial fibrillation, heart valve, or thrombosis44887
      Concomitant treatment with antiplatelet drugs:
       Acetylsalicylic acid only10520.4
       Clopidogrel only112.1
       >1 antiplatelet drug458.7
       Nonsteroidal anti-inflammatory drug61.2
       Polypharmacy (≥4 drugs)43885.0
      IQR=interquartile range; WHO=World Health Organization.
      low asterisk Active cancer or cancer in remission of<1 year.
      Patients could have>1 reason for oral anticoagulation therapy.
      Thrombus: mesenteric vein thrombosis (n=3), cervical vein thrombosis (n=1), Budd-Chiari syndrome (n=1), and septic jugular vein thrombosis (n=1).
      § Other anticoagulation indication: mitral stenosis (n=2), cardiac thermoablation (n=2), antiphospholipid syndrome (n=2), mitral valve surgery (n=1), cardiopulmonary bypass n=1, and retinal artery occlusion (n=1).

      Description of Outcomes

      Of the 43 major bleeding events that occurred during follow-up, 35 (81.4%) were a first event. Of the first bleeding events, 28 (80%) occurred on oral anticoagulation, 6 (17.1%) were related to a trauma or a surgical procedure, and 5 (14.3%) were fatal. The 12-month cumulative incidence of first major bleeds and fatal bleeds was 6.8% and 1.0%, respectively. The incidence of major bleeds decreased over time, with 20 events (57.1%) occurring during the first 4 months, 10 events (28.6%) occurring during months 5 to 8, and 5 events (14.3%) occurring during months 9 to 12. Over-anticoagulation, defined as an international normalized ratio greater than the upper limit of target international normalized ratio range, was present in 9 (25.7%) of the 35 first bleeding events, 8 (22.9%) were receiving concomitant antiplatelet treatment, and 3 (8.6%) were receiving antiplatelet treatment alone. The most common site of major bleeding was gastrointestinal (38%, n = 13), followed by intracerebral (17%, n = 6), urogenital (14%, n = 5), ear, nose, and throat (11%, n = 4), and miscellaneous others (20%, including postprocedural, retroperitoneal, spinal, and pulmonary). Overall, 59 patients (11.5%) died of nonbleeding causes during follow-up (including cardiorespiratory arrest [n = 14], heart failure [n = 13], respiratory failure [n = 12], cancer [n = 6], other causes [n = 7], and unknown causes [n = 7]).

      Comparison of Major Bleeding and Mortality

      All scoring systems classified the majority of patients (43.7%-79.6%) in the intermediate-risk category, except the Shireman et al
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      and Anticoagulation and Risk Factors in Atrial Fibrillation (ATRIA)
      • Fang M.C.
      • Go A.S.
      • Chang Y.
      • et al.
      A new risk scheme to predict warfarin-associated hemorrhage The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study.
      studies, which classified most as low risk (75.2% and 58.3%, respectively) (Table 3). The proportion of major bleeds increased with increasing risk category, but the trend was not statistically significant, except for the ATRIA score (P = .01). In contrast, all-cause mortality rates increased significantly with increasing risk category for all score tests, except the HAS-BLED score.
      Table 3Patients' Risk of Major Bleeding and Outcomes at 12 Months (N=515)
      Risk Groups (Points)Patients N (%)Major Bleeds N (%)All-Cause Mortality N (%)
      OBRI
      • Beyth R.J.
      • Quinn L.M.
      • Landefeld C.S.
      Prospective evaluation of an index for predicting the risk of major bleeding in outpatients treated with warfarin.
       Low (0)99 (19.2)3 (3.0)3 (3.0)
       Intermediate (1-2)381 (74.0)29 (7.6)58 (15.2)
       High (3-4)35 (6.8)3 (8.6)3 (8.6)
      P=.12
      P for trend using Wilcoxon rank-sum trend test.
      P=.03
      P for trend using Wilcoxon rank-sum trend test.
      Kuijer et al
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
       Low (0)78 (15.2)3 (3.8)1 (1.3)
       Intermediate (1.3-2.99)410 (79.6)30 (7.3)54 (13.2)
       High (≥3)27 (5.2)2 (7.4)9 (33.3)
      P=.33
      P for trend using Wilcoxon rank-sum trend test.
      P=.00
      P for trend using Wilcoxon rank-sum trend test.
      Shireman et al
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
       Low (≤1.07)387 (75.2)22 (5.7)37 (9.6)
       Intermediate (1.07-2.19)121 (23.5)12 (9.9)27 (22.3)
       High (≥2.19)7 (1.3)1 (14.3)0 (0)
      P=.07
      P for trend using Wilcoxon rank-sum trend test.
      P=.004
      P for trend using Wilcoxon rank-sum trend test.
      HEMORR2HAGES
      • Gage B.F.
      • Yan Y.
      • Milligan P.E.
      • et al.
      Clinical classification schemes for predicting hemorrhage: results from the National Registry of Atrial Fibrillation (NRAF).
       Low (0-2)97 (18.8)3 (3.1)5 (5.2)
       Intermediate (2-3)225 (43.7)15 (6.7)25 (11.1)
       High (≥4)193 (37.5)17 (8.8)34 (17.6)
      P=.07
      P for trend using Wilcoxon rank-sum trend test.
      P=.002
      P for trend using Wilcoxon rank-sum trend test.
      RIETE
      • Ruiz-Gimenez N.
      • Suarez C.
      • Gonzalez R.
      • et al.
      Predictive variables for major bleeding events in patients presenting with documented acute venous thromboembolism Findings from the RIETE Registry.
       Low (0)138 (26.7)6 (4.3)5 (3.6)
       Intermediate (1-4)351 (68.2)25 (7.1)51 (14.5)
       High (≥4)26 (5.1)4 (15.4)8 (30.8)
      P=.06
      P for trend using Wilcoxon rank-sum trend test.
      P<.001
      P for trend using Wilcoxon rank-sum trend test.
      HAS-BLED
      • Pisters R.
      • Lane D.A.
      • Nieuwlaat R.
      • et al.
      A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey.
       Low (0)71 (13.8)2 (2.8)4 (5.6)
       Intermediate (1-2)349 (67.8)24 (6.9)48 (13.8)
       High (≥3)95 (18.4)9 (9.5)12 (21.6)
      P=.10
      P for trend using Wilcoxon rank-sum trend test.
      P=.24
      P for trend using Wilcoxon rank-sum trend test.
      ATRIA
      • Fang M.C.
      • Go A.S.
      • Chang Y.
      • et al.
      A new risk scheme to predict warfarin-associated hemorrhage The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study.
       Low (0-3)303 (58.3)15 (5.0)21 (6.9)
       Intermediate (4)65 (12.6)3 (4.6)7 (10.8)
       High (5-10)147 (28.5)17 (11.6)36 (24.5)
      P=.01
      P for trend using Wilcoxon rank-sum trend test.
      P<.001
      P for trend using Wilcoxon rank-sum trend test.
      OBRI=Outpatient Bleeding Risk Index; HEMORR2HAGES = Hepatic or renal disease, Ethanol abuse, Malignancy, Older, Reduced platelet count or function, Rebleeding risk, Hypertension, Anemia, Genetic factors, Excessive fall risk, Stroke; RIETE=Registry of Patients with Venous Thromboembolism; HAS-BLED=Hypertension, Abnormal renal-liver function, Stroke, Bleeding history, Labile INR, Elderly, Drugs/alcohol; ATRIA=Anticoagulation and Risk Factors in Atrial Fibrillation.
      low asterisk P for trend using Wilcoxon rank-sum trend test.

      Comparison of Predictive Accuracy and Discriminatory Power of Score Tests

      The discriminatory power of score tests in the whole cohort, expressed as the C statistics, ranged from 0.54 to 0.61 (Table 4), with no significant differences between the scores (P = .84). No score performed better than due to chance alone except the ATRIA score (C statistic, 0.61; 95% confidence interval [CI], 0.52-0.70). By using death from any cause and the first bleeding event as a combined outcome, the discriminatory power for all scores ranged from 0.55 (95% CI, 0.50-0.60) to 0.66 (95% CI, 0.60-0.72; results not shown).
      Table 4Discriminative Power of the Scores
      Prediction ScoreCurrent Study
      Statistical comparison across scores: P=.84.
      Same Indication as Derivation Study
      OBRI, all indications for the anticoagulation and outcomes were measured at 12 mo; Kuijer et al,3 only venous thromboembolism and outcomes were measured at 4 mo; Shireman et al,4 only atrial fibrillation and outcomes were measured at 4 mo; HEMORR2HAGES, only atrial fibrillation and outcomes were measured at 12 mo; RIETE, only venous thromboembolism and outcomes were measured at 4 mo; HAS-BLED, only atrial fibrillation and outcomes were measured at 12 mo; ATRIA, only atrial fibrillation and outcomes were measured at 12 mo.
      Derivation Study
      C-statistic (95% CI or SD)
      OBRI
      • Beyth R.J.
      • Quinn L.M.
      • Landefeld C.S.
      Prospective evaluation of an index for predicting the risk of major bleeding in outpatients treated with warfarin.
      0.56 (0.50-0.62)0.56 (0.50-0.62)0.72
      Kuijer et al
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      0.54 (0.48-0.60)0.50 (0.28-0.73)0.82 (0.66-0.98)
      Shireman et al
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      0.57 (0.48-0.65)0.59 (0.41-0.76)0.63
      HEMMOR2HAGES
      • Gage B.F.
      • Yan Y.
      • Milligan P.E.
      • et al.
      Clinical classification schemes for predicting hemorrhage: results from the National Registry of Atrial Fibrillation (NRAF).
      0.58 (0.50-0.67)0.59 (0.48-0.70)0.67 (0.04)
      RIETE
      • Ruiz-Gimenez N.
      • Suarez C.
      • Gonzalez R.
      • et al.
      Predictive variables for major bleeding events in patients presenting with documented acute venous thromboembolism Findings from the RIETE Registry.
      0.57 (0.49-0.65)0.28 (0.10-0.47)NA
      HAS-BLED
      • Pisters R.
      • Lane D.A.
      • Nieuwlaat R.
      • et al.
      A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey.
      0.57 (0.49-0.65)0.58 (0.49-0.68)0.72 (0.64-0.79)
      ATRIA
      • Fang M.C.
      • Go A.S.
      • Chang Y.
      • et al.
      A new risk scheme to predict warfarin-associated hemorrhage The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study.
      0.61 (0.52-0.70)0.65 (0.53-0.76)0.70 (0.70-0.78)
      CI=confidence interval; SD=standard deviation; OBRI=Outpatient Bleeding Risk Index; HEMMOR2HAGES = Hepatic or renal disease, Ethanol abuse, Malignancy, Older, Reduced platelet count or function, Rebleeding risk, Hypertension, Anemia, Genetic factors, Excessive fall risk, Stroke; RIETE=Registry of Patients with Venous Thromboembolism; HAS-BLED=Hypertension, Abnormal renal-liver function, Stroke, Bleeding history, Labile INR, Elderly, Drugs/alcohol; ATRIA=Anticoagulation and Risk Factors in Atrial Fibrillation; NA=not available.
      low asterisk Statistical comparison across scores: P=.84.
      OBRI, all indications for the anticoagulation and outcomes were measured at 12 mo; Kuijer et al,
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      only venous thromboembolism and outcomes were measured at 4 mo; Shireman et al,
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      only atrial fibrillation and outcomes were measured at 4 mo; HEMORR2HAGES, only atrial fibrillation and outcomes were measured at 12 mo; RIETE, only venous thromboembolism and outcomes were measured at 4 mo; HAS-BLED, only atrial fibrillation and outcomes were measured at 12 mo; ATRIA, only atrial fibrillation and outcomes were measured at 12 mo.
      We did not find any improvement in the performance of the scores when we restricted our analysis to patients with the specific disease (atrial fibrillation, venous thromboembolism) or follow-up period for which the scores were originally developed (Table 4).

      Physicians' Subjective Assessments

      The median clinical experience of the physicians was 3.0 (interquartile range, 2.0-4.5) years. The median annual risk of major bleeds based on physicians' subjective assessments was 5% (interquartile range, 2-10) when physicians were asked to estimate annual risk for each patient on a scale from 0% to 100%. Overall, 55%, 38%, and 7% of patients were estimated to have a low, intermediate, and high annual risk of bleeding, respectively. Physician estimation had a C statistic for accurately predicting major bleeds of 0.55 (95% CI, 0.46-0.65). The C statistics did not differ significantly between the predictive accuracy of the scoring systems and the physicians' subjective assessments (P = .94).

      Discussion

      We found that existing clinical scores poorly predict major bleeding events in unselected internal medicine patients receiving oral anticoagulant therapy. The C statistics ranged from 0.54 to 0.61 with 95% CIs crossing 0.50 for all scores except the ATRIA score. This suggests that the discriminatory power was not better than would have been expected due to chance alone for all scores except the ATRIA score, which performed only slightly better.
      • Hanley J.A.
      • McNeil B.J.
      The meaning and use of the area under a receiver operating characteristic (ROC) curve.
      Furthermore, overall score performances were not better than physicians' subjective assessments, which had a similar C statistic of 0.55. The 12-month cumulative incidence of a first major bleeding event on oral anticoagulant therapy was 6.8%, which is comparable to previous studies.
      • Wysowski D.K.
      • Nourjah P.
      • Swartz L.
      Bleeding complications with warfarin use: a prevalent adverse effect resulting in regulatory action.
      • Schulman S.
      • Beyth R.J.
      • Kearon C.
      • Levine M.N.
      Hemorrhagic complications of anticoagulant and thrombolytic treatment: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition).
      Our results contrast with the findings of the internal validation studies of these clinical scoring systems: In these, the C statistics for the prediction of major bleeds ranged from 0.63 to 0.82 (Table 4).
      • Beyth R.J.
      • Quinn L.M.
      • Landefeld C.S.
      Prospective evaluation of an index for predicting the risk of major bleeding in outpatients treated with warfarin.
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      • Gage B.F.
      • Yan Y.
      • Milligan P.E.
      • et al.
      Clinical classification schemes for predicting hemorrhage: results from the National Registry of Atrial Fibrillation (NRAF).
      • Fang M.C.
      • Go A.S.
      • Chang Y.
      • et al.
      A new risk scheme to predict warfarin-associated hemorrhage The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study.
      • Ruiz-Gimenez N.
      • Suarez C.
      • Gonzalez R.
      • et al.
      Predictive variables for major bleeding events in patients presenting with documented acute venous thromboembolism Findings from the RIETE Registry.
      • Pisters R.
      • Lane D.A.
      • Nieuwlaat R.
      • et al.
      A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey.
      For example, the C statistic in the derivation study by Kuijer et al
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      was 0.82 (95% CI, 0.66-0.98) compared with 0.54 (95% CI, 0.48-0.60) in our study. These differences may be attributable to several factors. First, with the exception of the Outpatient Bleeding Risk Index (OBRI), all scores were derived and internally validated in a sample of patients with specific diseases (eg, atrial fibrillation or venous thromboembolism), whereas we applied the scores to a patient population receiving oral anticoagulation for a varied range of indications. When we restricted the analyses to the clinical indications for which each score was originally intended, however, their predictive performances did not improve. Second, changes in the target anticoagulation therapeutic ranges, as well as the target population treated with oral anticoagulant therapy over these last decades, might have influenced bleeding risk. The most striking example is the OBRI, which was derived from data collected between 1977 and 1983 when more aggressive oral anticoagulant regimens were common.
      • Beyth R.J.
      • Quinn L.M.
      • Landefeld C.S.
      Prospective evaluation of an index for predicting the risk of major bleeding in outpatients treated with warfarin.
      Third, patients were followed for varying lengths of time in the different studies: 3 months in the Kuijer et al,
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      Shireman et al,
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      and Registry of Patients with Venous Thromboembolism (RIETE) studies;
      • Ruiz-Gimenez N.
      • Suarez C.
      • Gonzalez R.
      • et al.
      Predictive variables for major bleeding events in patients presenting with documented acute venous thromboembolism Findings from the RIETE Registry.
      12 months in the HAS-BLED;
      • Pisters R.
      • Lane D.A.
      • Nieuwlaat R.
      • et al.
      A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey.
      36 months in the Hepatic or renal disease, Ethanol abuse, Malignancy, Older, Reduced platelet count or function, Rebleeding risk, Hypertension, Anemia, Genetic factors, Excessive fall risk, Stroke (HEMORR2HAGES);
      • Gage B.F.
      • Yan Y.
      • Milligan P.E.
      • et al.
      Clinical classification schemes for predicting hemorrhage: results from the National Registry of Atrial Fibrillation (NRAF).
      48 months in the OBRI;
      • Beyth R.J.
      • Quinn L.M.
      • Landefeld C.S.
      Prospective evaluation of an index for predicting the risk of major bleeding in outpatients treated with warfarin.
      and 6 years in the ATRIA study.
      • Fang M.C.
      • Go A.S.
      • Chang Y.
      • et al.
      A new risk scheme to predict warfarin-associated hemorrhage The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study.
      Because bleeding risk is known to vary with time,
      • Fihn S.D.
      • McDonell M.
      • Martin D.
      • et al.
      Risk factors for complications of chronic anticoagulation A multicenter study. Warfarin Optimized Outpatient Follow-up Study Group.
      these different follow-up times may have contributed to different bleeding risks across the studies. Although we re-ran the analyses using the same anticoagulation indications as in the derivation studies, we were able to adjust for the follow-up duration only when these were 12 months or less. Finally, the definition of major bleeds varied across studies (Table 1), which is likely to have led to differences in the observed rates of major bleeds.
      Our study has clinical and research implications. Existing clinical bleeding risk scores have a limited accuracy and discriminative power and do not offer a clear benefit beyond physicians' subjective assessments alone. Thus, novel clinical risk assessment models are needed that accurately and reliably predict the risk of major bleeding in patients receiving oral anticoagulant therapy.
      Our study has several strengths. First, we prospectively identified consecutive patients receiving oral anticoagulation and no patient was lost to follow-up, reducing the risk of selection bias. Second, the inclusion of a broad sample of internal medicine patients increases the generalizability of our findings. Finally, all outcome events were adjudicated by a blinded, independent panel, reducing the risk of assessment bias.
      Our study also has several limitations. First, we excluded approximately 21% of patients, mainly because they were unwilling or unable to give informed consent. The excluded patients were older and more likely to have been sicker and have cognitive dysfunction. Therefore, we cannot rule out the possibility that these patients would have had a higher bleeding risk than our study sample. Nevertheless, the bleeding rates in our sample were comparable to those observed in previous prospective studies of patients receiving oral anticoagulant therapy.
      • Schulman S.
      • Beyth R.J.
      • Kearon C.
      • Levine M.N.
      Hemorrhagic complications of anticoagulant and thrombolytic treatment: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition).
      Second, we had neither information about CYP 2C9 mutations (included in the HEMORR2HAGES score) nor international normalized ratio lability (included in the HAS-BLED score) in our patient sample and assumed these to have been normal. The value of testing for the CYP 2C9 mutation in the prediction of bleeding risk is still controversial and not widely used in clinical practice.
      • Eckman M.H.
      • Rosand J.
      • Greenberg S.M.
      • Gage B.F.
      Cost-effectiveness of using pharmacogenetic information in warfarin dosing for patients with nonvalvular atrial fibrillation.
      • Rosove M.H.
      • Grody W.W.
      Should we be applying warfarin pharmacogenetics to clinical practice? No, not now.
      • Ginsburg G.S.
      • Voora D.
      The long and winding road to warfarin pharmacogenetic testing.
      Third, we enrolled both naive and non-naive patients to oral anticoagulants in our study. Although the derivation studies of most of the scores included both types of patients (Shireman et al,
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      HEMORR2HAGES, HAS-BLED, ATRIA), those of the Kuijer et al,
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      OBRI, and RIETE studies included only oral anticoagulant-naive patients. Thus, we cannot entirely exclude the possibility that these 3 scores would have shown a better prognostic accuracy in an oral anticoagulant-naive patient sample. Finally, our follow-up duration (12 months) was shorter than in the derivation studies of OBRI, HEMORR2HAGES, and ATRIA but longer than in the derivation studies of Kuijer et al,
      • Kuijer P.M.
      • Hutten B.A.
      • Prins M.H.
      • Buller H.R.
      Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
      Shireman et al,
      • Shireman T.I.
      • Mahnken J.D.
      • Howard P.A.
      • et al.
      Development of a contemporary bleeding risk model for elderly warfarin recipients.
      and RIETE, which may have decreased the discriminative power of these scores in our sample.

      Conclusions

      Our results indicate that clinical scores poorly predict the risk of major bleeding in unselected patients receiving oral anticoagulant therapy. Score performances do not seem to be better than physicians' subjective assessments. Novel clinical risk assessment methods that accurately and reliably predict the risk of major bleeds among those receiving oral anticoagulant therapy are needed.

      Acknowledgments

      The authors thank Dr Jeffrey Schnipper, assistant professor of medicine at the Harvard School of Medicine (Boston, MA), for careful review of the article. Dr J. Schnipper played no role in the design or conduct of the study, or in the collection, management, or interpretation of the data.

      References

        • Wysowski D.K.
        • Nourjah P.
        • Swartz L.
        Bleeding complications with warfarin use: a prevalent adverse effect resulting in regulatory action.
        Arch Intern Med. 2007; 167: 1414-1419
        • Beyth R.J.
        • Quinn L.M.
        • Landefeld C.S.
        Prospective evaluation of an index for predicting the risk of major bleeding in outpatients treated with warfarin.
        Am J Med. 1998; 105: 91-99
        • Kuijer P.M.
        • Hutten B.A.
        • Prins M.H.
        • Buller H.R.
        Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism.
        Arch Intern Med. 1999; 159: 457-460
        • Shireman T.I.
        • Mahnken J.D.
        • Howard P.A.
        • et al.
        Development of a contemporary bleeding risk model for elderly warfarin recipients.
        Chest. 2006; 130: 1390-1396
        • Gage B.F.
        • Yan Y.
        • Milligan P.E.
        • et al.
        Clinical classification schemes for predicting hemorrhage: results from the National Registry of Atrial Fibrillation (NRAF).
        Am Heart J. 2006; 151: 713-719
        • Fang M.C.
        • Go A.S.
        • Chang Y.
        • et al.
        A new risk scheme to predict warfarin-associated hemorrhage.
        J Am Coll Cardiol. 2011; 58: 395-401
        • Ruiz-Gimenez N.
        • Suarez C.
        • Gonzalez R.
        • et al.
        Predictive variables for major bleeding events in patients presenting with documented acute venous thromboembolism.
        Thromb Haemost. 2008; 100: 26-31
        • Pisters R.
        • Lane D.A.
        • Nieuwlaat R.
        • et al.
        A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey.
        Chest. 2010; 138: 1093-1100
        • Barcellona D.
        • Vannini M.L.
        • Fenu L.
        • et al.
        Warfarin or acenocoumarol: which is better in the management of oral anticoagulants?.
        Thromb Haemost. 1998; 80: 899-902
        • Oliva Berini E.
        • Galan Alvarez P.
        • Pacheco Onrubia A.M.
        [Comparison of quality and hemorrhagic risk of oral anticoagulant therapy using acenocoumarol versus warfarin].
        Med Clin (Barc). 2008; 131: 96-97
        • Beinema M.
        • Brouwers J.R.
        • Schalekamp T.
        • Wilffert B.
        Pharmacogenetic differences between warfarin, acenocoumarol and phenprocoumon.
        Thromb Haemost. 2008; 100: 1052-1057
        • Ganz D.A.
        • Bao Y.
        • Shekelle P.G.
        • Rubenstein L.Z.
        Will my patient fall?.
        JAMA. 2007; 297: 77-86
        • Schulman S.
        • Kearon C.
        Definition of major bleeding in clinical investigations of antihemostatic medicinal products in non-surgical patients.
        J Thromb Haemost. 2005; 3: 692-694
        • Seto A.C.
        • Kenyon K.
        • Wittkowsky A.K.
        Discrepancies in identification of major bleeding events in patients taking warfarin.
        Pharmacotherapy. 2008; 28: 1098-1103
        • Nieto J.A.
        • Solano R.
        • Ruiz-Ribo M.D.
        • et al.
        Fatal bleeding in patients receiving anticoagulant therapy for venous thromboembolism: findings from the RIETE registry.
        J Thromb Haemost. 2010; 8: 1216-1222
        • Bergqvist D.
        • Clement D.
        Adjudication of end points in studies on substances influencing haemostasis—an example from vascular surgery.
        Eur J Vasc Endovasc Surg. 2008; 36: 703-704
        • Bartfay E.
        • Bartfay W.J.
        Accuracy assessment of prediction in patient outcomes.
        J Eval Clin Pract. 2008; 14: 1-10
        • Rosman A.S.
        • Korsten M.A.
        Application of summary receiver operating characteristics (sROC) analysis to diagnostic clinical testing.
        Adv Med Sci. 2007; 52: 76-82
        • Hanley J.A.
        • McNeil B.J.
        The meaning and use of the area under a receiver operating characteristic (ROC) curve.
        Radiology. 1982; 143: 29-36
        • Schulman S.
        • Beyth R.J.
        • Kearon C.
        • Levine M.N.
        Hemorrhagic complications of anticoagulant and thrombolytic treatment: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition).
        Chest. 2008; 133: 257S-298S
        • Fihn S.D.
        • McDonell M.
        • Martin D.
        • et al.
        Risk factors for complications of chronic anticoagulation.
        Ann Intern Med. 1993; 118: 511-520
        • Eckman M.H.
        • Rosand J.
        • Greenberg S.M.
        • Gage B.F.
        Cost-effectiveness of using pharmacogenetic information in warfarin dosing for patients with nonvalvular atrial fibrillation.
        Ann Intern Med. 2009; 150: 73-83
        • Rosove M.H.
        • Grody W.W.
        Should we be applying warfarin pharmacogenetics to clinical practice?.
        Ann Intern Med. 2009; 151 (W295): 270-273
        • Ginsburg G.S.
        • Voora D.
        The long and winding road to warfarin pharmacogenetic testing.
        J Am Coll Cardiol. 2010; 55: 2813-2815