Advertisement

Disorganized Systematic Reviews and Meta-analyses: Time to Systematize the Conduct and Publication of These Study Overviews?

Published:October 29, 2015DOI:https://doi.org/10.1016/j.amjmed.2015.10.009

      Abstract

      The number of meta-analyses published annually has increased more than 20-fold between 1994 (n = 386) and 2014 (n = 8203). In examining how much of this increase in meta-analysis publication has genuinely represented novel contributions to clinical medicine and public health, it became clear that there was an abundance of redundant and disorganized meta-analyses, creating confusion and generating considerable debate. Ironically, meta-analyses, which should prevent redundant research, have become a victim of it. Recently, 17 meta-analyses were published based on the results of only 3 randomized controlled trials that studied the role of transcatheter closure of patent foramen ovale for prevention of cryptogenic stroke. In our search of the published literature, we identified at least 10 topics that were the subject of 10 meta-analyses. In the context of overlapping meta-analyses, one questions what needs to be done to put this “runaway train” back on track. In this review we examine the practice of redundant meta-analyses and the reasons for its disturbing “popularity.” The registration of systematic reviews should be mandatory in prospective registries, such as PROSPERO, and the PRISMA checklist should be updated to incorporate new evidence and mandate the reference of previously published reviews and rationale for any new study.

      Keywords

      Clinical Significance
      • There is an alarming increase in the publication of redundant systematic reviews, which are often poorly conducted.
      • The registration of systematic reviews should be mandatory in prospective registries such as PROSPERO.
      • The PRISMA checklist, which has not been updated since 2009, should be updated to incorporate new evidence.
      • The PRISMA checklist should require reconciling the conclusions of current and prior reviews, along with a table reconciling the studies included.
      Systematic reviews and meta-analyses are considered to be at a very high level in the hierarchy of evidence-based medicine. Since their initial publication in 1979, meta-analyses and systematic reviews have become a very popular type of research study.
      • Starr M.
      • Chalmers I.
      • Clarke M.
      • Oxman A.D.
      The origins, evolution, and future of The Cochrane Database of Systematic Reviews.
      Hence, the number of publications using this analytic tool has increased dramatically over the past several decades. As an example, the number of meta-analyses published on topics related to cardiology has increased dramatically, by 1800% between 1993 and 2012.
      • Osnabrugge R.L.
      • Capodanno D.
      • Cummins P.
      • Kappetein P.
      • Serruys P.W.
      Review and recommendations on the current practice of meta-analyses: a guide to appraise the evidence.
      The evidence from meta-analyses is frequently incorporated in guidelines and clinical decision making.
      • Davidoff F.
      • Haynes B.
      • Sackett D.
      • Smith R.
      Evidence based medicine.
      Systematic reviews and meta-analyses increase statistical power and the ability to discover potentially meaningful findings when individual studies are combined. They also help to identify gaps and methodological flaws in the existing medical and public health literature, assist in identifying possible sources of heterogeneity between investigations, and generate important future research directions.
      • Egger M.
      • Smith G.D.
      Meta-analysis. Potentials and promise.
      Systematic reviews and meta-analyses are also useful in avoiding redundant research. For example, a systematic review of 64 published trials revealed that the role of aprotinin in reducing the risk of bleeding in patients undergoing cardiac surgery could have been established after the 12th published trial, thereby avoiding the extensive funding and effort devoted to the subsequent conduct of 52 additional clinical trials.
      • Fergusson D.
      • Glass K.C.
      • Hutton B.
      • Shapiro S.
      Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding?.
      However, it is unfortunate that the logarithmic increase in publications has been accompanied by redundant publications. A survey in 2010 showed that approximately two-thirds of meta-analyses are followed by a redundant meta-analysis.
      • Siontis K.C.
      • Hernandez-Boussard T.
      • Ioannidis J.P.
      Overlapping meta-analyses on the same topic: survey of published studies.
      This disturbing coincidence was also highlighted by Heras et al
      • Heras M.
      • Avanzas P.
      • Bayes-Genis A.
      • Pan M.
      • Pérez de Isla L.
      • Sanchis J.
      Unplanned redundant publication: a consequence of too many cardiovascular journals?.
      when they reported 4 very similar meta-analyses with regard to the use of simple vs complex stenting in patients with coronary bifurcation lesions.
      Redundant publications are usually thought of as research reported by the same investigator and sent to 2 or more different journals.
      • Heras M.
      • Avanzas P.
      • Bayes-Genis A.
      • Pan M.
      • Pérez de Isla L.
      • Sanchis J.
      Unplanned redundant publication: a consequence of too many cardiovascular journals?.
      Redundancy in meta-analyses is unique in the sense that neither the authors nor the editors are aware of the duplicate publications because the manuscripts are written and accepted during the same period in different journals.
      The purpose of this report was to review the published literature and identify examples of redundant and disorganized systematic reviews and analyze one example in detail. We have also provided recommendations that could help improve the conduct and publication of systematic reviews.

      Methods

      We searched Medline (via PubMed) to identify trends in the publication of systematic reviews and meta-analyses and examples of redundant systematic reviews. We performed a Medline search using the search term “meta-analysis” as the publication type to identify the number of published meta-analyses during the past 20 years. The search was performed at 2 time points, namely 1994 and 2014, for purposes of examining recent 20-year trends in the publication of this study type. We identified examples of redundant meta-analysis using a modified strategy described by Konstantinos et al.
      • Siontis K.C.
      • Hernandez-Boussard T.
      • Ioannidis J.P.
      Overlapping meta-analyses on the same topic: survey of published studies.
      As a case study, we used the redundant meta-analyses involving the use of transcatheter closure devices for patent foramen ovale vs medical therapy in patients with cryptogenic stroke.
      • Wang M.T.
      • Wang T.K.
      The tsunami of meta-analyses of patent foramen ovale closure for secondary prevention of cryptogenic stroke.
      We identified all citations on this topic by performing a literature search using Medline (PubMed and Ovid SP), Embase, Web of Science, Cochrane Library (Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials), and conference proceedings using the following search terms: patent foramen ovale, inter-atrial shunt, trans-catheter closure, and cryptogenic stroke.
      We critically examined the potential reasons to explain the redundancy and disorganization in the publication of systematic reviews. Finally, after reviewing the key literature on the topic, as well as examining the case study, we provide recommendations that can guide the future conduct and reporting of systematic reviews.

      Results

      Magnitude of Redundant Systematic Review and Meta-analysis Publications

      The results of our Medline search (via Pubmed) using the search term “meta-analysis” as the publication type showed that nearly 8000 new meta-analyses or systematic reviews are being indexed annually in Medline every year. Thus, 24 new meta-analyses are indexed on a daily basis in Medline. The number of meta-analyses published annually has increased more than 20-fold during the period 1994 (n = 386) to 2014 (n = 8203).
      We identified at least 10 topics that were the subject of a minimum of 10 meta-analyses. These included the pharmacologic treatment of fibromyalgia, use of statins for the prevention of atrial fibrillation, and administration of percutaneous coronary intervention vs coronary artery bypass graft surgery in patients with left main coronary artery disease. More than 20 meta-analyses have been published on the role of vitamin D and fracture prevention and comparisons of open vs laparoscopic surgery for patients with colorectal cancer (Table 1).
      Table 1Examples of Overlapping Systematic Reviews and Meta-analyses
      TopicNo. of Overlapping Systematic Reviews and Meta-analyses
      Comparison of laparoscopic and open surgery for colorectal cancer27
      Vitamin D supplements and fracture24
      Transcatheter patent foramen ovale closure vs medical therapy for cryptogenic stroke17
      Efficacy and safety of new oral anticoagulants in atrial fibrillation15
      Prevention of colorectal cancer13
      Pharmacological treatment in fibromyalgia12
      Percutaneous coronary intervention vs coronary artery bypass graft surgery in patients with left main coronary artery disease12
      Statins for prevention for atrial fibrillation11
      Anterior cruciate ligament reconstruction comparing hamstring autograft with bone-patellar tendon bone autograft11
      Reduced-function CYP2C19 genotype and risk of adverse clinical outcomes in clopidogrel users11
      Antiepileptics for refractory epilepsy8
      Intracoronary vs intravenous administration of abciximab8

      Case Study

      Using the example of patent foramen ovale closure for our case study, we searched the literature and found 17 meta-analyses on this topic using the published information from only 3 randomized controlled trials: CLOSURE 1, PC, and RESPECT.
      • Furlan A.J.
      • Reisman M.
      • Massaro J.
      • et al.
      Closure or medical therapy for cryptogenic stroke with patent foramen ovale.
      • Carroll J.D.
      • Saver J.L.
      • Thaler D.E.
      • et al.
      Closure of patent foramen ovale versus medical therapy after cryptogenic stroke.
      • Meier B.
      • Kalesan B.
      • Mattle H.P.
      • et al.
      Percutaneous closure of patent foramen ovale in cryptogenic embolism.
      Among these 17 meta-analyses, 13 were published over a very short duration of 6 months and were included in PubMed between October 2013 and March 2014 (Table 2). Eleven of these articles were published in cardiology journals, and 2 were published in neurology journals. The results from the 3 published randomized controlled trials on this topic were quite similar: all 3 trials failed to demonstrate the superiority of percutaneous closure over the use of medical therapy using an intention-to-treat analysis (Table 3). Unfortunately, the results and interpretation of the randomized controlled trial results in the meta-analyses, using data from the same 3 trials, were quite diverse. The authors from 9 studies concluded that there was no benefit of transcatheter patent foramen ovale closure vs medical therapy, whereas the authors from 4 studies interpreted the results in favor of transcatheter closure of patent foramen ovale.
      • Wang M.T.
      • Wang T.K.
      The tsunami of meta-analyses of patent foramen ovale closure for secondary prevention of cryptogenic stroke.
      Table 2Examples of Redundant Meta-analysis on a Single Topic: Transcatheter Patent Foramen Ovale Closure vs Medical Therapy for Cryptogenic Stroke
      AuthorsPublication TimePrimary EndpointType of AnalysisResultsSecondary OutcomesStudies IncludedType of Meta-analysis
      Kwong et al
      • Kwong J.S.
      • Lam Y.Y.
      • Yu C.M.
      Percutaneous closure of patent foramen ovale for cryptogenic stroke: a meta-analysis of randomized controlled trials.
      October 2013Stroke

      TIA

      All-cause mortality
      RandomStroke OR 0.65 (0.36-1.20)

      TIA HR 0.77 (0.45-1.32)

      Mortality OR 0.65 (0.23-1.85)
      New-onset AF

      Bleeding
      CLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Ntaios et al
      • Ntaios G.
      • Papavasileiou V.
      • Makaritsis K.
      • Michel P.
      PFO closure vs. medical therapy in cryptogenic stroke or transient ischemic attack: a systematic review and meta-analysis.
      October 2013StrokeFixedStroke OR 0.64 (0.37-1.10)TIA

      Death

      MI

      New-onset AF
      CLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Moreno et al
      • Rengifo-Moreno P.
      • Palacios I.F.
      • Junpaparp P.
      • Witzke C.F.
      • Morris D.L.
      • Romero-Corral A.
      Patent foramen ovale transcatheter closure vs. medical therapy on recurrent vascular events: a systematic review and metaanalysis of randomized controlled trials.
      November 2013Composite endpoint of death and recurrent vascular eventsRandomComposite endpoint HR 0.67 (0.44-1.00)AF

      Bleeding
      CLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Nagaraja et al
      • Nagaraja V.
      • Raval J.
      • Eslick G.D.
      • Burgess D.
      • Denniss A.R.
      Is transcatheter closure better than medical therapy for cryptogenic stroke with patent foramen ovale? A meta-analysis of randomized trials.
      November 2013Stroke

      TIA
      RandomStroke OR 0.65 (0.36-1.19)

      TIA OR 0.77 (0.41-1.43)
      AF

      Major vascular complications
      CLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Pineda et al
      • Pineda A.M.
      • Nascimento F.O.
      • Yang S.C.
      • Kirtane A.J.
      • Sommer R.J.
      • Beohar N.
      A meta-analysis of transcatheter closure of patent foramen ovale versus medical therapy for prevention of recurrent thromboembolic events in patients with cryptogenic cerebrovascular events.
      November 2013Composite of stroke and TIARandomComposite endpoint OR 0.70 (0.47 1.05)Stroke and TIA individuallyCLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Hakeem et al
      • Hakeem A.
      • Marmagkiolis K.
      • Hacioglu Y.
      • et al.
      Safety and efficacy of device closure for patent foramen ovale for secondary prevention of neurological events: comprehensive systematic review and meta-analysis of randomized controlled trials.
      November 2013Composite of stroke, death, and fatal strokeRandomComposite endpoint RR 0.70 (0.48-1.06)Adverse eventsCLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Riaz et al
      • Riaz I.B.
      • Dhoble A.
      • Mizyed A.
      • et al.
      Transcatheter patent foramen ovale closure versus medical therapy for cryptogenic stroke: a meta-analysis of randomized clinical trials.
      December 2013Composite of stroke, TIA, and deathRandomComposite endpoint HR 0.66 (0.43-1.01)AF

      Bleeding
      CLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Khan AR et al
      • Khan A.R.
      • Bin Abdulhak A.A.
      • Sheikh M.A.
      • et al.
      Device closure of patent foramen ovale versus medical therapy in cryptogenic stroke: a systematic review and meta-analysis.
      December 2013StrokeFixedHR 0.67 (0.44-1.00)CLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Salas-Gutierrez et al
      • Salas-Gutierrez I.
      • Reyes-Melo I.
      • Chiquete E.
      • Ruiz-Flores M.L.
      • Cantu-Brito C.
      Efficacy of percutaneous closure of patent foramen ovale in preventing recurrent events, compared with standard medical therapy in patients with cryptogenic ischemic stroke: systematic review and meta-analysis.
      2013Composite of stroke, TIA, and deathRandomComposite endpoint RR 0.72 (0.49-1.06)CLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Chen L et al
      • Chen L.
      • Luo S.
      • Yan L.
      • Zhao W.
      A systematic review of closure versus medical therapy for preventing recurrent stroke in patients with patent foramen ovale and cryptogenic stroke or transient ischemic attack.
      February 2014Composite of stroke and TIAFixedComposite of stroke and TIA RR 0.70 (0.47-1.04)DeathCLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Capodanno D et al
      • Capodanno D.
      • Milazzo G.
      • Vitale L.
      • et al.
      Updating the evidence on patent foramen ovale closure versus medical therapy in patients with cryptogenic stroke: a systematic review and comprehensive meta-analysis of 2,303 patients from three randomised trials and 2,231 patients from 11 observational studies.
      March 2014Stroke

      TIA
      RandomFor RCTs, stroke HR 0.62 (0.34-1.11) and TIA HR 0.77 (0.46-1.32)

      For observational studies, stroke HR 0.23 (0.11-0.49)
      New-onset AFCLOSURE I

      RESPECT

      PC TRIAL

      And 11 observational studies
      Individual studies
      Spencer FA et al
      • Spencer F.A.
      • Lopes L.C.
      • Kennedy S.A.
      • Guyatt G.
      Systematic review of percutaneous closure versus medical therapy in patients with cryptogenic stroke and patent foramen ovale.
      March 2014StrokeRandomStroke RR 0.61 (0.34-1.07)CLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Wolfrum et al
      • Wolfrum M.
      • Froehlich G.M.
      • Knapp G.
      • et al.
      Stroke prevention by percutaneous closure of patent foramen ovale: a systematic review and meta-analysis.
      March 2014StrokeRandomFor RCTs, stroke RR 0.66 (0.37-1.19)

      For observational studies, stroke RR 0.37 (0.20-0.67)
      Bleeding

      Death

      AF
      CLOSURE I

      RESPECT

      PC TRIAL

      And 11 observational
      Individual studies
      Pandit A et al
      • Pandit A.
      • Aryal M.R.
      • Pandit A.A.
      • et al.
      Amplatzer PFO occluder device may prevent recurrent stroke in patients with patent foramen ovale and cryptogenic stroke: a meta-analysis of randomised trials.
      April 2014StrokeFixedStroke HR 0.62 (0.36-1.07)CLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Pickett CA et al
      • Pickett C.A.
      • Villines T.C.
      • Ferguson M.A.
      • Hulten E.A.
      Percutaneous closure versus medical therapy alone for cryptogenic stroke patients with a patent foramen ovale: meta-analysis of randomized controlled trials.
      August 2014Composite of stroke, death, and TIARandomComposite endpoint HR 0.67 (0.44-1.01)Bleeding

      AF
      CLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Udell JA et al
      • Udell J.A.
      • Opotowsky A.R.
      • Khairy P.
      • et al.
      Patent foramen ovale closure vs medical therapy for stroke prevention: meta-analysis of randomized trials an review of heterogeneity in meta-analyses.
      October 2014Composite of stroke and TIARandomComposite endpoint RR 0.73 (0.50-1.07)AFCLOSURE I

      RESPECT

      PC TRIAL
      Individual studies
      Stortecky S et al
      • Stortecky S.
      • da Costa B.R.
      • Mattle H.P.
      • et al.
      Percutaneous closure of patent foramen ovale in patients with cryptogenic embolism: a network meta-analysis.
      January 2015StrokeRandomStroke RR 0.39 (0.17-0.84)TIA

      AF

      All-cause mortality
      CLOSURE I

      RESPECT

      PC TRIAL

      Hornung et al
      Networking
      In “Results” column, values in parentheses are 95% confidence intervals.
      AF = atrial fibrillation; HR = hazard ratio; MI = myocardial infarction; OR = odds ratio; RCT = randomized, controlled trial; RR = relative risk; TIA = transient ischemic attack.
      Table 3Study Characteristics of 3 Randomized Controlled Trials on Transcatheter Patent Foramen Ovale Closure vs Medical Therapy for Cryptogenic Stroke
      Trial NameNFollow-Up (y)Age (y)Primary EndpointEndpoint Outcomes, n (Closure vs Medical)Outcome Measure
      CLOSURE9092.046.0Composite of stroke or TIA and death23 vs 29HR 0.78 (0.45-1.35)
      PC4144.144.5Composite of death, nonfatal stroke, TIA, and peripheral embolism7 vs 11HR 0.63 (0.24-1.62)
      RESPECT9802.145.9Composite of death, nonfatal and fatal stroke9 vs 16HR 0.49 (0.22-1.11)
      Values in parentheses are 95% confidence intervals.
      HR = hazard ratio; TIA = transient ischemic attack.
      It is interesting to note how the positive results from these meta-analyses have added to the confusion about which of these therapeutic approaches to use in patients with patent foramen ovale, when none of the major clinical trials showed any clear benefit of transcatheter patent foramen ovale in their analyses. A careful review of the studies reveals that differences in results, despite pooling the data from the same 3 randomized controlled trials, were likely due to different definitions of the primary trial endpoints, utilization of different software for analysis, variability in the models used for analysis, such as fixed vs random effects, and different choices for the measures of effect, as well as different interpretation of similar results (Table 2).
      • Capodanno D.
      Reply to the letter to the editor: the case for duplicate meta-analyses of patent foramen ovale closure in patients with cryptogenic stroke: more a ripple than a tsunami.
      Moreover, the problem is more than just mere redundancy. The lack of quality among many reviews was a significant issue. For example, often when describing results there was inappropriate interpretation and account of heterogeneity, as well as a lack of subsequent analyses based on the prior analyses. Most analyses did not use the already established Grades of Recommendation, Assessment, Development and Evaluation (GRADE) approach to assess the quality of included studies. The first 16 meta-analyses did not note that there is substantial bias when the meta-analysis involves a small number of studies, and that the I2 of 0 is misleading when only 3 trials are analyzed. Thus, it is pertinent to note that 16 systematic reviews were performed before a network meta-analysis was performed that compared different devices and included data from 4 trials. The results of the networking analysis, unlike the previous publications, concluded that the Amplatzer device was superior to medical therapy in preventing strokes in patients with cryptogenic embolism.
      • Stortecky S.
      • da Costa B.R.
      • Mattle H.P.
      • et al.
      Percutaneous closure of patent foramen ovale in patients with cryptogenic embolism: a network meta-analysis.

      Discussion

      Our review demonstrates that logarithmic increases in the number of systematic reviews over the past 2 decades have accompanied a disturbing increase in redundant and duplicate systematic reviews of low quality. This is a wasted effort for investigators and peer reviewers. We believe that these findings have significant academic and clinical implications.

      Findings in Context

      The quality of published systematic reviews should be tightly regulated, especially in the context of redundant publications. The characteristics such as trial selection, primary and secondary outcome selection, adequacy of their characterization, and analytic methods should be examined carefully to identify any sources of bias or as an explanation of difference in reported results. The reported outcomes should be examined as compared with the protocols. Bolland and Grey
      • Bolland M.J.
      • Grey A.
      A case study of discordant overlapping meta-analyses: vitamin D supplements and fracture.
      critically examined 7 published meta-analyses looking at the effects of vitamin D supplementation on bone fractures and found that 4 studies did not include all the potentially eligible studies, and 2 studies included trials that clearly did not meet their eligibility criteria. A study inspected 288 systematic reviews with their available protocols and found that every fourth systematic review had at least 1 discrepant outcome in the completed review. These differences often resulted in significant differences in the observed outcomes.
      • Moher D.
      • Shamseer L.
      • Clarke M.
      • et al.
      Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P). 2015 statement.
      The inconsistencies in the analysis and reporting of systematic reviews and meta-analyses have led to the publication of critical appraisals of published systematic reviews.
      • Cook D.J.
      • Reeve B.K.
      • Guyatt G.H.
      • et al.
      Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses.
      • Poolman R.W.
      • Abouali J.A.
      • Conter H.J.
      • Bhandari M.
      Overlapping systematic reviews of anterior cruciate ligament reconstruction comparing hamstring autograft with bone-patellar tendon-bone autograft: why are they different?.
      • Thorlund K.
      • Druyts E.
      • Avina-Zubieta J.A.
      • Wu P.
      • Mills E.J.
      Why the findings of published multiple treatment comparison meta-analyses of biologic treatments for rheumatoid arthritis are different: an overview of recurrent methodological shortcomings.
      Bolland and Grey
      • Bolland M.J.
      • Grey A.
      A case study of discordant overlapping meta-analyses: vitamin D supplements and fracture.
      showed that the relative risks from 10 out of 15 individual randomized controlled trials used in meta-analyses of vitamin D supplements and bone fractures differed owing to the use of different definitions of outcomes and analytic approaches.
      Despite several issues with redundant reviews, one may argue that redundant meta-analyses can improve the reliability of a given finding by pointing out the occasional errors, such as incorrect inclusion or exclusion of a study and errors in data extraction. Although certain aspects of multiple systematic reviews on similar topics may be beneficial, this practice is a waste of time for busy investigators, reviewers, and scientific journals. Keeping in mind that most reviewers work unpaid and are taking time out from their own work to perform peer review, redundant meta-analysis makes the altruistic behavior of these peer reviewers a frustrating experience.
      • Wager E.
      Why is redundant publication a problem?.
      Redundant meta-analyses can also have an adverse effect on a journal's impact factor, because the various journals that have published similar meta-analyses will be competing for the citations on the same topic.
      • Heras M.
      • Avanzas P.
      • Bayes-Genis A.
      • Pan M.
      • Pérez de Isla L.
      • Sanchis J.
      Unplanned redundant publication: a consequence of too many cardiovascular journals?.
      When a redundant meta-analysis is published, editors might have to reject excellent-quality work from other investigators because of space constraints.
      • Wager E.
      Why is redundant publication a problem?.

      Reasons for the Redundant and Disorganized Systematic Review Publications

      Arguably, one of the root causes of redundant meta-analysis is a lack of registration of systematic reviews. An evaluation of 300 systematic reviews demonstrated that none of them had been registered into a central source.
      • Moher D.
      • Tetzlaff J.
      • Tricco A.C.
      • Sampson M.
      • Altman D.G.
      Epidemiology and reporting characteristics of systematic reviews.
      This can lead several researchers to work inadvertently on the same topic. Similarly, redundant publications can occur when journals have a long lag time between receiving a manuscript and publishing the article. This is very likely for a meta-analysis involving a small number of randomized controlled trials, which is relatively easy to undertake. Another common reason for overlap in meta-analysis is the publication of updated versions of previously published systematic reviews. Approximately 20% of new meta-analyses belong to this category.
      • Moher D.
      • Tetzlaff J.
      • Tricco A.C.
      • Sampson M.
      • Altman D.G.
      Epidemiology and reporting characteristics of systematic reviews.
      One can argue that redundant meta-analyses help to maintain the necessary updating of available scientific and clinical evidence. For example, the beneficial effects of intravenous magnesium on mortality in the early post–myocardial infarction period observed through meta-analyses of the first 10 published randomized controlled trials was attenuated when a meta-analysis that included a total of 20 randomized controlled trials was subsequently published.
      • Shrier I.
      • Boivin J.F.
      • Platt R.W.
      • et al.
      The interpretation of systematic reviews with meta-analyses: an objective or subjective process?.

      The Way Forward and Formulation of Recommendations

      Registration of Systematic Reviews and Meta-analyses

      We recommend that prospective registration of protocols for systematic reviews should be mandatory in a central registry such as PROSPERO. This will help to promote transparency, reduce bias, and avoid selective reporting biases as well as duplication of meta-analyses by maintaining a public record. Such a registry will enable authors who are planning to undertake new reviews to check whether there are any similar reviews “in the pipeline,” thereby ensuring that global research funding would be invested astutely.
      • Stewart L.
      • Moher D.
      • Shekelle P.
      Why prospective registration of systematic reviews makes sense.
      In accordance with the 2010 PRISMA statement supporting registration of systematic review protocols, PROSPERO, an international prospective registry for systematic reviews, was developed by the Centre for Reviews and Dissemination in 2011. Within 8 months of its launch, 200 protocols from 26 different countries had been registered at PROSPERO.
      • Booth A.
      • Clarke M.
      • Dooley G.
      • et al.
      The nuts and bolts of PROSPERO: an international prospective register of systematic reviews.
      PROSPERO is an open public access and user-friendly registry with no charges for registering a study protocol. It requires a nominated named contact, such as the corresponding author, to ensure that the submitted information is valid and updated when necessary. It also requires submission of a minimum dataset so that protocols can only be registered once for a systematic review. After acceptance of the protocol, a unique number is assigned. The PROSPERO records are permanent, and any potential discrepancies between the initial trial protocol and the published review can be examined. This can result in improved transparency because endpoints are prespecified and prevent reporting bias. However, PROSPERO also allows amendments to the protocols when deemed necessary by the authors. The changes are visible to the readers to ensure that bias has not been introduced due to significant changes in the trial methodology, inclusion criteria, or in the analytic approaches utilized. The complete process of registering requires approximately 30 minutes.
      • Booth A.
      • Clarke M.
      • Dooley G.
      • et al.
      The nuts and bolts of PROSPERO: an international prospective register of systematic reviews.
      There are no published studies to date, however, that have examined the barriers to registering using this seemingly simple process. The factors such as lack of awareness regarding the existence of PROSPERO and simplicity of the registration process, perception of registration as an unnecessary step because it is not mandated by journals, the fear that the idea may be stolen, and limitations associated with prespecification of the protocol are the potential barriers to the registration process. We also believe that if PROSPERO is to be promoted, it needs modifications to support living systematic reviews by requiring completion dates and supporting revisions when new trials emerge.

      Reporting of Systematic Reviews

      We suggest that the PRISMA checklist, which has not been modified since 2009, should be updated to support the registration of systematic reviews. The updates should be prompt and dynamic as a better understanding of methodology and conduct of systematic reviews emerges. The checklist should be edited to indicate the type of meta-analysis, the number of previous meta-analyses on a given topic, brief conclusions of previous meta-analyses, and how the current meta-analysis is different from previous meta-analyses in terms of study selection, statistical analyses, and conclusions.
      • Siontis K.C.
      • Hernandez-Boussard T.
      • Ioannidis J.P.
      Overlapping meta-analyses on the same topic: survey of published studies.
      • Bolland M.J.
      • Grey A.
      A case study of discordant overlapping meta-analyses: vitamin D supplements and fracture.
      We also recommend including the risk of bias and the summary of findings tables in PRISMA to endorse the GRADE recommendations. Implementation and adherence to checklist items also should be enforced by journal editors and peer reviewers.
      • Moher D.
      • Shamseer L.
      • Clarke M.
      • et al.
      Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P). 2015 statement.
      A clear reason should be documented when a meta-analysis of a small number of studies is performed. Although it is methodologically correct to perform a meta-analysis on a small number of studies, when there is one large randomized controlled trial included, a meta-analysis using a small number of randomized controlled trials is likely to be biased toward the result of the large trial, as well as be subject to redundancy. There is emerging consensus that I2 has a substantial bias when meta-analyses are performed on a small number of studies.
      • Von Hippel P.T.
      The heterogeneity statistic I2 can be biased in small meta-analyses.
      When such analyses are appropriate, the synthesis of results must include confidence intervals around I2.

      The Role of Living Systematic Reviews

      Living systematic reviews are up to date online summary evidence created from millions of published studies. They can help to make an evidence ecosystem guiding clinical decision making. Commonly, the last search date used in conventional systematic reviews is more than a year old at the time of publication, which seriously hampers the translation of new evidence into action. Not only can “living” systematic reviews help in establishing accurate, up to date evidence, they would also hopefully reduce publication of many redundant systematic reviews of low quality. Although the scientific community does not have much current experience in living systematic reviews, and many challenges still exist, living systematic reviews are an exciting prospect. The modification of PROSPERO and PRISMA to support living systematic reviews should be considered.
      • Elliott J.H.
      • Turner T.
      • Clavisi O.
      • et al.
      Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap.

      Guidelines for Updating Systematic Reviews

      Updating meta-analyses too frequently can result in bias as well as a waste of resources, as previously noted. On the other hand, failure to update meta-analyses in a timely manner can result in physicians and researchers using outdated evidence. Although several models have been proposed to standardize the process of updating meta-analyses, there is no consensus at this time on when updating should occur. Recently, Takwoingi et al
      • Takwoingi Y.
      • Hopewell S.
      • Tovey D.
      • Sutton A.J.
      A multicomponent decision tool for prioritising the updating of systematic reviews.
      developed a multicomponent decision tool for prioritizing the updating of systematic reviews and meta-analyses. This multicomponent tool consists of 3 steps: it asks whether a clinical question is either no longer relevant or has already been answered; whether there are new factors to be considered relevant to the existing review, and whether information from new studies is available. In the case of a clinical question being no longer relevant, the review is flagged with “Don't update; historical question.” Similarly, if there are no new studies available, or the available studies have limited potential to change present conclusions as judged by a statistical prediction tool, the review is flagged with “Don't update; current question considered to be updated.” We believe such a multicomponent decision tool would lead to greater transparency and decreased redundancy.

      Conclusions

      The phenomenon of “redundant” or “overlapping” meta-analyses has clear implications related to the quality of medical research and ethics. Measures such as better communication between trialists, specific quality control at the level of editors and reviewers, promoting living systematic reviews, prospective registration of protocols for systematic reviews, and updating of the PRISMA checklist to address redundancy and selective reporting bias are crucial steps that would help to reduce the publication of redundant meta-analyses. These steps will also improve the standard of reporting summary trial results of meaning to patients, their providers, and health policy advocates. A collective effort on the part of authors, peer reviewers, and editors is needed in this important area. Because meta-analyses are an essential component in guiding governmental health policies and individual patient treatment decisions, it is critical to systematize the conduct and publication of systematic reviews, to protect the integrity of arguably the most potent tool in the armamentarium of evidence-based medicine.

      References

        • Starr M.
        • Chalmers I.
        • Clarke M.
        • Oxman A.D.
        The origins, evolution, and future of The Cochrane Database of Systematic Reviews.
        Int J Technol Assess Health Care. 2009; 25: 182-195
        • Osnabrugge R.L.
        • Capodanno D.
        • Cummins P.
        • Kappetein P.
        • Serruys P.W.
        Review and recommendations on the current practice of meta-analyses: a guide to appraise the evidence.
        EuroIntervention. 2014; 9: 1013-1020
        • Davidoff F.
        • Haynes B.
        • Sackett D.
        • Smith R.
        Evidence based medicine.
        BMJ. 1995; 310: 1085-1086
        • Egger M.
        • Smith G.D.
        Meta-analysis. Potentials and promise.
        BMJ. 1997; 315: 1371-1374
        • Fergusson D.
        • Glass K.C.
        • Hutton B.
        • Shapiro S.
        Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding?.
        Clin Trials. 2005; 2: 218-229
        • Siontis K.C.
        • Hernandez-Boussard T.
        • Ioannidis J.P.
        Overlapping meta-analyses on the same topic: survey of published studies.
        BMJ. 2013; 347: f4501
        • Heras M.
        • Avanzas P.
        • Bayes-Genis A.
        • Pan M.
        • Pérez de Isla L.
        • Sanchis J.
        Unplanned redundant publication: a consequence of too many cardiovascular journals?.
        EuroIntervention. 2010; 6: 179-181
        • Wang M.T.
        • Wang T.K.
        The tsunami of meta-analyses of patent foramen ovale closure for secondary prevention of cryptogenic stroke.
        EuroIntervention. 2014; 10: 771-772
        • Furlan A.J.
        • Reisman M.
        • Massaro J.
        • et al.
        Closure or medical therapy for cryptogenic stroke with patent foramen ovale.
        N Engl J Med. 2012; 366: 991-999
        • Carroll J.D.
        • Saver J.L.
        • Thaler D.E.
        • et al.
        Closure of patent foramen ovale versus medical therapy after cryptogenic stroke.
        N Engl J Med. 2013; 368: 1092-1100
        • Meier B.
        • Kalesan B.
        • Mattle H.P.
        • et al.
        Percutaneous closure of patent foramen ovale in cryptogenic embolism.
        N Engl J Med. 2013; 368: 1083-1091
        • Kwong J.S.
        • Lam Y.Y.
        • Yu C.M.
        Percutaneous closure of patent foramen ovale for cryptogenic stroke: a meta-analysis of randomized controlled trials.
        Int J Cardiol. 2013; 168: 4132-4138
        • Ntaios G.
        • Papavasileiou V.
        • Makaritsis K.
        • Michel P.
        PFO closure vs. medical therapy in cryptogenic stroke or transient ischemic attack: a systematic review and meta-analysis.
        Int J Cardiol. 2013; 169: 101-105
        • Rengifo-Moreno P.
        • Palacios I.F.
        • Junpaparp P.
        • Witzke C.F.
        • Morris D.L.
        • Romero-Corral A.
        Patent foramen ovale transcatheter closure vs. medical therapy on recurrent vascular events: a systematic review and metaanalysis of randomized controlled trials.
        Eur Heart J. 2013; 34: 3342-3352
        • Nagaraja V.
        • Raval J.
        • Eslick G.D.
        • Burgess D.
        • Denniss A.R.
        Is transcatheter closure better than medical therapy for cryptogenic stroke with patent foramen ovale? A meta-analysis of randomized trials.
        Heart Lung Circ. 2013; 22: 903-909
        • Pineda A.M.
        • Nascimento F.O.
        • Yang S.C.
        • Kirtane A.J.
        • Sommer R.J.
        • Beohar N.
        A meta-analysis of transcatheter closure of patent foramen ovale versus medical therapy for prevention of recurrent thromboembolic events in patients with cryptogenic cerebrovascular events.
        Catheter Cardiovasc Interv. 2013; 82: 968-975
        • Hakeem A.
        • Marmagkiolis K.
        • Hacioglu Y.
        • et al.
        Safety and efficacy of device closure for patent foramen ovale for secondary prevention of neurological events: comprehensive systematic review and meta-analysis of randomized controlled trials.
        Cardiovasc Revasc Med. 2013; 14: 349-355
        • Riaz I.B.
        • Dhoble A.
        • Mizyed A.
        • et al.
        Transcatheter patent foramen ovale closure versus medical therapy for cryptogenic stroke: a meta-analysis of randomized clinical trials.
        BMC Cardiovasc Disord. 2013; 13: 116
        • Khan A.R.
        • Bin Abdulhak A.A.
        • Sheikh M.A.
        • et al.
        Device closure of patent foramen ovale versus medical therapy in cryptogenic stroke: a systematic review and meta-analysis.
        JACC Cardiovasc Interv. 2013; 6: 1316-1323
        • Salas-Gutierrez I.
        • Reyes-Melo I.
        • Chiquete E.
        • Ruiz-Flores M.L.
        • Cantu-Brito C.
        Efficacy of percutaneous closure of patent foramen ovale in preventing recurrent events, compared with standard medical therapy in patients with cryptogenic ischemic stroke: systematic review and meta-analysis.
        Rev Mex Neuroci. 2013; 14: 299-305
        • Chen L.
        • Luo S.
        • Yan L.
        • Zhao W.
        A systematic review of closure versus medical therapy for preventing recurrent stroke in patients with patent foramen ovale and cryptogenic stroke or transient ischemic attack.
        J Neurol Sci. 2014; 337: 3-7
        • Capodanno D.
        • Milazzo G.
        • Vitale L.
        • et al.
        Updating the evidence on patent foramen ovale closure versus medical therapy in patients with cryptogenic stroke: a systematic review and comprehensive meta-analysis of 2,303 patients from three randomised trials and 2,231 patients from 11 observational studies.
        EuroIntervention. 2014; 9: 1342-1349
        • Spencer F.A.
        • Lopes L.C.
        • Kennedy S.A.
        • Guyatt G.
        Systematic review of percutaneous closure versus medical therapy in patients with cryptogenic stroke and patent foramen ovale.
        BMJ Open. 2014; 4: e004282
        • Wolfrum M.
        • Froehlich G.M.
        • Knapp G.
        • et al.
        Stroke prevention by percutaneous closure of patent foramen ovale: a systematic review and meta-analysis.
        Heart. 2014; 100: 389-395
        • Pandit A.
        • Aryal M.R.
        • Pandit A.A.
        • et al.
        Amplatzer PFO occluder device may prevent recurrent stroke in patients with patent foramen ovale and cryptogenic stroke: a meta-analysis of randomised trials.
        Heart Lung Circ. 2014; 23: 303-308
        • Pickett C.A.
        • Villines T.C.
        • Ferguson M.A.
        • Hulten E.A.
        Percutaneous closure versus medical therapy alone for cryptogenic stroke patients with a patent foramen ovale: meta-analysis of randomized controlled trials.
        Tex Heart Inst J. 2014; 1: 357-367
        • Udell J.A.
        • Opotowsky A.R.
        • Khairy P.
        • et al.
        Patent foramen ovale closure vs medical therapy for stroke prevention: meta-analysis of randomized trials an review of heterogeneity in meta-analyses.
        Can J Cardiol. 2014; 30: 1216-1224
        • Stortecky S.
        • da Costa B.R.
        • Mattle H.P.
        • et al.
        Percutaneous closure of patent foramen ovale in patients with cryptogenic embolism: a network meta-analysis.
        Eur Heart J. 2015; 36: 120-128
        • Capodanno D.
        Reply to the letter to the editor: the case for duplicate meta-analyses of patent foramen ovale closure in patients with cryptogenic stroke: more a ripple than a tsunami.
        EuroIntervention. 2014; 10: 772
        • Bolland M.J.
        • Grey A.
        A case study of discordant overlapping meta-analyses: vitamin D supplements and fracture.
        PLoS One. 2014; 9: e115934
        • Moher D.
        • Shamseer L.
        • Clarke M.
        • et al.
        Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P). 2015 statement.
        Syst Rev. 2015; 4: 1
        • Cook D.J.
        • Reeve B.K.
        • Guyatt G.H.
        • et al.
        Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses.
        JAMA. 1996; 275: 308-314
        • Poolman R.W.
        • Abouali J.A.
        • Conter H.J.
        • Bhandari M.
        Overlapping systematic reviews of anterior cruciate ligament reconstruction comparing hamstring autograft with bone-patellar tendon-bone autograft: why are they different?.
        J Bone Joint Surg Am. 2007; 89: 1542-1552
        • Thorlund K.
        • Druyts E.
        • Avina-Zubieta J.A.
        • Wu P.
        • Mills E.J.
        Why the findings of published multiple treatment comparison meta-analyses of biologic treatments for rheumatoid arthritis are different: an overview of recurrent methodological shortcomings.
        Ann Rheum Dis. 2013; 72: 1524-1535
        • Wager E.
        Why is redundant publication a problem?.
        Int J Occup Environ Med. 2015; 6: 3-6
        • Moher D.
        • Tetzlaff J.
        • Tricco A.C.
        • Sampson M.
        • Altman D.G.
        Epidemiology and reporting characteristics of systematic reviews.
        PLoS Med. 2007; 4: e78
        • Shrier I.
        • Boivin J.F.
        • Platt R.W.
        • et al.
        The interpretation of systematic reviews with meta-analyses: an objective or subjective process?.
        BMC Med Inform Decis Mak. 2008; 8: 19
        • Stewart L.
        • Moher D.
        • Shekelle P.
        Why prospective registration of systematic reviews makes sense.
        Syst Rev. 2012; 1: 7
        • Booth A.
        • Clarke M.
        • Dooley G.
        • et al.
        The nuts and bolts of PROSPERO: an international prospective register of systematic reviews.
        Syst Rev. 2012; 1: 2
        • Von Hippel P.T.
        The heterogeneity statistic I2 can be biased in small meta-analyses.
        BMC Med Res Methodol. 2015; 15: 35
        • Elliott J.H.
        • Turner T.
        • Clavisi O.
        • et al.
        Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap.
        PLoS Med. 2014; 11: e1001603
        • Takwoingi Y.
        • Hopewell S.
        • Tovey D.
        • Sutton A.J.
        A multicomponent decision tool for prioritising the updating of systematic reviews.
        BMJ. 2013; 347: f7191