Systematic review and meta-analysis: Which pitfalls to avoid during this process

INTRODUCTION

In our daily life, we face with decision-making and treatment choice, which, according to our ability to choose reliable sources and evidence-based literature allows us to select the best diagnostic and therapeutic alternative for our patients.

Publications in medical literature have increased over the years, and systematic reviews (SRs) have become progressively popular in medicine. Clinicians read them as an efficient manner of keeping up-to-date with their content area; they are a useful starting point for guideline development, helping to improve clinical practice (¹, ²).

What is the matter?

This starting point, part of the uncertainty, insomuch as change is more likely to occur if collective uncertainty exists; this doubt often reflects variations in medical practice. Asking a question that has already been answered by common sense or by powerful empirical evidence is of little use unless evidence suggests that the real answer is wrong. There are unknown data or a lack of usable data for some important questions, thereby demonstrating the demand for future research (¹–³).

Systematic reviews and meta-analyses are essential tools for summarizing evidence accurately and faithfully. They might help clinicians to keep up-to-date; provide evidence for policymakers to evaluate risks, benefits, and harms of health maintenance behaviors and interventions; gather together and summarize related research for patients and their care-providers; give a starting point for clinical practice guideline development; and provide summaries of previous research for funders wishing to support new research (¹–³).

Addressing a focused clinical question in a structured and reproducible manner, using systematic and explicit methods to identify, select, and critically appraise relevant research, and collecting and analyzing data from the studies that are included in the review, attempt to collate all empirical evidence that fits pre-specified eligibility criteria to answer a specific research question. It uses explicit, systematic methods that are selected to minimize bias, thus providing reliable findings from which conclusions can be drawn and decisions made. Otherwise, meta-analysis refers to the use of statistical techniques in a systematic review to integrate the results of included studies (³).

A mistake that is usually made in practice is to confuse systematic reviews with meta-analyzes, these two concepts have many important differences: Systematic Reviews attempts to collate all empirical evidence that conforms to pre-specified eligibility criteria to answer a particular research question. It uses explicit, systematic methods that are selected with a view to minimizing bias; this includes a comprehensive, exhaustive search for primary studies on a focused clinical question, selection of studies using clear and reproducible eligibility criteria, critical appraisal of primary studies for quality, and synthesis of results according to a predetermined and explicit method; thus providing reliable findings from which conclusions can be drawn and decisions made. In the other hand, Meta-analysis (MA) use statistical techniques to integrate and summarize the results of included studies. MA can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review. It is a two-stage process. The first stage involves the calculation of a measure of the treatment effect, with its 95% confidence intervals (CI) for each study. The summary statistics that are usually used to measure treatment effect include the odds ratios (OR), relative risks (RR), risk differences (RD), Hazard Ratios (HR) and mean differences (MD). In the second stage of meta-analysis, authors calculate an overall treatment effect based on the weight of each study, and a model (random or fixed) according to the heterogeneity (³, ⁴).

Issues to point out

• Some of the mistakes usually made when conducting a study are:

• Reviews did not report key aspects of systematic review methodology, thus impairing confidence in their results and conclusions.

Even when the possibility of publication bias is assessed, there is no guarantee that systematic reviewers have assessed or interpreted it appropriately.

Unfortunately, there is considerable evidence that key information is often poorly reported in systematic reviews, thus diminishing their potential usefulness. In this sense, evidence-based practice (EBP) anticipates methodologies and processes to identify evidence of whether certain treatment or diagnosis is effective, strategies to assess the quality of studies, and mechanisms to implement it with caution, and to conduct according to scientific precepts (³–⁶).

Traditional literature reviews (nowadays called narrative reviews) have been criticized for a long time because the bibliographic search and study selection method is not standardized and explicit. The results obtained through such reviews are biased, do not exhaust all the literature available about the theme, do not have a critical appraisal of the literature, and are usually inconclusive (⁷, ⁸).

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. PRISMA focuses on the reporting of reviews evaluating randomized trials, but can also be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions. It helps authors to report a wide array of systematic reviews to assess the benefits and harms of a health care intervention. It provides further details regarding its background and development. This accompanying document explains the significance and rationale for each of the 27 checklist items (³, ⁴).

Step by step

Initially, researchers must determine what type of intervention want to carry out and the population to which it will be directed, one useful strategy is establishing the PICO question tool. It focuses on the Population, Intervention, Comparison and Outcomes. It helps readers as it provides key information about the focus of the review. Specifying the design(s) of the studies included, as shown in the examples, may also help some readers and the searching databases (⁹).

The research question must address what is important to patients and clinicians; it must contribute to the community, solving a question, giving a new point of view. There are systematic reviews which cannot include any relevant studies and are referred to as an “empty review”. They are important to scientific knowledge, but they do not usually help the clinician in the decision-making process. Perhaps, it had a highly specific PICO question and an overly stringent methodological inclusion criterion, which is important but with no relevant results for clinical settings (³, ⁴).

Secondly, write a protocol, register, and create a searching strategy. A protocol is an essential part of the review process and should include sufficient data to enable independent replication of the methods. Adherence to a pre-defined protocol is a key method with which to avoid the introduction of selection bias, as it ensures that all-important decisions have been made in advance of knowledge of the results. Even so, sometimes, it is usual to find changes to the protocol to improved quality, which is important to keep in mind since it is liable to fall into the bias (⁹).

A wide range of health-related bibliographic databases exists; three bibliographic databases are generally considered to be the most important sources for searching: Central, Medline and Embase (³, ⁹, ¹⁰). Employing more than one database is a useful tool, based on a search strategy and whose objective is to solve, support or justify the hypothesis. Searches should be motivated directly by the eligibility criteria for the review, and, significantly, all types of eligible studies are considered when planning the search (¹¹).

Further, registration of a systematic review, typically with a protocol and registration number, is not yet common but may reduce the risk of multiple reviews addressing the same question, reduce publication bias, and provide greater transparency when updating systematic reviews (⁴, ¹²–¹⁴).

Thirdly, establish a selection and information collection criteria. The knowledge of the eligibility criteria is essential in appraising the validity, applicability, and comprehensiveness of a review is based on the search strategy; which is conducted according to the standards established with the PICO tool. In this manner, all these steps must be performed for at least two investigators to reduce the possibility of eliminating the relevant report. Indeed, the authors should describe these methods, including any steps taken to reduce bias and mistakes during data collection and data extraction. Also, there must be standardized protocols for data collection, including training of study personnel; minimizing inter-observer variability when multiple individuals are gathering and entering data (⁴, ¹⁴–¹⁷).

Fourthly, assess the risk of bias and quality of the included studies. Bias refers to systematic error, meaning that multiple replications of the same study would reach the wrong answer on average. It can occur at any phase of research, including study design, data collection, data analysis and publication. There are multiple instruments or validated scales to avoid this pitfall. The most common tools are Cochrane risk of bias tool, Newcastle - Ottawa Scale (NOS), MINORS, ROBINS-I, QUADAS2 and Grading of Recommendations Assessment, Development and Evaluation (GRADE). Authors must choose according to the kind of the included study (¹⁴, ¹⁶–²⁰).

Fifthly, include all languages. Selection bias is an important pitfall in recent systematic reviews. Multiple publications only include English language; however, what Cochrane recommends is including all kinds of languages for lowering this kind of bias (²¹).

Finally, in order to critically appraise the SRs, there are useful strategies: The A Measurement Tool to Assess Systematic Reviews (AMSTAR) created in 2007, is an 11-item tool that has been developed to evaluate SR quality and determine whether the most important elements are reported. AMSTAR-2 an update to the original AMSTAR tool allows a more detailed evaluation of SRs that also includes non-randomized studies; the latter contains a questionnaire with 16 domains, through which the quality of the systematic reviews can be evaluated, including defects that may have arisen due to the misconduct of the other instrument of risk of bias in SRs. In this sense, it differs from another instrument for risk of bias in SR, the ROBIS, which is a triphasic instrument that focuses on the risk of bias introduced. ROBIS is an effective tool for assessing the risk of bias in systematic reviews, but compared to AMSTAR and AMSTAR 2 with the ROBIS tool; the last shows lower agreement and it is more difficult to use (¹⁷, ¹⁹, ²⁰, ²²–¹⁷).

There are also quicker ways to assess systematic reviews, as described by Taylor et al. (²⁸). Ten questions to easily assess systematic reviews:

1. Is the study question relevant?

2. Does the study add anything new?

3. What type of research question is being asked?

4. Was the study design appropriate for the research question?

5. Did the study methods address the most important potential sources of bias?

6. Was the study performed according to the original protocol?

7. Does the study test a stated hypothesis?

8. Were the statistical analyses performed correctly?

9. Do the data justify the conclusions?

10. Are there any conflicts of interest?

In conclusion, systematic reviews and meta-analyses have become increasingly popular in medicine, and are essential tools for summarizing evidence accurately and reliably. They help clinicians keep up-to-date, and to create elements to evaluate information in an organized and structured way, allowing the results to be reproducible. Following the standard rules for performing systematic reviews will limit the possibility of making bias and will increase the transparency and reliability of the results. Hence, understanding research bias allows readers to critically and independently review the scientific literature and avoid treatments that are suboptimal or potentially harmful.

REFERENCES

• ¹
  Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009; 62:e1-34.
• ²
  Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med. 1997; 127:380-7.
• ³
  Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007; 4:e78.
• ⁴
  Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Reprint––preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Phys Ther. 2009; 89:873-80.
• ⁵
  Akobeng AK. Understanding systematic reviews and meta-analysis. Arch Dis Child. 2005; 90:845-8.
• ⁶
  Selçuk AA. A Guide for Systematic Reviews: PRISMA. Turk Arch Otorhinolaryngol. 2019; 57:57-8.
• ⁷
  Froom P, Froom J. Deficiencies in structured medical abstracts. J Clin Epidemiol. 1993; 46:591-4.
• ⁸
  da Costa Santos CM, de Mattos Pimenta CA, Nobre MR. The PICO strategy for the research question construction and evidence search. Rev Lat Am Enfermagem. 2007; 15:508-11.
• ⁹
  Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014; 14:579.
• ¹⁰
  Pollock A, Berge E. How to do a systematic review. Int J Stroke. 2018; 13:138-56.
• ¹¹
  Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study. PLoS Med. 2016; 13:e1002028.
• ¹²
  Centre for Reviews and Dissemination. Systematic Reviews: CRD's guidance for undertaking reviews in health care. York: University of York; 2009. Available at. < https://www.york.ac.uk/crd/guidance/>
  » https://www.york.ac.uk/crd/guidance/
• ¹³
  Major MP, Major PW, Flores-Mir C. An evaluation of search and selection methods used in dental systematic reviews published in English. J Am Dent Assoc. 2006; 137:1252-7.
• ¹⁴
  Silagy CA, Middleton P, Hopewell S. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA. 2002; 287:2831-4.
• ¹⁵
  Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. An international registry of systematic-review protocols. Lancet. 2011; 377:108-9.
• ¹⁶
  Garcia-Perdomo HA Evidence synthesis and meta-analysis a practical approach. Int J Urol Nurs. 2016;10:30-6.
• ¹⁷
  Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. J Clin Epidemiol. 2005; 58:741-2.
• ¹⁸
  Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. An international registry of systematic-review protocols. Lancet. 2011; 377:108-9.
• ¹⁹
  Pannucci CJ, Wilkins EG. Identifying and avoiding bias in research. Plast Reconstr Surg. 2010; 126:619-25.
• ²⁰
  Godlee F. Milestones on the long road to knowledge. BMJ. 2007; 334(Suppl 1):s2-3.
• ²¹
  Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available at. <http://www.handbook.cochrane.org>.
  » http://www.handbook.cochrane.org
• ²²
  Herney Andrés García-Perdomo. Conceptos fundamentales de las revisiones sistemáticas/metaanálisisBasic concepts on systematic review and meta-analysis. Urología Colombiana 24, 2015, 28-34.
• ²³
  García-Perdomo HA. Enhancing the quality and transparency of systematic reviews. Colomb Med (Cali). 2018; 49:251-253.
• ²⁴
  Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017; 358:j4008.
• ²⁵
  Whiting P, Savović J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016; 69:225-34.
• ²⁶
  Pieper, D., Puljak, L., González-Lorenzo, M., & Minozzi, S. (2019). Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. Journal of clinical epidemiology, 108, 26–33.
• ²⁷
  Banzi R, Cinquini M, Gonzalez-Lorenzo M, Pecoraro V, Capobussi M, Minozzi S. Quality assessment versus risk of bias in systematic reviews: AMSTAR and ROBIS had similar reliability but differed in their construct and applicability. J Clin Epidemiol. 2018; 99:24-32.
• ²⁸
  Taylor P, Hussain JA, Gadoud A. How to appraise a systematic review. Br J Hosp Med (Lond). 2013; 74:331-4.

Publication Dates

• Publication in this collection
  30 July 2021
• Date of issue
  Sep-Oct 2021

History

• Received
  19 Aug 2020
• Accepted
  20 Aug 2020
• Published
  20 Dec 2020