How to avoid bias in systematic reviews of observational studies

ABSTRACT

Although systematic reviews have numerous advantages, they are vulnerable to biases that can mask the true results of the study and therefore should be interpreted with caution. This article aims at critically reviewing the literature about systematic reviews of observational studies, emphasizing the errors that can affect this type of study design and possible strategies to overcome these errors. This is an integrative literature review whose search was conducted in the databases States National Library of Medicine, Scientific Electronic Library Online and Google Scholar. The following descriptors were used: review, bias (epidemiology) and observational studies as the subject, including relevant books and documents which were consulted. Data collection was conducted between June and July 2016. The most known errors present in the design of systematic reviews were those related to the selection and publication. Although this type of study is subject to possible errors, preventive measures used during the planning of systematic reviews and even during and after their implementation can help ensure scientific rigor. This literature can serve as an important tool for the development and interpretation of systematic reviews of observational studies.

Keywords:
Review; Bias (Epidemiology); Observational Studies as Topic

RESUMO

Embora as revisões sistemáticas tenham inúmeras vantagens, eles são vulneráveis a vieses que podem mascarar os verdadeiros resultados do estudo e, portanto, devem ser interpretados com cautela. A proposta deste artigo é revisar criticamente a literatura acerca das revisões sistemáticas de estudos observacionais, enfatizando os erros que podem afetar este tipo de desenho de estudo e as possíveis estratégias para superar esses erros. Trata-se de uma revisão integrativa da literatura. A pesquisa bibliográfica foi conduzida nas bases de dados: States National Library of Medicine, Scientific Electronic Library Online, e Google Scholar. Para a busca em bases de dados foram utilizados os descritores: revisão, viés (epidemiologia) e estudos observacionais como assunto. Também foram consultados livros e documentos com relevância para o tema. A coleta dos dados foi realizada entre junho e julho de 2016. Os erros mais conhecidos, presentes no delineamento de revisões sistemáticas, foram os relacionados com a seleção e publicação. Embora este tipo de estudo esteja sujeito a possíveis erros, as medidas preventivas postas em prática durante o planejamento de revisões sistemáticas e até mesmo durante e após a sua execução podem ajudar a garantir o rigor científico. Esta revisão da literatura pode servir como um instrumento importante para o desenvolvimento e interpretação de revisões sistemáticas de estudos observacionais.

Descritores:
Revisão; Viés (Epidemiologia); Estudos Observacionais como Assunto

Introduction

The aim of epidemiological research is to build knowledge and obtain valid and accurate results. Deep reflection on options and planning in all methodologic phases are required when conducting epidemiological studies. To maintain the necessary methodological rigor, definition of the study project is a crucial step. Systematic review design has gained popularity in the public health sector, however, as any study design, this is susceptible to certain biases that must be rigorously planned and/or accounted for by researchers.

Literature on a specific subject is used as a source of data in a systematic review. This type of investigation provides a synthesis of evidences related to a specific intervention strategy, through the use of explicit and systematized search methods, critical analysis and summary of the selected information. Systematic reviews are particularly useful for integrating information from a study group to answer a broader research question than is possible from an individual study, which may present conflicting and/or overlapping results, as well as identify issues in need of evidence, assisting in future research¹. By combining the evidence from several studies, it is important to consider the risk of bias in each study. If study methods are not adequate^2,3, results from observational studies may be biased.

Complete systematic reviews are time consuming, often lasting up to three years and require two reviewers to evaluate each article to reduce the number of errors. One study estimated that 80% of risk assessments of biases took between 10 and 60 minutes to be conducted^4,5. In addition, risk judgments of biases are sometimes imperfect. Studies have shown that reviewers often report different levels of the risk of bias for the same studies^4,6-10. This can happen, for instance, if a reviewer misses "key" phrases throughout the text¹⁰.

Among the advantages of making a systematic review are the likelihood of assessing the consistency and generalization of results among populations or groups, as well as specificities and variations of treatment protocols and production of subsidies for the formulation¹¹ of health policy.

There is no consistency in the words used to describe systematic reviews, considering that some include a statistical synthesis of the results of the studies and others do not. Authors point out that systematic reviews with meta-analysis are different from other reviews due to their meta-analytic component¹¹. Meta-analysis is a literature review study in which the results of several independent studies are grouped and synthesized by means of statistical procedures (analysis of the analysis), in order to produce a single estimate or index that characterizes the effect of a given predictor/intervention^12,13.

Systematic reviews have historically been developed to systematize the findings of intervention studies and, more recently, have been applied to observational studies, which are applicable in many cases related to human health where these designs are the most appropriate ones. However, in choosing a study design, both risks and benefits must be considered. Although systematic reviews have many advantages, they are also more susceptible to certain types of biases, especially in reviews of observational studies.

The aim of this study was to review the literature to critically evaluate the factors that affect this type of study design, as well as possible methods to overcome or minimize these distortions to help the researchers who choose this type of epidemiological study design.

Methods

To achieve the proposed goal, bibliographic research was conducted at the States National Library of Medicine (PubMed), Scientific Electronic Library Online (SciELO), and Google Scholar. For the search in databases the descriptors used were: review, systematic review, bias (epidemiology) and observational studies as topic. Books and documents relevant to the topic were also consulted. These were included because it is believed that books and other documents are important and current production of knowledge on the subject in focus.

Criteria for inclusion were: complete publications available online, in Portuguese, English and Spanish, which addressed bias analyzes in systematic reviews and without a limit of publication date. Criteria for exclusion were: duplicate publications and/or strictly focused on randomized clinical trials. Data collection was carried out between June 2016 and June 2017. All the works analyzed were quoted together with their respective authors, thus respecting the ethical principles in research.

Literature Review

The field of epidemiology has sought to understand the factors that can interfere in testing hypotheses, including errors that may mask the actual results of a study. Before presenting commonly found errors in systematic reviews, it is necessary to have a basic understanding of the concepts and types of errors known as random and systematic errors.

A random error occurs when the value measured in the sample diverges, randomly, from the actual value of the population. Random error may occur in any study design, and is not specific to review studies. Random error can be reduced using a larger sample, and this type of error can be estimated through statistical tests. The second type of error, known as systematic error or bias, is defined as any process, at any stage of the research, in which the result is distorted using incorrect methodology during the course of the investigation.

Bias includes any and all distortions during the investigation process, which can occur in any type of design. However, there are certain types of bias to which systematic reviews are susceptible to. The types of bias can be classified in the following categories: selection bias, information bias, and confounding bias. These will be explored in this paper in the context of the systematic review of observational studies along with the possible strategies that researchers can use to overcome them.

A systematic review consists of primary studies, published and unpublished, meeting the criteria of inclusion and exclusion determined a priori and must be representative of all the results of the primary studies available at the time the review was conducted. However, if only published studies with language restriction (e.g., English) are used from certain electronic databases, as is the case with many published reviews, a large number of eligible studies may be missed, resulting in a substantial bias on selection¹⁴.

An inadequate or non-comprehensive search strategy that includes only full-text publications are other common errors that may reduce the representativeness of identified or included studies. Comprehensive research should include "grey literature"¹⁵. Grey literature may include search in annals of scientific events pertinent to the researched topic, search in bases of international theses and dissertations, OAIster, Google, openDOAR, OpenGrey, among others.

On the other hand, self-selection bias may arise when researchers choose not to publish some of their primary studies for various reasons, null, unexpected, or unexplained findings. In addition, journal editors and reviewers are more likely to accept studies with statistically significant results, especially if this happens even when the size of the sample is relatively small¹⁴.

The data selected to compose the meta-analysis may be influenced by publication bias. Publication bias is understood as the tendency to publish results that are systematically different from reality. Failure to publish results may be due to the decision of the author or funder of the study, who does not submit for publication unfavorable findings, or from editors of scientific journals, who may not be interested in disclosing negative (with no statistical relevance) findings¹⁶.

In systematic reviews, the presence of this bias can be identified through funnel chart and statistical tests². The use of these techniques is recommended for meta-analysis with ten or more studies and is based on estimation and precision questions. Inaccurate studies, usually performed with small samples, by chance, can find positive or negative results (statistically significant or not). They would be distributed symmetrically in the widest part of the funnel. Higher precision studies, generally smaller in number, would be closer to the real value and located in the narrower part of the funnel. It is also possible to estimate the presence of this bias in the bases of study registries while searching for those studies not published¹⁶.

Often, authors of systematic reviews mention contacting authors and researchers in the related field to obtain information on studies that were not published or were not included in the databases used in the research. However, the response rate and other details related to these questions are rarely reported. The low response rate to such queries may also increase the bias of self selection¹⁴.

At present, the use of tools such as Researchgate has contributed to contact researchers around the world, so that in our experience, the attempts of contact with research groups and researchers have been more effective in the search of published data and more details on articles that we seek to include in the systematic reviews we conducted.

The process of retrieving studies information for inclusion in systematic reviews is usually more problematic than in case series, such as data to be collected or extracted from primary studies, as they are more prone to misclassification than demographic and clinical information of patients in case series. Exposure/intervention, results and potential confounding factors cannot be defined or categorized in the same way in all primary studies, and errors in classification can result of their recategorization with the objective of grouping the findings¹⁴.

In systematic review data extraction process it is common, in order to minimize such errors, to have two reviewers independently gathering information from primary studies, and resolving disagreements with a third reviewer or by consensus. In addition, since a systematic review of observational studies uses information presented in primary studies, any measure taken or errors reported in later studies may invalidate the findings of the review. In addition, assessing the quality of primary studies to be included in a systematic review is of the utmost importance. Only studies with reasonably valid association results should be included^17-20.

It is essential to ensure that the studies included in the systematic review have comparable methodological quality and the necessary information to allow the identification of selection, gauging and confounding biases were avoided. In addition, the basic data to identify whether the populations surveyed in the studies are comparable is also critical.

Confounding is not a usual bias for qualitative systematic reviews, as group comparisons in primary studies should be very similar in terms of prognostic factors²¹. The adoption of stricter inclusion/exclusion criteria to improve homogeneity among primary studies may help, but at the expense of external generalization and applicability. The issue of non-comparability is further complicated by meta-analysis that include observational studies or non-randomized trials in which adjusted confounding factors may be very different in terms of numbers, definitions and categorizations, in different primary studies. A narrative synthesis with deep critical evaluations of individual primary studies may be more informative¹⁴.

Conclusion

Systematic reviews are studies that like any other design have advantages and disadvantages. The present manuscript discusses the main disadvantages reported in the literature and the main biases that commonly occur in this type of design. It should be noted that the most widely recognized biases in systematic reviews of observational studies are those related to selection and publication bias. Although the design of this study has the possibility of errors, there are strategies that can be used in planning the study, as well as during and even after its execution to ensure scientific rigor, especially since systematic reviews have a recognized scientific importance.

References

• ¹
  Linde K, Willich SN. How objective are systematic reviews? Differences between reviews on complementary medicine. J R Soc Med. 2003;96(1):17-22.
• ²
  Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: Wiley. 2008.
• ³
  Schulz KF, Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet. 2002;359(9306):614-8.
• ⁴
  Millard LA, Flach PA, Higgins JP. Machine learning to assist risk-of-bias assessments in systematic reviews. Int J Epidemiol. 2016;45(1):266-77.
• ⁵
  Savovic J, Weeks L, Sterne JA, Turner L, Altman DG, Moher D et al. Evaluation of the Cochrane Collaboration's tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation. Syst Rev. 2014;3:37.
• ⁶
  Armijo-Olivo S, Ospina M, da Costa BR, Egger M, Saltaji H, Fuentes J et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One. 2014;9(5):e96920.
• ⁷
  Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs SJ et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012.
• ⁸
  Hartling L, Bond K, Vandermeer B, Seida J, Dryden DM, Rowe BH. Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma. PLoS One. 2011;6(2):e17242.
• ⁹
  Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013;66(9):973-81.
• ¹⁰
  Lensen S, Farquhar C, Jordon V. Risk of bias: are judgements consistent between reviews? Cochrane Database Syst Rev Suppl. 2014;1:30.
• ¹¹
  Akobeng AK. Understanding randomised controlled trials. Arch Dis Child. 2005;90(8):840-4.
• ¹²
  Bradley L, Law M. Systematically reviewing the evidence. In: Law M, McDermid J. Evidence-based rehabilitation: a guide to practice. 3.ed. Thorofare (NJ): SLACK Inc. 2002. p. 143-59.
• ¹³
  Sampaio RF, Mancini MC. Estudos de Revisão Sistemática: Um Guia para Síntese Criteriosa da Evidência Científica. Rev.bras.fisioter. 2007;11(1):83-9.
• ¹⁴
  Yu IT, Tse SL. Clinical Epidemiology Workshop 11 - Sources of bias in systematic reviews with or without meta-analysis. Hong Kong Med J. 2013;19(2):156-8.
• ¹⁵
  Fourth International Conference on Grey Literature: New Frontiers in Grey Literature. GreyNet, Grey Literature Network Service, Washington DC, USA. 1999. p.4-5.
• ¹⁶
  Pereira GP, Galvão TF. Heterogeneidade e viés de publicação em revisões sistemáticas. Epidemiol.Serv.Saúde. 2014;23(2):775-8.
• ¹⁷
  Yu IT, Tse SL. Clinical Epidemiology Workshop 7 - Appraising a study on diagnosis. Hong Kong Med J. 2012;18:325-6.
• ¹⁸
  Yu IT, Tse SL. Clinical Epidemiology Workshop 8-Appraising a study on therapy. Hong Kong Med. 2012;18:19-20.
• ¹⁹
  Yu IT, Tse SL. Clinical Epidemiology Workshop 9 - Appraising a study on prognosis. Hong Kong Med J. 2012;18:507-8.
• ²⁰
  Yu IT, Tse SL. Clinical Epidemiology Workshop 10-Appraising a study on risk factors or aetiology. Hong Kong Med J. 2013;19:50-1.
• ²¹
  Yu IT, Tse SL. Clinical Epidemiology Workshop 3-Sources of bias in case series, patient cohorts, and randomised controlled. Hong Kong Med J. 2011;17:478-9.

• Sources of funding: CAPES - Bolsa de estudo para cursar doutorado. CNPq - produtividade em pesquisa.

Publication Dates

• Publication in this collection
  Aug 2017

History

• Received
  25 Jan 2017
• Accepted
  21 June 2017