Systematic Review and its Relationship with Evidence-Based Practice in Health

Abstracts

As revisões sistemáticas (RS) possuem relevância no mundo e na América Latina pela sua credibilidade na busca, coleta, ordenação e análise das pesquisas sobre as intervenções de saúde, que têm sido realizadas num período de tempo. É assim que a prática baseada na evidência tem usado as RS como forma de obter as melhores evidências da efetividade clínica. Neste artigo, revisa-se a metodologia, processo e utilidade da RS e a implicância que tem nas profissões da saúde, como a enfermagem e medicina.

Pesquisa; Revisão; Metanálise

Las revisiones sistemáticas (RS) poseen relevancia en el mundo y en Latinoamérica por su credibilidad en la búsqueda, recolección, ordenamiento y análisis de las investigaciones sobre intervenciones de salud que se han realizado en un periodo de tiempo. Es así que la práctica basada en la evidencia ha usado las RS como la forma de obtener las mejores evidencias de la efectividad clínica. En este artículo se revisa la metodología, proceso y utilidad de la RS y las implicancias que tiene en las profesiones de la salud como enfermería y medicina.

Investigación; Revisión; Metaanálisis

THEORETICAL ARTICLE

Systematic Review and its Relationship with Evidence-Based Practice in Health

Eugenia Urra Medina^I; René Mauricio Barría Pailaquilén^II

^IRN, Ph.D. in Nursing, Assistnt Professor, Universidad de la Serena, Coquimbo, Chile. E-mail: [e-mail]

^IIRN, M.Sc. in Clinical Epidemiology, Researcher, Instituto de Enfermería, Facultad de Medicina, Universidad Austral de Chile, Valdivia, Chile. E-mail: [e-mail]

^{Corresponding Author}

ABSTRACT

Systematic reviews (SR) have gained relevance in the world and Latin America because of their credibility in the search, compilation, arranging and analysis of the information obtained from research about health interventions, during a period of time. Consequently, evidence-based practice uses SR as a way to capture the best evidence of clinical effectiveness. This article reviews SR methodology, process, and its usefulness in health professions like nursing and medicine.

Descriptors: Research; Review; Meta-Analysis.

Introduction

The continuous growth of knowledge developed through research and the current demands for evidence-based practice (EBP) have given rise to the need to collect, analyze and summarize knowledge about previous research. In this sense, several methods can be used to review a research, depending on the focus of interest in the collection and assessment of this research knowledge. Existing research review methods can be classified as: integrated review, systematic review (SR), meta-analysis and methods combining qualitative research (meta-summary, meta-synthesis, meta-studies and grounded theory for example)⁽¹⁾.

Systematic reviews mainly focus on quantitative studies and are used as a summary to produce a meta-analysis about the collection of the best possible research evidence to develop EBP. They are also useful to recollect knowledge about existing studies and research methods used in quantitative or qualitative primary studies. SRs represent an adequate proposal to obtain evidence and synthesize knowledge on a health theme, as they stimulate convergence and inclusion of all relevant studies, providing a comprehensive look on the focus area.

This article describes the SR, its steps and relevance for EBP, in order to reflect on the range of elements involved and the strictness that should be applied in a review. The SR permits synthesizing knowledge in a study area with a view to its application in practice. Therefore, it should be stimulated in our Latin American region.

What is a systematic review?

The SR is a process developed to identify the core of a literature review of interest to practice, performing the search and extracting what is most relevant according to criteria audited and respected by others⁽²⁾. Specifically, it is a research in itself, with previously planned methods and assembling original studies, considered as the research subjects. SRs synthesize the results of multiple primary studies, using strategies to reduce bias and random errors. These strategies include the exhaustive search of all potentially relevant articles and explicit and replicable criteria in the selection of articles for the review. Thus, the designs and characteristics of primary studies are evaluated, data are synthesized and results interpreted⁽³⁾. SR is defined as the "application of scientific strategies that limit the bias of systematic assembly, critical assessment and synthesis of all relevant studies on a specific topic"⁽⁴⁾, and as "a scientific tool that can be used to summarize, extract and communicate the results and implications of a number of studies that could not be administered in any other way"⁽²⁾. In the accomplishment of this process, scientific rigor is such that the SR is considered a study on its own merit, capable of summarizing evidence on a specific clinical problem, at the center of interests of EBP initiatives⁽¹⁾, and accepted as a reliable source of information about the efficacy of health care. Its goals are: to reveal knowledge based on existing research about a defined topic, so as to collect the best research evidence possible for the development of EBP; to combine previous research results as if it were a scientific study by itself and to identify knowledge gaps. The benefits of summarizing available evidence on one theme is that it favors a minimum of errors and bias, which can sometimes interfere in an appropriate literature review or selection. Thus, the SR is characterized as a clearly established system of objectives, with predefined eligibility criteria for the studies; an explicit and reproducible method; a systematic search that tries to identify all studies that would comply with the eligibility criteria; an assessment of the result validity of the included studies (through the evaluation of bias for example) and a systematic presentation and synthesis of the characteristics and results of the included studies⁽⁵⁾.

In comparison, a traditional literature review considers a broad topic and not necessarily specific literature sources. Although the selected documents can be summarized and criticized, this is not accomplished in a sophisticated or strict form. The research can later be summarized in a narrative, as part of a qualitative style. That is a source of bias, such as a reviewer's predisposition or prejudice by including articles that support his view and excluding those that do not. This might happen unconsciously, but prevents the reader from getting an objective view of research about a theme, thus distorting the truth⁽⁶⁾. Consequently, traditional narrative reviews are useful as a general look on a broad theme, but should not be considered solid evidence for the efficacy of specific interventions, as required in EBP. In summary, traditional reviews have been fundamentally criticized because they lack explicit method, strict definitions and/or standardized techniques. In response, different review methods have been developed to offer evidence, including meta-analysis. Since its appearance in 1976⁽⁷⁾ and across more than three decades, a progressive evolution in review methods has occurred, originating in different subject areas, reflecting in a range of documents and manuals for SR. In Latin America, however, limited progress has been observed, basically due to health professionals' lack of knowledge and preparation regarding these methods. Since the start, SRs have concentrated on trials, preferably randomized clinical trials (RCT), thus focusing on the efficacy of interventions. RCTs are a special class of trials in which the efficacy of therapeutic interventions in patients is investigated⁽²⁾, as they offer less susceptibility to bias, considering that their design favors effects that are considered more reliable⁽⁸⁾. The first RCT in medicine was reported in the 1940's. Today, more than 100,000 have been published, and many have not been published, turning into an attractive field for development and review⁽⁹⁾. On the opposite, descriptive, observational and interpretative studies have not found room in SRs, which represents a limiting factor for nursing – as well as for other health subjects, as many care-related questions are determined by the impact of the disease or treatments, and have not been answered uprightly by interventions studies or trials. Therefore, SRs should expand their focus to summarize studies with other designs, including qualitative studies, guaranteeing the maintenance of the primary researcher's rigor. The clinical community is reluctant, however, which is why the substrate of SRs still predominantly comprises RCTs⁽⁷⁾.

The Cochrane Collaboration is the main international organization concentrating SRs. This non-profit entity was created in 1993 to increase the quality of health care decisions by "preparing, maintaining and promoting the accessibility of SRs about the effects of health interventions"⁽¹⁰⁾. It was inspired by the preliminary ideas of Archie Cochrane, a Scottish epidemiologist who defended that one of the most significant advances in medicine would be the notion that health care services should be sustained by scientific evidence more than by clinical impression, anecdotic experience, expert opinions or tradition⁽¹¹⁾.

Nowadays, the Cochrane Collaboration comprises review groups, centers, method groups and a consumer network. Canada, for example, had 308 reviewers associated with national and international groups and, in 2005, there were 9000 contributors in 90 countries⁽¹²⁾. Nevertheless, the Cochrane Library has been criticized for a lack of accessibility, flexibility and relevance⁽¹³⁾. The Cochrane reviews are disseminated through CD-ROM and Internet subscriptions on the website www.cochranelibrary.com, with free access to abstracts. The Cochrane Library is a the main source of SRs and includes 5 databases: database of systematic reviews (CRS), database of reviews of effects (DARE), central register of controlled trials (CREC), database of reviews of methods and the Cochrane methodology register⁽¹⁰⁾.

Steps in the systematic review

The steps in the SR involve defining its purpose, formulating the question, searching literature, assessing the data, analyzing the data and presenting the results^(6-7,14).

Purpose of the review and formulation of the question

The SR declares the purpose of the review and the problem or question on the selected topic. The formulation of the question starts the SR and is established as a known and delimited question that is accessible and identifiable in literature⁽¹⁵⁾. A good clinical question should include the type of patients investigated, the intervention under analysis, comparative interventions and results of interest used to measure the effects. These components direct the subsequent steps of the review process. The inclusion of these four elements gives focus to the selection process. It is useful to define detailed criteria in a selection process, specifying study designs, form and type of of publications to be included⁽¹⁵⁾. After defining the question, a preliminary search is performed to detect whether an SR has already been done on the determined topic.

Literature Search

Definition of criteria to select the studies

After clarifying the elements that appropriately reflect the research questions, the reviewers should detail the criteria to select the research designs that will be included. Initially, designs should be chosen that generate valid results and are available to answer the questions. The search can contain criteria to limit the period, language and type of article. Ideally, articles should be considered from peer-reviewed journals, as well as studies published as abstracts, without peer review. Thus, checklists should be prepared, detailing the selection criteria (Figure 1). These forms can be pre-tested, that is, tested by other reviewers for bias or application problems.

Finally, a selection form should be elaborated as part of a broader protocol for the total review, helping reviewers in two ways: it provides a document that explicitly states the review questions and selection criterion, and establishes a selection protocol that identifies what will be done, by whom, how, when and why. Thus, the protocol provides a communication channel for the review team. As the search is a critical step in the SR, expert searchers are needed to avoid errors which, in this phase, could lead to bias or result in an incomplete evidence base⁽¹⁶⁾.

Accomplishment of the literature search

After detailing the criteria and elaborating the forms, it should be checked in the SR databases and other electronic bases whether reviews have been published on the selected topic. This step is very time-consuming, with descriptions, for example, of a search for an SR on pharmacological pain treatment that took more than 90 days⁽¹⁷⁾. This is vital as, if the search is poor or does not include adequate database, the SR process will be impaired and might lead to false conclusions. Thus, to avoid bias, it is fundamental for the literature search to be complete and identify all studies on the selected topic, either published or not, wherever possible⁽⁶⁾.

Essential elements of the search include: databases, journals related to the selected theme and unpublished literature (dissertations or unpublished research). As the purpose of SRs in this part is to determine the extent of the literature review, the researcher should consider not only published high-quality studies, but also lower quality and unpublished research. Different methods are used in the search: first, one can search published studies in relevant databases on the topic of interest. The literature search is usually additional to the manual search of journals and registered studies specified in the electronic search. The manual literature search can also include academic dissertations, books and abstracts. Next, a search should be carried out in the so-called grey literature, which corresponds to unpublished studies that might contain important research evidence. This includes conferences, discussion papers, procedures, theses and pharmaceutical aspects (drugs for example). Finally, the literature search looks at the Internet or contacts with other researchers. The search strategy should be carefully documented as a fundamental part of the review process. This documentation permits the replication of the search by other researchers if necessary. Databases for review include MEDLINE, Cumulative Nursing Index, DARE, CINAHL, EMBASE, among others, as other databases are available that might be related to the research question. It is estimated that half of all relevant articles can be found through electronic bases, so that it is recommended to review print journals related to the topic. Moreover, it should be taken into account that the most recent articles may not have been indexed yet in the electronic bases. For the manual search, the researchers identify libraries with complete journal collections. In summary, this step involves the detection of documents, through the use of databases and direct manual review, based on terms and search limits.

Data assessment

After detecting sufficient literature, the availability of this material is verified. In this step, criteria are set to define what studies will be assessed because of their relevance for the research question. An independent review by more than one reviewer is recommended for each article, with detailled records of rejections and the reasons, which should be specified in the full study. Document reviewers should be "blind" (external peers of the author and sources) to reduce the reviewer's bias. The study assessment aims for three goals: to assess study validity, discover the reasons for difference among study results and offer readers sufficient information to judge the applicability of the SR. Although the SR selection criteria define the population, interventions and results of interest, the assessment phase comprises an exhaustive assessment of patients in each study, interventions that were compared and results measured. Differences in these characteristics can be an important source of variation among study findings⁽¹⁵⁾.

The power of evidence taken from primary studies often serves as an inclusion criterion in an SR conducted to develop EBP, as it centers on detecting studies with the highest evidence level, leading to an exhaustive examination of methodological quality. Quality refers to the study design's ability, accomplishment and analysis to minimize potential bias. Therefore, qualitative studies are excluded from the SRs, mainly because evidence from RCTs is considered more reliable than qualitative evidence. Nevertheless, the contribution of qualitative studies should be acknowledged, offering information on non-quantifiable aspects of impression and experiences regarding health or disease phenomena. After defining what studies will be included in the review, the research data can be collected. From each study, information should be taken about the patient group, treatments, place, design, interventions, result measurement and follow-up. At least two people should independently extract data in order to minimize transcription errors.

Data analysis and synthesis

In an SR, selected material is carefully read and primary study results are condensed for the sake of additional analysis. This analysis involves organizing, categorizing and combining data from primary studies to respond to the problems or questions. Reviewers should perform this analysis independently. Statistical or qualitative analysis can be used, depending on the purpose and collected material. The most used method in SRs that study summarized quantitative data is meta-analysis. This corresponds to the statistical analysis of results from individual studies with a view to integrating findings into one more simple and generalizable finding⁽¹⁸⁾. Hence, it is defined as an SR that uses statistical methods to combine results from two or more primary studies to obtain more precise, powerful and convincing conclusions⁽³⁾. Hence, meta-analysis can be used to combine the results of small studies, individually lacking statistical power, but which produce conclusive results when combined.

Meta-analyses are graphically represented, demonstrating the effect of interventions by combining the study participants from different studies. Figure 2 shows the structure of a meta-analysis. The central vertical line indicates absence of effect (relative risk = 1,0) indicating no inter-group in terms of the effect of the studied interventions. The horizontal line, in turn, marks relative risk levels. Levels towards the left of the vertical axis indicate that the findings favor the control intervention, while levels to the right favor the treatment under analysis. For each study, the central point represents the punctual estimate of the effect and length of the line at both sides, with a 95% confidence interval. The latter corresponds to the range of values inside with it is assumed that the true value of the effect is found with 95% certainty, permitting the determination of sense and precision. Shorter confidence interval lines indicate greater result precision, improving validity. In this graph, the result of each study is specified, followed by the detail of the meta-analysis (summary measure), which allows the reader to easily assess the clinical effects of the treatment being reviewed. Data from different studies can only be combined in the meta-analysis if they have similar characteristics (types of participants, interventions and results measured). In meta-analysis, it is essential to investigate statistical differences in the study results. Variability in the estimated results can be assessed, observing homogeneity when the sense of the confidence interval lines coincide and heterogeneity when they coincide hardly or not.

The most important conclusion of a meta-analysis is the quantitative summary of outcomes, for example, expressed as the odds ratio (OR) and its 95% confidence interval⁽¹⁹⁾, as a relative risk or specifying the effect of treatments throught the number needed to treat (NNT). The latter is an association measure of the probable clinical impact and indicates how many patients need to be treated with the intervention to obtain a positive result (or avoid the negative outcome). Results can also indicate the possibility of damage related to the intervention, represented by the number needed for damage or an adverse effect (NND). As observed, implications vary, but some plead for a focus that filters the information using the NNT, which refers to the result of the specific treatment and describes the difference between the treatment and control groups in terms of a specific clinical outcome⁽²⁰⁾.

When the reviews compile results of different studies but without a global statistical analysis, they are called qualitative SRs and their analysis method is called meta-synthesis. If the research abstracts lack a systematic methodological description, they are called narrative reviews. These are alternatives when the best evidence available does not come from RCTs or the studies cannot be combined in a meta-analysis, when a broader narrative review is more proper. In the case of nursing, the range of interpretative qualitative studies is considerable, but their impact on the EBP movement is limited, which has been a factor for exclusion from SRs. Nevertheless, some studies have started to synthesize qualitative research findings, based on the argument that an isolated study by itself does not contribute to the understanding of the problem. Thus, the meta-synthesis provides a framework to synthesize non-experimental studies related to the phenomena of interest. However, meta-analysis and meta-synthesis are mutually exclusive and cannot be used in the same review⁽⁷⁾.

Presentation of results

SR results can take the form of conclusions, result analysis or synthesis. The conclusions show primary study outcomes as different categories or themes. Result analyses offer conclusions and descriptions, as well as the assessment of methods, findings and their applicability. The synthesis of results is a higher level of abstraction than conclusions or analysis, as it includes a new model or framework for the topic of interest. SRs are a type of research and, as such, the researchers are responsible for adequately disseminating their results. They should be written in comprehensible language and their methods and conclusions should be communicated clearly. To give an example, the result presentation guide issued by York University includes: abstract, information antecedents, details of the research question, review methods, details for included or excluded studies, review results, analysis of result strength, discussion of limitations and potential bias, implications of the review, list of references and implications for further research⁽⁶⁾. It is essential for all components to be included, so that the reader can evaluate the review's scientific rigor and, thus, apply the evidence in practice.

Utility of systematic reviews

The SR is considered one of the solutions to help and use studies, as it permits a synthesis of available studies on a particular theme and, thus, orient practice based on scientific knowledge⁽²¹⁾. A good SR is of unquestionable value for practice as it helps to solve clinical problems. Therefore, SRs more commonly intend to summarize evidence on the efficacy of an intervention and, hence, have focused on RCT.

Through the critical analysis of primary studies, SRs exposed inconsistencies among distinct pieces of research evidence. The researchers need SRs to summarize existing data, refine hypotheses, estimate sample sizes and help to define future work agendas. In addition, without SRs, researchers might undertake studies about questions that have already been solved⁽³⁾.

Practical guides developed based on SRs offer clear and credible recommendations for practice, make available summaries of evidence and support actions to be taken with regard to health problems. Nevertheless, they differ according to the source, the degree of confidence in the literature review, expert opinions and the prevailing opinion⁽²²⁾.

Conclusions

The use of SRs as a source of evidence for making health decisions has rapidly increased. This evolution follows the volume of health literature, the qualitative range of published research and the quantity of evidence needed to support practice, as a consequence of the number of available health care interventions. As a result, SRs play a fundamental role in health care, substituting primary studies as a source of evidence to support decision making. The main study focus of SRs is the efficacy of an intervention, which is important as a component of the evidence needed to support health decisions. SRs represent the gold standard of research summaries, which is why they should be capable of directing all types of health research, and not only summarize RCTs to determine the effectiveness of an intervention. Moreover, they should provide a framework for the synthesis of many independent study findings, in order to provide valid evidence on a topic of interest. The search for random clinical trials and quasi-experimental designs as a reference would provide a better standard of evidence. In their absence, however, a narrative synthesis of existing knowledge could be made, updating existing evidence that can also be included in EBP.

This article offers a global but structured perspective on the construction of systematic reviews, emphasizing fundamental conceptual and practical aspects for their accomplishment or their application in clinical contexts; thus, the goal is to stimulate the use of this tool to face health professionals' daily needs and help them in clinical decision making.

Received: May. 31^st 2009

Accepted: Sep. 2^nd 2009

• 1. Whittemore R. Combining evidence in nursing research: methods and implications. Nurs Res 2005;54(1):56-62.
• 2. Newell R, Burnard P. Vital Notes for Nurses: Research for evidence-based practice. New Jersey: Wiley Blackwell; 2006.
• 3. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med 1997;126(5):376-80.
• 4. Greener J, Grimshaw J. Using meta-analysis to summarise evidence within systematic reviews. Nurse Res 1996;4(1):27-38.
• 5. Green S, Higgins JP, Alderson P, Clarke M, Mulrow C, Oxman A. Introduction. In: Higgins JP, Green S. (Eds). Cochrane handbook for systematic reviews of interventions. England: Willey-Blackwell; 2008.
• 6. Magarey JM. Elements of a systematic review. Int J Nurs Pract 2001;7(6):376-82.
• 7. Evans D, Pearson A. Systematic reviews: gatekeepers of nursing knowledge. J Clin Nurs 2001;10(5):593-9.
• 8. Hek G, Langton H, Blunden G. Systematically searching and reviewing literature. Nurse Res 2000;7(3):40-57.
• 9. Mulrow CD, Cook DJ, Davidoff F. Systematic reviews: critical links in the great chain of evidence. Ann Intern Med 1997;126(5):389-91.
• 10. Clark K, Ohlsson A. The contribution of the Cochrane Collaboration and the Canadian Cochrane Network and Centre. J Can Chiropr Assoc 2002;46(3):137-43.
• 11. Levin A. The Cochrane Collaboration. Ann Intern Med 2001;135(4):309-12.
• 12. Green S, McDonald S. Cochrane Collaboration: more than systematic reviews? Intern Med J 2005;35(1):4-5.
• 13. Summerskill W. Cochrane Collaboration and the evolution of evidence. Lancet 2005;366(9499):1760.
• 14. Holopainen A, Hakulinen-Viitanen T, Tossavainen K. Systematic review--a method for nursing research. Nurse Res 2008;16(1):72-83.
• 15. Meade MO, Richardson WS. Selecting and appraising studies for a systematic review. Ann Intern Med 1997;127(7):531-7.
• 16. McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc 2005;93(1):74-80.
• 17. Sindhu F, Dickson R. The complexity of searching the literature. Int J Nurs Pract 1997;3(4):211-7.
• 18. Lynn MR. Meta-analysis: appropriate tool for the integration of nursing research? Nurs Res 1989;38(5):302-5.
• 19. Thompson SG. Systematic Review: Why sources of heterogeneity in meta-analysis should be investigated. BMJ 1994;309(6965):1351-5.
• 20. McQuay HJ, Moore RA. Using numerical results from systematic reviews in clinical practice. Ann Intern Med 1997;126(9):712-20.
• 21. Galvăo CM, Sawada NO, Trevizan MA. Revisăo sistemática: recurso que proporciona a incorporaçăo das evidęncias na prática da enfermagem. Rev. Latino-Am. Enfermagem. 2004; 12(3):549-56.
• 22. Cook DJ, Greengold NL, Ellrodt AG, Weingarten SR. The relation between systematic reviews and practice guidelines. Ann Intern Med 1997;127(3):210-6.

Publication Dates

• Publication in this collection
  27 Sept 2010
• Date of issue
  Aug 2010

History

• Received
  31 May 2009
• Accepted
  02 Sept 2009