Reflection about systematic review in public health

Reflexão sobre revisão sistemática em saúde pública

Reflexión sobre la revisión sistemática en salud pública

Abstracts

This study is a reflection on conducting a systematic review in public health. Systematic reviews of public health are fraught with challenges. Complexity is inherent and this may be due to multi-component interventions, diverse study populations, multiple outcomes measured, mixed study designs, or implementation and effectiveness of the review. For policy makers and practitioners to use systematic reviews to implement effective systematic reviews of public health program, the reviews must include this information, which seeks to answer the questions posed by decision-makers, including recipients of the program. We discuss methodological and practice issues that need to be considered when undertaking systematic reviews in this field, including recommendations to reviewers on the issues to address within a systematic public health review and, indirectly, provides advice to researchers on the reporting requirements of primary studies for the production of high quality systematic reviews.

Review; Methods; Public health

Este estudo é uma reflexão sobre a condução da revisão sistemática em saúde pública. Revisões sistemáticas em saúde pública são repletas de desafios. A complexidade é inerente devido a multicomponentes de intervenções, diversas populações de estudo, vários resultados medidos, desenho de estudo misto, aplicação e eficácia. Para os políticos e profissionais usarem revisões sistemáticas para implementar um programa de saúde pública eficaz, as revisões devem incluir estas informações, que buscam responder às questões colocadas pelos tomadores de decisão, incluindo beneficiários do programa. Discutimos questões metodológicas e práticas que precisam ser considerados quando proceder a revisões sistemáticas neste campo, incluindo recomendações para os revisores sobre as questões a abordar dentro de uma revisão sistemática em saúde pública e, indiretamente, prestamos assessoria aos pesquisadores sobre os requisitos de relatórios de estudos preliminares para a produção de revisões sistemáticas de alta qualidade.

Revisão; Métodos; Saúde pública

Este estudio es una reflexión sobre la conducción de la revisión sistemática en salud pública. Las revisiones sistemáticas de la salud pública están llenos de desafíos. La complejidad es inherente, esto puede ser debido a intervenciones con múltiples componentes, las poblaciones de diversos estudios, los resultados de múltiples medidas, el diseño del estudio mixto, aplicación y eficacia. Para los responsables políticos y profesionales a utilizar revisiones sistemáticas para implementar un programa eficaz de salud pública; las revisiones sistemáticas que incluyen esta información, que trata de responder a las preguntas formuladas por los tomadores de decisiones, incluidos los beneficiarios del programa. Se discuten cuestiones metodológicas y prácticas que deben tenerse en consideración al llevar a cabo revisiones sistemáticas en este campo, incluyendo las recomendaciones a los evaluadores sobre los asuntos que deben abordarse dentro de una revisión sistemática de la salud pública e, indirectamente, proporciona asesoramiento a los investigadores sobre los requisitos de información de los estudios primarios para la producción de las revisiones sistemáticas de alta calidad.

Revisión; Métodos; Salud pública

INTRODUCTION

The use of systematic reviews as a source of evidence to organize the increasing number of interventions and scientific information has been increasing rapidly, replacing primary research in decision-making in public health.^{1 - 3} The difficulty of conducting systematic reviews of public health interventions, though, directly reflects the complexity of the interventions reviewed and the subsequent determination of effectiveness.

Some of the key challenges in the public health field include the focus on populations rather than individuals, multi-component interventions, qualitative as well as quantitative approaches, an emphasis on processes of implementation, and the complexity and long-term nature of the interventions and outcomes.⁴ Despite the many methodological challenges, many organizations are systematically reviewing the public health literature and, consequently, contribute to the methodological knowledge of how systematic reviews should be conducted.

Users of public health reviews have raised a number of criticisms of systematic reviews relating to the methodological criteria for including studies, insufficient attention to the quality of the interventions reviewed, and a lack of assessment of the theoretical foundations of the intervention and processes of implementation. Furthermore, reviews have been criticized for their focus on limited individual health education interventions rather than complex environmental or structural interventions, as well as the poor coverage of issues relating to the social determinants of health.⁵ For reviews to be useful and relevant, reviewers need to address these concerns and, therefore, include information that seeks to answer the broad questions posed by decision makers.

This paper discuss the challenges of reviews in this area by providing criteria to enable reviewers to produce high quality systematic reviews that meet the needs of users.

ISSUES TO ADDRESS WHEN REVIEWING PUBLIC HEALTH INTERVENTIONS

Developing a protocol

Before undertaking a systematic review it is important to develop a formal protocol detailing the background, objectives, inclusion criteria, search methods, and proposed analytical methods to be used in the review. If reviewers do not develop a protocol a protocol before initiating the review, there is a danger that the results of the review may exclude studies with unexpected or undesirable results.⁶ Developing and following a detailed protocol protects against this potential bias.

Specifying the review question(s)

The research question is the first step in developing a systematic review. The question must be specific, scientifically relevant and formulated in a systematic way.^{7 - 8} A research question that meets these requirements serves as a guiding principle in determining the review of the criteria for inclusion/exclusion, the method of conducting the review and its logical organization.^{8 - 10}

Clear review questions are generally essential, whatever the exact approach that is adopted. In a policy or management context, such questions should be developed iteratively at or near the beginning of the review process through discussions between the relevant decision-makers, commissioners and the reviewers as they begin to examine the evidence, and as they learn more about the policy environment in which the review will be used.

Inclusion of study designs

The field of public health is characterized by a high degree of methodological pluralism,¹¹ using methods that include Randomized Controlled Trials (RCTs), controlled before-and-after studies, uncontrolled studies, Interrupted Time Series (ITS) designs, surveys and qualitative studies. The study designs to be included in a public health review should be dictated by the interventions being reviewed (methodological appropriateness) and not vice versa.¹²

The production of qualitative research in public health has increased significantly and for this reason there is an important role for systematic reviews to answer questions that go beyond effectiveness, including the appropriateness of interventions for participants³ and the factors that promote and/or impede the implementation of an intervention.¹³ The answers for these questions are seldom found in quantitative studies of effectiveness. Evidence regarding the factors that impinge on the implementation of interventions may be particularly important in the context of complex, multifaceted public health interventions. Moreover, qualitative research can also contribute to reviews through assisting the identification or refining of the questions to be addressed in a systematic review.¹³

Searching for public health literature

Searches for public health studies can be very difficult and time-consuming to develop (up to 5 days), test and re-test the review search strategy in order to ensure that it captures all of the relevant studies. A number of electronic databases should be searched to cover the range of disciplines relevant to the topic area. We recommend that reviewers use care in applying study filters to the search strategy if studies other than RCTs are to be included, as there are no validated filters for non-randomized studies.

Given that public health literature is likely to be widely dispersed, it is important that other retrieval methods, beyond electronic database searches, are utilized.¹⁴ This includes manually searching generalist public health journals and journals related to the topic, which may also result in additional citations to include in the review.

Another method includes searching the Internet to find organizations that may undertake studies, accessing government or public health service provision level reports and asking experts in the area to identify studies that may have been missed. Research-based reviews that exclude grey literature (publicly available, foreign or domestic, open source information that is usually available only through special channels and may not enter normal channels or systems of publication, distribution, bibliographic control or acquisition by book sellers or subscription agents) are more likely to skew the representativeness of studies with statistically significant findings and to artificially inflate estimates of effect size.¹⁵

Quality assessment

The usefulness of systematic reviews in public health depends largely on the quality of the included studies. A critical assessment is the stage where all the selected studies are evaluated using a rigorous methodology to investigate whether the methods and results of research are sufficiently valid to be considered. There is a wide range of views on the selection of studies for reviews.

Particular sources of threats to the validity of public health studies include the data collection methods used, especially where outcomes are subjective and the potential for the control group became "contaminated". The recommended quality assessment tool for quantitative research assesses the following quality criteria: selection bias, study design, confounders, blinding, data collection methods, withdrawals and dropouts, intervention integrity and statistical analyses.¹⁶

Appraising the quality of qualitative research is a much-discussed issue in relation to the role of qualitative research in systematic reviews.³ A number of checklists can be used by the reviewer, including a framework for assessing qualitative research¹⁷ and the Cochrane Qualitative Research Methods Group (http://cqrmg.cochrane.org/) also has a number of tools available to reviewers. We recommend that reviewers use the Critical Appraisal Skills Programme (CASP) appraisal checklist.¹⁸ This tool presents 10 questions that deal very broadly with some of the principles or assumptions that characterize qualitative research: Was there a clear statement of the aims of the research? Is a qualitative methodology appropriate? Was the research design appropriate to address the aims of the research? Was the recruitment strategy appropriate to the aims of the research? Were the data collected in a way that addressed the research issue? Has the relationship between the researcher and participants been adequately considered? Have ethical issues been taken into consideration? Was the data analysis sufficiently rigorous? Is there a clear statement of findings? How valuable is the research?¹⁸

Theoretical frameworks for interventions

The use of theory within systematic reviews may help to explain success or failure in different interventions, by identifying the key elements and highlighting the possible impact of differences between what was planned and what actually happened in the implementation of a programme.¹⁹

Reviewers should seek to examine the impact of an intervention's theoretical framework on the effectiveness of the intervention. For example, when combining the findings from different studies, reviewers can group interventions by their theoretical basis.⁶ Alternatively, reviewers may consider grouping interventions depending on whether they seek to influence individual behavior, interpersonal relationships or community or structural factors or whether they used a Program Logic or Program Theory approach.²⁰

Systematic reviews would be greatly enhanced by attention to the theoretical coverage of interventions in their discussions, e.g., a large number of interventions seek to address individual changes (behavior, attitudes, etc.) but fail to incorporate theories, which seek to change the broader environment within which people make their choices.

Integrity of interventions

Another important issue faced by reviewers is the lack of a generally accepted classification of quality improvement strategies; as a result, it is vital that reviewers clearly define the intervention of interest. Developing the definition of an intervention that can be operationalized within a systematic review frequently requires several interactions, preferably with involvement of a content expert outside the review team to ensure that the resulting definitions are robust and meaningful.

Assessing the degree to which interventions are implemented as planned is important in preventive interventions, which are often implemented in conditions that present numerous obstacles to complete delivery.^{6 , 21} Reviewers should seek to determine whether findings of ineffectiveness within primary studies are simply due to incomplete delivery of the intervention (failure of implementation) or to a poorly conceptualized intervention (failure of the intervention concept or theory).

In order to provide a comprehensive picture of intervention integrity Dane and Schneider²¹ assert that five dimensions of the intervention should be measured. These factors are adherence, exposure, quality of delivery, participant responsiveness and programme differentiation (to prevent contamination).

Heterogeneity

Variability in results between public health studies may arise due to differences in populations, settings, outcomes, interventions and study designs. For these reasons, reviewers should expect considerable heterogeneity across studies and should consider a priori the most appropriate method for synthesis. It is important to explore the potential causes of this in a narrative and statistical manner (where appropriate).⁶

CONCLUSION

Systematic reviews are increasingly recognized as the best source of evidence regarding the effectiveness of different quality improvement strategies. In this paper we have discussed issues that reviewers face when conducting reviews of public health.

Therefore, understanding the concept of developing a systematic review can assist the reader in understanding this type of study. However, the reader must still be prepared to assess the quality of a systematic review and select what interests exist between different reviews. It is important to consider how the conclusions of this type of study can be applied in public health practice, taking into account the patient and the context in which this will be implemented.

Public health practitioners, policy makers and managers need to have access to the findings of high quality systematic reviews to enable them to make informed decisions about whether or not to implement, or participate in, a specific intervention. This paper has sought to provide comprehensive criteria to enable reviewers to produce high quality systematic reviews that are relevant to the questions being asked by end-users.

REFERENCES:

• ¹
  DeLuca JB, Mullins MM, Lyles CM, Crepaz N, Kay L, Thadiparthi S. Developing a comprehensive search strategy for evidence based systematic reviews. Evid Based Libr Inf Pract. 2008 Mar; 3(1):3-32.
• ²
  Guanilo MCDTU; Takahashi RF; Bertolozzi MR. Systematic review: general notions. Rev Esc Enferm USP. 2011 Oct; 45(5):1260-6.
• ³
  Lopes ALM; Fracolli LA. Systematic review of literature and qualitative metasynthesis: considerations about their application in nursing research. Texto Contexto Enferm. 2008 Out-Dez; 17(4):771-8.
• ⁴
  Jackson SF, Edwards RK; Kahan B; Goodstadt M. An assessment of the methods and concepts used to synthesize the evidence of effectiveness in health promotion: a review of 17 initiatives. The Canadian Best Practices Portal for Health Promotion and Chronic Disease Prevention - Public Health Agency of Canada. 2001 [acesso 2012 Mai 28]. Disponível em: http://www.utoronto.ca/chp/CCHPR/synthesisfinalreport.pdf
  » http://www.utoronto.ca/chp/CCHPR/synthesisfinalreport.pdf
• ⁵
  Tilford S. Evidence-based health promotion. Health Educ Res. 2000 Dec; 15(6):659-63.
• ⁶
  May N; Pope C; Popay J. Systematic reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy. 2005; 10(suppl 1):6-20
• ⁷
  Needleman IG. A guide to systematic reviews. J Clin Periodontol. 2002; 29(Suppl 3):6-9.
• ⁸
  Cordeiro AM, Oliveira GM, Rentería JM, Guimarães CA. Revisão sistemática: uma revisão narrativa. Rev Col Brás Cir. 2007 Nov-Dec; 34(6):428-31.
• ⁹
  Richardson WS, Wilson MC, Nishikawa J. The well-built clinical question: a key to evidence-based decisions. ACP J Club. 1995 Nov-Dec; 123(3):12-32.
• ¹⁰
  Galvão CM, Sawada NO, Trevisan MA. Revisão sistemática: recurso que proporciona a incorporação das evidências na prática em enfermagem. Rev Latino Am Enfermagem. 2004 May-Jun; 12(3):549-56.
• ¹¹
  Canesqui AM. Produção científica das Ciências Sociais e Humanas em saúde e alguns significados. Saúde Soc. 2012 Jan-Mar; 21(1):15-23.
• ¹²
  Nutbeam D. The challenge to provide evidence in health promotion. Health Promot Int. 1999 Jun; 14:99-101.
• ¹³
  Centre for Reviews and Dissemination (CRD). Systematic reviews. Guidance for undertaking reviews in health care. Of Research on Effectiveness. University of New York (US). 2001 [acesso 2012 Mai 28]. Disponível em: http://www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf
  » http://www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf
• ¹⁴
  Peersman G, Oakley A. Learning from research. In: Oliver A; Peersman G, organizador. Using research for effective health promotion. Oxford (UK), Oxford University Press; 2001. p.32-43.
• ¹⁵
  Higgins JPT, Green S. editors. Cochrane Handbook for systematic reviews of interventions 4.2.5. 2006 [acesso 2012 Mai 28]. Disponível em: http://www.cochrane.org/sites/default/files/uploads/Handbook4.2.6Sep2006.pdf
  » http://www.cochrane.org/sites/default/files/uploads/Handbook4.2.6Sep2006.pdf
• ¹⁶
  Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG. Assessment of study quality for systematic reviews: a comparison of the cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J Eval in Clin Pract. 2012 Feb; 18(1):12-8.
• ¹⁷
  Spencer L, Ritchie J, Lewis J, Dillon L. Quality in qualitative evaluation: A framework for assessing research evidence. London (UK): Government Chief Social Researcher's Office; 2003.
• ¹⁸
  The University of Kent. Critical Appraisal of the Journal Literature. Critical Appraisal skills programme (CASP). evaluation tool for quantitative studies. England (UK): Public Health Resource Unit; 2006 [acesso 2012 Mai 28]. Disponível em: http://www.casp-uk.net/wp-content/uploads/2011/11/CASP_Qualitative_Appraisal_Checklist_14oct10.pdf
  » http://www.casp-uk.net/wp-content/uploads/2011/11/CASP_Qualitative_Appraisal_Checklist_14oct10.pdf
• ¹⁹
  Sampaio RF, Mancini MC. Estudos de revisão sistemática: um guia para síntese criteriosa da evidência científica. Rev Bras Fisioter. 2007 Jan-Feb; 11(1):83-9.
• ²⁰
  Patton, M.Q. (2002). Utilization-focused evaluation (U-FE) checklist. [acesso 2012 Mai 20]. Disponível em: http://www.wmich.edu/evalctr/archive_checklists/ufe.pdf
  » http://www.wmich.edu/evalctr/archive_checklists/ufe.pdf
• ²¹
  Dane AV, Schneider BH. Program integrity in primary and easly secondary prevention: are implementation effects out of control? Clin Psychol Rev. 1998 Jan; 18(1):23-45.

Publication Dates

• Publication in this collection
  Apr-Jun 2014

History

• Received
  13 June 2012
• Accepted
  20 Mar 2014