Services on Demand
- Similars in SciELO
Print version ISSN 0034-7094
Rev. Bras. Anestesiol. vol.62 no.2 Campinas Mar./Apr. 2012
Fabiano Timbó BarbosaI; Aldemar Araújo CastroII; Cláudio Torres de MirandaIII
IMaster of Health Sciences, Universidade Federal de Alagoas; Professor of Basic Anesthetic and Surgical Technique, Universidade Federal de Alagoas (UFAL)
IIMaster of Vascular Surgery, Universidade Federal de São Paulo; Professor of Scientific Methodology, Universidade Estadual de Ciências da Saúde de Alagoas
IIIPhD in Psychiatry and Medical Psychology, Universidade Federal de São Paulo; Professor of the Health and Society, Universidade Federal de Alagoas
BACKGROUND AND OBJECTIVES: Systematic reviews organize literature data by combining results from published studies in order to resolve conflicts in the area of medical knowledge describing the interventions. The inadequate reporting of systematic reviews can damage the credibility and interfere in the results' quality. The objective of this study was to determine the frequency of good quality systematic reviews comparing neuraxial anesthesia with general anesthesia for procedures on the lower half of the body.
METHODS: Systematic review of systematic reviews. Primary variable: The frequency of good quality systematic reviews. The information was analyzed from the following databases: LILACS (January 1982 to December 2010); PubMed (January 1950 to December 2010); The Cochrane Database of Systematic Review and Database of Abstracts of Reviews of Effects (volume 10, 2010); and SciELO (December 2010). The quality of systematic reviews was determined by the Overview Quality Assessment Questionnaire. The sample size calculation showed that it was necessary to analyze eight systematic reviews, taking into account that the frequency of good quality systematic reviews was 5%, an absolute precision of 15%, and a significance level of 5%.
RESULTS: Were identified 1,995 articles. The selection process eliminated 1,968 articles. Twenty-seven articles of systematic reviews were read in full, 9 were excluded due to incompatibility with the inclusion criteria, and 8 were duplicate publications. Ten systematic reviews were assessed for their quality. The frequency of good quality systematic reviews was 40% (4/10; 95% CI 9.6 to 70.4%).
CONCLUSION: The frequency of good quality systematic reviews was 40%.
Keywords: Anesthesia; Anesthesia, General; Anesthesia, Conduction; Meta-Analysis; Publications.
Systematic reviews organize data existing in the literature combining the results of individual clinical trials for the purpose of resolving conflicts in the medical knowledge investigating interventions such as: diagnostic, therapeutic or prophylactic approaches 1. Several kinds of bias have been identified in this research, such as selection, publication, and pooling biases 2.
The systematic review of the systematic reviews is a research that assesses potential sources of bias in a systematic review to improve the quality of this type of publication 3. It is a research designed primarily to summarize data of multiple reviews focusing on the effects of potential clinical interventions for a condition in health care 1. The main objective is to analyze the quality of systematic reviews to inform readers how flaws can influence the results 1. This research has been done to answer the following question:
What is the frequency of good quality systematic reviews comparing neuraxial anesthesia with general anesthesia for procedures on the lower half of the body?
The purpose of this review was to determine the frequency of good quality systematic review that compared neuraxial anesthesia with general anesthesia for procedures on the lower half of the body.
This systematic review was carried out using methods established by the Cochrane Collaboration 1. The protocol of this systematic review is available at http://tinyurl.com/timbo01. The inclusion criterion was: systematic reviews of the randomized controlled trials that compared neuraxial anesthesia and general anesthesia for surgeries on the lower half of the body. The exclusion criteria were: narrative reviews, case reports with appended literature review, economic evaluations, guidelines and systematic reviews of observational studies.
Cochrane Database of Systematic Review (CDSR) and Database of Abstracts of Reviews of Effects (DARE) were searched in The Cochrane Library (Issue 10, 2010), and conducted electronic searches were made on PUBMED (January 1950 to December 2010), LILACS (January 1982 to December 2010), and SCIELO (last search in December 2010). The following search strategy used for PUBMED was: ("anesthesia, general" [MeSH Terms] OR "anesthes*[Text word]) AND systematic [sb]. The terms "randomized controlled trials" for LILACS, "systematic review" for SCIELO and "anesthesia" for The Cochrane Database were used.
Two reviewers (Barbosa FT and Castro AA) independently assessed titles, abstracts, or both hits retrieved from the databases. Discrepancies were resolved by consensus meeting. There was no language restriction, but all systematic reviews included in this research were published in English. The systematic reviews identified according to the inclusion criteria were fully read to extract data.
The Overview Quality Assessment Questionnaire (OQAQ) was used to evaluate the quality of systematic reviews 4,5. This index has 10 questions, with the first nine analyzing search strategy, selection strategies, quality assessment, pooling, and results 4. These questions were answered as 'yes', 'no', or 'partially/can't tell'. The last question focusing on the overall scientific quality of the systematic review consisted of a 7-point scale: 1 and 2 for extensive flaws, 2 to 4 for major flaws, 4 to 6 for minor flaws, and 6 and 7 for minimal flaws 4,6. Only one reviewer analyzed the quality.
As stated by Kelly et al.6 "The results presented were adjudicated and for ease of interpretation the OQAQ scores were grouped to delete overlapping of scores as follows: one and two indicate extensive flaws, three and four indicate major flaws, five and six indicate minor flaws, and seven indicates minimal flaws". Six and seven points were considered as good quality systematic review.
The primary outcome was the frequency of good quality systematic reviews. The secondary outcomes were: frequency of the OQAQ questions answered, frequency of the PRISMA STATEMENT items reported 7. Complementary data were: the number of clinical trials used, the number of databases used, and the number of reviewers.
Sample size calculation indicated that eight systematic reviews were needed, considering a 5% frequency for good quality systematic reviews, 5% significance level, and 15% absolute precision. Were used simple frequency for all outcomes, except for the number of clinical trials used and the number of reviewers who reported the median and total range of data. The 95% confidence interval for the main result was used. Concordance between the authors was analyzed with Kappa statistic. The main result was compared with data used to calculate the sample size using chi-square test. A significant level of 5% was considered.
The literature search identified 1,995 articles of potential relevance. The study selection process eliminated 1,968 articles reviewing only abstracts and titles. Only 27 systematic reviews were read in full text, but 9 were excluded because they were not compatible with the inclusion criterion, and 8 were published in more than one journal or cited more than once in databases. This process left us with 10 systematic reviews for qualitative analysis 8-17. Kappa statistics for screening systematic reviews was 0.82. This research started in January 2010 and finished in January 2011. The last run of the search strategy was in December 2010.
The frequency of good quality systematic reviews were 40% (4/10, 95% CI 9.6% to 70,4%). The result was statistically different from data used in the hypotheses (p < 0.0001). Table I shows the frequency of the OQAQ questions answered. Overall scientific quality after adjudication was: 0% (0/10) for scores 1, 2, and 3; 50% (5/10) for score 4; 10% (1/10) for scores 5 and 6; and 30% (3/10) for score 7 (Table I).
The frequency of the PRISMA STATEMENT items reported was 100% for rationale, information sources, and summary of evidence. The frequency of the other items is in Table II. The median of the clinical trial numbers was 17 (4 to 24). The frequency for each databases was: 100% (10/10) PUBMED; 80% (8/10) EMBASE; 70% (7/10) CENTRAL; 20% (2/10) CINAHL; and 10% (1/10) LILACS, ISI Web of Science, MD Consult, BIOSIS, Cochrane Bone, Joint and Muscle Trauma Group specialized register, and Cochrane Pregnancy and Childbirth Groups Trials Register. The reviewers median was 3.5 (one to five).
Ten systematic reviews with potential to answer the research question were found. Four systematic reviews were considered as good quality. There was no sufficient information available to consider other six as good quality because there were missing data and their results cannot be considered reproducible in clinical practice. Only 50% of reviews combined the results properly because few authors reported if a sensitivity analysis was performed after combining the results.
Analyzing the methods to execute this research, there were some doubts when question six was marked in OQAQ scale. Some authors did not state one clear way to analyze the validity of the randomized controlled trial included in their systematic review. The authors of the present study contacted Oxman et al. 4 by email to elucidate doubts about question six and received some instructions about this analysis.
There were three limitations: First, the PRISMA STATEMENT has some topics with items composed for multiple components and the recommendation is that all components must be reported. For example, "Provide a general interpretation of the results in the context of other evidence, and implications for future research". In such cases if only one component of the item was reported adequately the item was considered as reported correctly and marked as present in the systematic review. This may have overestimated the results. Second, only one reviewer conducted the score (OQAQ), thus some bias may have occurred in the final result. Third, review's authors were not contacted to clarify the negative points because this approach is not generally used by readers.
Some considerations are possible after analyzing OQAQ: all systematic reviews stated the research methods to search for studies, and 90% of these systematic reviews used more than MEDLINE to identify relevant studies. This approach is correct to identify relevant studies and is recommended to improve quality 18. Fifty percent of the systematic reviews reported the way to assess the validity of the randomized controlled trial. The quality of the studies included in a systematic review should be assessed because the inclusion of trials with lower quality is more likely to produce positive results and compromise the results of the systematic review 19. Findings that combined results properly were seen in 50% of the systematic review analyzed. The reviewers need to report the sensitivity analysis, how heterogeneity was evaluated, and the number of reviewers who evaluated the quality of the review to make it clear that their work has been performed correctly 1.
Analyzing the frequency of the PRISMA STATEMENT items reported, it was identified that only rationale, information sources, and summary of evidence were reported in 100% of the systematic reviews (Table II). In other items, the reviewers did not describe the methods used in sufficient detail to allow a proper analysis, so it was not possible to evaluate the review's quality. Reviewers should pay more attention to their reviews to improve the quality of their work 19.
In view of these results, an implication for future researches can be made. Authors of systematic reviews should report the approach used in their work to analyze the studies and describe in detail what was done and how they performed the work. This attitude offers the reader a good overview of the quality present in a systematic review and gives more credibility to the results presented.
The implication for clinical practice is that the anesthesiologists need to pay attention to the methods used in a systematic review before making decisions about their own practices when choosing the best anesthetic technique for surgical procedures on the lower half of the body. Reading only one systematic review and making decisions about which anesthetic technique should be used can be misleading, as approximately 60% of systematic reviews in this area of knowledge are regarded as poor quality. The anesthesiologist must have the ability to make a critical analysis of this type of publications to evaluate advantages, disadvantages, and limitations and then identify the results that can be reproducible in clinical practice to select the best anesthetic technique at the time of choice.
In conclusion, the frequency of the good quality systematic reviews that compared neuraxial anesthesia with general anesthesia for procedures in inferior half of the body was 40%.
1. Higgins JPT, Green S (editors) - Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated March 2011]. The Cochrane Collaboration, 2011. Disponível em: www.cochrane-handbook.org. Acessado em: 1 de maio 2011. [ Links ]
2. Flather MD, Farkouh ME, Pogue JM, Yusuf S - Strengths and limitations of meta-analysis: larger studies may be more reliable. Control Clin Trials, 1997;18:568-579. [ Links ]
3. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group - Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med, 2009;151:264-269. [ Links ]
4. Oxman AD, Guyatt GH - Validation of an index of the quality of review articles. J Clin Epidemiol, 1991;44:1271-1278. [ Links ]
5. Oxman AD, Guyatt GH, Singer J et al - Agreement among reviewers of review articles. J Clin Epidemiol, 1991;44:91-98. [ Links ]
6. Kelly KD, Travers A, Dorgan M, Slater L, Rowe BH - Evaluating the quality of systematic reviews in the emergency medicine literature. Ann Emerg Med, 2001;38:518-526. [ Links ]
7. Moher D, Liberati A, Tetzlaff J, Altman DG; The PRISMA Group (2009) - Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med, 6(6). e1000097. doi:10.1371/journal.pmed1000097. Acessado em: 1 de dezembro de 2010. [ Links ]
8. Barbosa FT, Cavalcante JC, Jucá MJ, Castro AA - Neuraxial anaesthesia for lower-limb revascularization. Cochrane Database Syst Rev. 2010;(1):CD007083 [Acessado em: 1 de novembro de 2010] [ Links ].
9. Macfarlane AJ, Prasad GA, Chan VW, Brull R - Does regional anaesthesia improve outcome after total hip arthroplasty? A systematic review. Br J Anaesth, 2009;103:335-345. [ Links ]
10. Hu S, Zhang ZY, Hua YQ, Li J, Cai ZD - A comparison of regional and general anaesthesia for total replacement of the hip or knee: a metaanalysis. J Bone Joint Surg Br, 2009;91:935-942. [ Links ]
11. Mauermann WJ, Shilling AM, Zuo Z - A comparison of neuraxial block versus general anesthesia for elective total hip replacement: a metaanalysis. Anesth Analg, 2006;103:1018-1025. [ Links ]
12. Guay J - The effect of neuraxial blocks on surgical blood loss and blood transfusion requirements: a meta-analysis. J Clin Anesth, 2006;18:124-128. [ Links ]
13. Parker MJ, Handoll HHG, Griffiths R - Anaesthesia for hip fracture surgery in adults. Cochrane Database Syst Rev. 2001;(4):CD000521 [update in 2004]; [Acessado em: 1 de novembro de 2010] [ Links ].
14. Urwin SC, Parker MJ, Griffiths R - General versus regional anaesthesia for hip fracture surgery: a met -analysis of randomized trials. Br J Anaesth, 2000;84:450-455. [ Links ]
15. Afolabi BB, Lesi AFE, Merah NA - Regional versus general anaesthesia for caesarean section. Cochrane Database Syst Rev. 2006;(4):CD004350 [Acessado em: 1 de novembro de 2010] [ Links ].
16. Craven PD, Badawi N, Henderson-Smart DJ, OBrien M - Regional (spinal, epidural, caudal) versus general anaesthesia in preterm infants undergoing inguinal herniorrhaphy in early infancy. Cochrane Database Syst Rev. 2003;CD003669 [Acessado em: 1 de novembro de 2010] [ Links ].
17. Liu SS, Strodtbeck WM, Richman JM, Wu CL - A comparison of regional versus general anesthesia for ambulatory anesthesia: a meta-analysis of randomized controlled trials. Anesth Analg, 2005;101:1634-1642. [ Links ]
18. Wilson A, Henry DA - Principles behind practice: meta-analysis. Part 2: assessing the quality of published meta-analyses. Med J Aust, 1992;156:173-187. [ Links ]
19. Moher D, Pham B, Jones A et al. - Does quality of reports of randomized trials affect estimates of intervention efficacy reported in metaanalyses? Lancet, 1998.22;352:609-613. [ Links ]
Correspondence to: Submitted on May 4, 2011 Received from Universidade Federal de Alagoas, Brazil.
Dr. Fabiano Timbó Barbosa
Universidade Federal de Alagoas Campus A. C. Simões Faculdade de Medicina
Av. Lourival Melo Mota, S/N Tabuleiro do Martins
57072-900 - Maceió, AL, Brazil
Approved on June 19, 2011
Submitted on May 4, 2011
Received from Universidade Federal de Alagoas, Brazil.