Overview of systematic reviews - a new type of study. Part II

ABSTRACT CONTEXT AND OBJECTIVE: Overviews of Systematic Reviews (OoRs) are a new type of study in which multiple evidence from systematic reviews (SRs) is compiled into an accessible and useful document. The aim here was to describe the state of the art and critically assess Cochrane OoRs that have been published. DESIGN AND SETTING: Descriptive study conducted at a research center. METHODS: The OoRs identified through the filter developed in Part I of this study were evaluated in five domains: methodological quality; quality of evidence; implications for practice; general profile of OoRs; and length of work. RESULTS: All 13 OoRs included had high methodological quality. Some OoRs did not present sufficient data to judge the quality of evidence; using sensitivity analysis, the quality of evidence of the OoRs increased. Regarding implications for practice, 64% of the interventions were judged as beneficial or harmful, while 36% of them showed insufficient evidence for judgment. It is expected (with 95% confidence interval) that one OoR will include 9,462 to 64,469 patients, 9 to 29 systematic reviews and 80 to 344 primary studies, and assess 6 to 21 interventions; and that 50 to 92% of OoRs will produce meta-analysis. The OoRs generated 2 to 26 meta-analyses over a period of 18 to 31 months. CONCLUSION: The OoRs presented high methodological quality; the quality of evidence tended to be moderate/high; most interventions were judged to be beneficial/harmful; the mean length of work was 24 months. The OoR profile adds power to decision-making.


INTRODUCTION
Overviews of Systematic Reviews (OoRs) are a new type of study that has been proposed by the Cochrane Collaboration in order to compile multiple evidence from systematic reviews (SRs) into a single document that is accessible and useful.[3][4][5][6] One SR rarely addresses all potential interventions for a condition, and healthcare policymakers may have difficulty in finding, evaluating, comparing and summarizing the information from all the relevant SRs. 3,4Thus, the main objective of OoRs is to serve as a friendly front end for the Cochrane Collaboration with regard to healthcare decision-making.The relevant SRs are integrated and/or summarized into a single document, i.e. an OoR.[3][4][5][6] Therefore, the primary audience for OoRs are healthcare decision-makers, such as healthcare professionals, policymakers and informed consumers who, through the Cochrane Library, seek evidence on treatments for various health conditions. 1,2e first part 1,2 of this series of three articles on OoRs focused on the growth of publications with the best level of evidence available for healthcare decision-making.It provided justifications for implementing this new type of study, as well as defining who the target audience are.Furthermore, a filter was created and applied in order to search for specific OoRs in the Cochrane Library.This second part of the series continues to address this topic by describing the state of the art (state of knowledge) of Cochrane Collaboration OoRs, through a critical assessment of Cochrane OoRs.In Part III, a new hierarchy for the pyramid of evidence will be proposed, taking this new type of study into consideration.

OBJECTIVE
To critically assess Cochrane Overviews of Systematic Reviews, through analyzing the characteristics, approaches and methodological aspects of this type of study.Data synthesis was performed using descriptive statistics.Contingency tables were used to summarize dichotomous data as frequencies and proportions.Quantitative data were summarized using the mean and standard deviation.Sensitivity analysis were performed to assess the robustness of the results.

RESULTS
The search filter for OoRs that was developed in Part I 1,2 of this study was updated, validated 9 [sensitivity = 1.00 (95% CI = 0.86 to 1.00); specificity = 0.99 (95% CI = 0.99 to 1.00)] and used (on November 5, 2013).Through this process, 1207 titles were retrieved, of which 95% were excluded because they had not been published in the Cochrane Database of Systematic Reviews.
Ninety-three references were checked, but 52 of these were systematic reviews and were excluded; and another 26 potential OoRs  were excluded because they were at the protocol stage.
All the OoRs that were included 26,[35][36][37][38][39][40][41][42][43][44][45][46] had high methodological quality, reaching between 9 and 11 points out of the 11 points possible in AMSTAR (Table 1).However, some OoRs lost points regarding methodological quality for the following reasons: they did not record the protocol 36 or, if it was cited, did not make it available; 38 they only included searches in the Cochrane Database of Systematic Reviews (CDSR); 26,[35][36][37]39,43,44,46 they did not consider the quality of evidence in formulating conclusions; 46 they were unable to "meta-analyze" the data; 40,42,43,46 and they did not assess the risk of bias among the reviews included. 36 With regrd to judging the quality of evidence, it was found that many categories/studies were unclear, and thus the proportion of the categories/studies that presented high-quality evidence was low.If the OoRs 40,42,46 in which more than 10% of the studies/categories could not be clearly judged regarding the quality of evidence were excluded, the proportion of studies/categories that were of high quality doubled.The sensitivity analysis drastically altered the proportions of the other categories regarding the quality of evidence (Table 1).
Through the outcome of implications for practice, we found that about 64% of the interventions were judged to be beneficial or harmful.However, there was insufficient evidence to judge the interventions in 36% of them (Table 1).
Regarding general factors, with a 95% confidence interval, we expect that each new OoR will include between 9,462 and 64,469 patients, between 9 and 29 SRs and between 80 and 344 primary studies, and will assess between 6 and 21 interventions.Between 50 and 92% of OoRs that pool data qualitatively will generate between 2 and 26 meta-analyses.Additional details are analyzed in Table 1 OoRs excluded because they were protocols (n = 25)  OoRs assessed for inclusion (n = 41) OoRs included for analysis (n = 13)  41,43,48 in which more than 10% of the studies/categories could not be clearly judged regarding the quality of the evidence were excluded from the sensitivity analysis; d judgments based on main outcome conclusion; e some data could not be determined (patients/OoR, 39,41 SRs/ OoR 42 and studies/OoR 41 ), even by looking for results tables in the OoRs or SRs included, thus reducing the number of OoRs in the analysis; f studies in which no meta-analysis was conducted 37,41,[43][44][45] were not included; g CDSR is the abbreviation for Cochrane Database of Systematic Reviews; h only OoRs 39,[41][42][43]46 in which additional searches in the CDSR were conducted were included in the analysis; i the protocols for these OoRs 37,39 were not found in the Cochrane Library or in Archie, thus reducing the number of OoRs in the analysis; SD = standard deviation; range = minimum to maximum; SR = systematic review.
Most of the OoRs (62%) conducted searches for systematic reviews only in the Cochrane Database of Systematic Reviews (CDSR).However, those that conducted external searches, in addition to searching in the CDSR, used between one and ten databases (Table 1).
Table 1 also shows that, from the date when the OoR title was registered, the authors spent about two years (about one year for planning/preparing the protocol and another year for implementing the full study) until the date of publication.
Only 23% (3/13) of the OoRs did not identify the study type in the title.The same also occurred in the protocols for OoRs (6/28).

DISCUSSION
The Cochrane Collaboration's OoRs are supervised by one of the 16 methods groups, the Comparing Multiple Interventions Methods Group.This methods group of the Cochrane Collaboration was established in 2004 and initially was called the Umbrella Reviews Working Group. 47her milestones during the development of the OoRs methodology occurred in 1996, 1997, 1998 and 2005.Between   February 1997 and December 1998, the first series of OoRs, [48][49][50][51] four reviews relating to pregnancy were published as a partnership between researchers from the UK Cochrane Centre and the World Health Organization.In 1996, Julian Higgins and Anne Whitehead published the first article to describe the standard Bayesian approach towards multiple-treatment metaanalysis (MTM). 52In 2005, in the 13 th Cochrane Colloquium in Melbourne, Georgia Salanti made a presentation on MTM methodology, 53 co-authored with Julian Higgins and Valeria Marinho.This won the prize for best oral presentation and helped to popularize the technique.
As examples of Cochrane systematic reviews, 54,55 all the OoRs included in this review [35][36][37][38][39][40][41][42][43][44][45][46] had high methodological quality and therefore less risk of bias, according to the results obtained through AMSTAR. 8However, the reasons why not all the OoRs included obtained the maximum score need to be discussed: 1.The protocols for two OoRs 36,38 were not found in the Cochrane Library or in Archie, even though one of them cited the protocol. 3844,46 restricted the search to only one database, i.e. the CDSR.Moreover, we did not consider this to be a potential source of bias, taking the view that the search methods for systematic reviews in the CDSR are rigorous and comprehensive (including no restrictions on date, publication status or language), and therefore that there was a high probability of including all the relevant primary studies.

Eight OoRs
3. Four OoRs 40,42,43,46 could not match any of the systematic reviews included in their quantitative data synthesis, through either direct or indirect meta-analysis, and therefore summarized and integrated the evidence qualitatively.
4. One OoR 36 did not evaluate or discuss the risk of publication bias among the systematic reviews included.
Although AMSTAR 8 was developed to assess the methodological quality of systematic reviews, it needs to be borne in mind that many of the items that it assesses are also present in OoRs, even though OoRs are different studies.Thus, AMSTAR 8 can be considered to be an analogous tool for assessing OoRs, in which external validity is preserved.Development of an instrument for judging the methodological rigor and consistency of OoRs will be important for reducing the uncertainties in decision-making from these studies.
Despite the high methodological quality of the study by Ryan et al., 40 it only described how the systematic reviews included had assessed the quality of evidence, thus making it difficult to assess this outcome.Many studies (890/2141) may not have made judgments regarding the quality of evidence.In the second of these OoRs, Jones et al. 42 used the Cochrane risk of bias tool 58,61 to assess the quality of the evidence in Cochrane reviews, while for non-Cochrane reviews they used Jadad et al. 60 Despite the differences in the concepts of quality of evidence and methodological quality (which were not among the objectives of this discussion), these authors reported the proportion of studies that presented high quality.Thus, the description was not sufficiently clear or standardized for a judgment to be made regarding the overall quality of evidence.
To assess the quality of evidence, the most consistently used method among the OoRs assessed 26,[35][36][37][38][39][40][41][42][43][44][45][46] was the GRADE approach. 57Basically, this tool classifies evidence into four levels: (1) high quality -it is unlikely that future research will change the estimated effect; (2) moderate quality -future research may have a major impact on the estimated effect and change the result; (3) low quality -further research is very likely to have an important impact on the estimated effect and change the results; and (4) very low quality -there is no certainty in the estimated effect.In the GRADE approach to the classification of evidence, the assessment can be lowered or raised according to certain methodological factors.Methodological limitations, inconsistent results, imputations of evidence, imprecision of results and publication bias diminish the quality of evidence.The magnitude of the effect, controlling for confounders and dose-response gradients are methodological factors that may raise the level of evidence.
Furthermore, comparing the results from this descriptive systematic review of published studies with other OoRs 62,63 that assessed this new type of study, the Cochrane OoRs seem to have higher quality and methodological rigor than non-Cochrane OoRs.To reduce the inconsistency of OoRs, it is important that authors follow the existing methodological recommendations. 3,5,61en extracting data, attention was drawn to two factors: (1) three studies 37,39,43 did not declare the type of study, i.e. "overview of reviews" (there was a similar proportion with regard to protocols for OoRs: 6/28), which may be a limiting factor in that it becomes more difficult for readers to easily identify the type of study; (2) Singh et al. 41 only included primary studies from the list of references of systematic reviews and conducted search strategies in bibliographic databases.
Another feature that may pose a challenge in conducting this type of study is the complex statistical methods required for integrate and summarizing the evidence.The methodological basis for network meta-analysis, also known as multipletreatment meta-analysis and mixed-treatment comparisons, was established in 1996. 52  Participants: Physicians, dentists, nurses and allied healthcare professions (such as physiotherapists, speech therapists, etc.) involved in providing direct patient care.Healthcare providers could be targeted individually or at the level of the organization within which they worked.

Interventions:
Payment for working for a specified time period (e.g. a salary or sessional payment); payment for each service, episode or visit (fee-for-service); payment for providing care for a patient or specific population (e.g.capitation); payment for providing a pre-specified level or change in activity or quality of care (e.g.target payments or bonuses); and mixed and other systems (comprising more than one of the above groups, or not classifiable).

Main outcomes:
Measurements of healthcare professionals' clinical behavior such as the rates of performing preventive actions, diagnosis and treatment (e.g.immunization, blood pressure measurement, prescription and referral); measurements of health service use by patients, such as participation rates in immunization schemes or mammography screening programs; healthcare costs, either combined with measurements of healthcare professional behavior, quality of care or health outcomes to produce measurements of efficiency, or uncombined; including costs of (i) introducing the incentives; (ii) the transaction; (iii) the information systems required to implement the financial incentive; and (iv) monitoring.Quality of the OoR: High quality (10 points in AMSTAR).Quality of the evidence: The results were graded as low to moderate using interpretations of the studies included in the systematic reviews.

Authors' conclusion:
Financial incentives may be effective in changing healthcare professional practice.The evidence has serious methodological limitations and is also very limited in its completeness and generalizability.There is no evidence regarding the effect of financial incentives on patient outcomes.Future studies should use robust designs and include bias, data and economic evaluations.Adult participants with pain of at least moderate intensity that became established after surgery.

Interventions:
Mild analgesics, such as paracetamol, and non-steroidal anti-inflammatory drugs (NSAIDs) such as ibuprofen and celecoxib; weaker opioids such as codeine; and strong opioids such as oxycodone and fentanyl.Main outcomes: Total pain relief (TOTPAR); remedication time; and adverse events.Quality of the OoR: High quality (10 points in AMSTAR).Quality of the evidence: The quality of the evidence was good.
descriptive study was conducted at the research center of a federal university in Brazil and within one of its postgraduate study programs.We performed a search for OoRs in the Cochrane Library, as described in Part I 1,2 of this series.The flowchart for the OoRs is shown in Figure 1.The inclusion criteria were that the studies needed to be OoRs and to have been published in the Cochrane Database of Systematic Reviews, which is one of the six directories of the Cochrane Library. 7Protocols for Cochrane OoRs that have been published were excluded.After OoR selection, two of the present authors (VS and AJG) read them, extracted data and assessed the quality.Differences in data collection information were resolved by reaching a consensus.The data extracted from each OoR were organized using a specific form that sought information on the research question and objective, date of search, number of studies included, participants, interventions, main outcome, methodological quality of the review, quality of evidence and authors' conclusion.We used five items to critically assess the OoRs: (1) methodological quality, using the AMSTAR tool (Assessing the Methodological Quality of Systematic Reviews); 8 (2) quality of evidence, assessed using the GRADE tool (Grades of Recommendation, Assessment, Development and Evaluation) or any other method reported through OoRs; (3) implications for practice (the evidence confirms that the intervention presents benefits; the evidence confirms that the intervention presents harm/risk; or absence of evidence for a recommendation); (4) general profile of OoRs included (patients/OoR, SRs/OoR, studies/OoR, interventions/OoR, meta-analysis/OoR and search strategy); (5) Length of work, i.e. the time taken to publish the OoR (in years), obtained from the date of registration of the title (via Archie), date of publication of the protocol (via the Cochrane Library) and date of publication of the OoR (via the Cochrane Library).

Figure 1
Figure 1 presents the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) flowchart for the OoRs. .
39ore et al.39Review question/objective: What is the efficacy of pharmaceutical interventions for acute pain in adults after surgery?