Most Cochrane systematic reviews and protocols did not adhere to the Cochrane's risk of bias 2.0 tool

SUMMARY

OBJECTIVE:

The aim of this study was to identify the frequency of Cochrane systematic reviews and Cochrane systematic reviews protocols using (or planning to use) the risk of bias 2.0 tool to assess the risk of bias of the included randomized clinical trials.

STUDY DESIGN:

This is a meta-research study.

METHODS:

We included Cochrane systematic reviews or Cochrane systematic reviews protocols that planned to include randomized clinical trials. We assessed the Cochrane Database of Systematic Reviews and screened for issues published after the launch of risk of bias 2.0 tool (2019–2022). Two independent investigators performed the study selection and data extraction.

RESULTS:

We analyzed 440 Cochrane systematic reviews and 536 Cochrane systematic reviews protocols. Overall, 4.8% of the Cochrane systematic reviews and 28.5% of the Cochrane systematic reviews protocols used or planned to use risk of bias 2.0 tool. Although low, adherence is increasing over time. In 2019, 0% of Cochrane systematic reviews used risk of bias 2.0 tool, compared to 24.1% in 2022. In Cochrane systematic reviews protocols, adherence increased from 6.9% in 2019 to 41.5% in 2022. A total of 274 (62.1%) Cochrane systematic reviews had their protocols published before 2018; only one used risk of bias 2.0 tool and reported the change of versions in the “Differences between protocol and revision” section.

CONCLUSION:

The Cochrane's risk of bias 2.0 tool has low adherence among Cochrane protocols and systematic reviews. Further efforts are necessary to facilitate the implementation of this new tool.

KEYWORDS:
Systematic review; Research report; Publications; Methods; Systematic reviews as topic

INTRODUCTION

Assessing the risk of bias of individual studies is an essential step in developing a systematic review and a key component of the assessment that grades the certainty of the body of evidence. Ignoring potential biases can directly impact the estimated effects of the intervention and lead to uncertain conclusions¹-³. Since 2008, when Cochrane introduced the Risk of Bias (RoB) tool⁴, authors from systematic reviews of interventions were encouraged to use it to assess the internal validity of the included randomized clinical trials (RCTs). This tool was developed to fill gaps in the available methodological assessment instruments and evaluate the extent of confidence one can have in the RCT methodological steps and its influence on the results¹,⁴.

The original RoB tool comprises seven domains, and the judgment of the risk of bias is performed individually, where each domain can be classified as high, unclear, and low risk of bias. It is worth mentioning that the recommendation for some domains (blinding of participants, personnel and outcome assessors, and incomplete outcome data) is to be assessed not only at the individual study level but separately for each outcome analyzed in the review⁴. After the Cochrane Handbook updates in 2018³, the original RoB tool was replaced by the RoB version 2.0. This new instrument assesses the risk of bias no longer through individual studies and outcomes but by synthesizing study results (analytical level assessment). Like the original version, the new tool is structured in domains, through which bias can be introduced in the study result. In addition, there are “signaling questions,” which involve additional information relevant to the risk of bias assessment⁵. The answer options for these questions are: “yes,” “probably yes,” “probably not,” “no,” “no information,” and “not applicable.” Definitive “yes” and “no” answers often indicate that robust evidence is available. The “not applicable” option is only available for nonmandatory questions. Throughout the application of the tool, the responses fulfill some algorithms that determine the risk of bias for each domain as high risk of bias, low risk of bias, or any concern about bias³,⁵.

Despite the improvement in the interpretation of bias and its influence on the RCT results, RoB 2.0 has a more complex structure than its original version, and there is a growing discussion around its applicability and usability, which seems to limit its wide adoption⁶. Thus, this meta-research study aimed to identify the frequency of Cochrane systematic reviews (CSR) and CSR protocols using (or planning to use) the RoB 2.0 to assess the risk of bias of the included RCTs.

METHODS

Eligibility criteria

We included any CSR or CSR protocols that planned to include RCTs and were published between 2019 and 2022. Updated reviews and network meta-analyses were not considered.

Retrieval strategy

We assessed the Cochrane Database of Systematic Reviews and screened for issues published between January 2019 and March 2022. CSR and CSR protocols were then screened to see if they fulfilled our eligibility criteria. Two independent investigators performed this process. A third investigator solved the disagreements.

Data extraction

We extracted the following data from included reports using a pre-designed Excel spreadsheet:

• publication date of CSR and its respective protocol

• publication data of CSR protocols

• the version of the risk of bias tool used (CSR) or planned to use (CSR protocols)

• descriptions of the change of RoB versions in the section “Differences between protocol and review” (for CSR whose protocol has been published before 2018).

Two independent investigators performed the data extraction process, and a third investigator solved the disagreements.

Data synthesis and presentation

We summarized information using common descriptive statistics. Data were presented in tables. Stata v17 was used for data management and all descriptive analysis.

RESULTS

Considering the eligibility criteria, a total of 440 CSR and 536 CSR protocols were analyzed. Overall, 4.8% (21/440) of the CSR and 28.5% (153/536) of the CSR protocols used or planned to use the RoB 2.0. Table 1 presents the main findings.

Table 1
Adherence to risk of bias 2.0 tool from Cochrane systematic reviews and Cochrane systematic reviews protocols.

		RoB table (original version)	RoB 2.0
CSR (n=440)		419 (95.2%)	21 (4.8%)
	2019 (n=258)	258 (100%)	0 (0%)
	2020 (n=40)	40 (100%)	0 (0%)
	2021 (n=113)	99 (97.6%)	14 (12.4%)
	2022 (n=29)	22 (75.9%)	7 (24.1%)
CSR protocols (n=536)		383 (71.5%)	153 (28.5%)
	2019 (n=116)	108 (93.1%)	8 (6.9%)
	2020 (n=169)	136 (80.5%)	33 (19.5%)
	2021 (n=210)	115 (54.8%)	95 (45.2%)
	2022 (n=41)	24 (58.5%)	17 (41.5%)

• n: number of studies.

Figures 1 and 2 compared the adoption of the two versions of the RoB table by CSR and CSR protocols published between 2019 and 2022.

Figure 1
Risk of bias 2.0 tool adoption by year of publication of Cochrane review.

Figure 2
Risk of bias 2.0 tool adoption by year of publication of Cochrane review protocols.

A total of 274 (62.1%) CSRs had their protocols published prior to the introduction of RoB 2.0 (2018), but only one review used RoB 2.0 to assess the risk of bias of included RCTs and also reported the change of versions in the “Differences between protocol and revision” section, as follows:

RoB 2 tool used (had planned to use the risk of bias tool). Therefore, this section has been re-written in accordance with the editorial checklist for the RoB 2 tool.⁷

DISCUSSION

This meta-research study analyzed 440 CSR and 536 CSR protocols regarding the use of the new tool proposed for assessing the risk of bias, the RoB 2.0. The findings showed that a small proportion of complete reviews and protocols adopted or planned to adopt the RoB 2.0 in assessing their included RCTs. However, there has been increased adherence to RoB 2.0 over the years since its implementation.

Despite the methodological improvements that this new version offered in terms of results and bias interpretation, its structure and application are more complex than the original version, which may contribute to the lower adhesion of Cochrane reviewers. An inter-rater reliability study⁶ found poor agreement among experienced examiners in the overall RoB 2.0 judgment, ranging from slight to moderate for single domains. The complexity of the implementation was attributed to the difficulties in understanding the questions and applying the tool, mainly due to the new terminology and different approaches for some domains, such as “deviations from intended intervention” and “selection of reported results,” and also the conditionality of signaling questions that can raise the risk of wrong interpretation.

Furthermore, some critical issues have been removed and will likely impact the final risk of bias assessment. For example, in Rob 2.0, the absence of selective reporting and assessment of outcomes (reporting bias) has been discussed. In RCTs, the outcomes of interest must be defined in advance and disclosed. Selective reporting bias occurs in numerous situations, such as when planned outcomes and/or their results are not reported, are reported incompletely, or are reported in the final publication of the study, leading to possible overestimation of benefits and underestimation of harm of interventions.

Given the importance of assessing the risk of bias for the applicability of the results of a systematic review, it is important to question why most Cochrane reviewers chose not to use the RoB 2.0. Future survey studies can be a good way to hear from the reviewers themselves about the difficulties and challenges they encountered in applying the tool. Understanding the different versions of this important tool and how to interpret its results helps the review authors critically evaluate the RCTs included in a systematic review and an individual analysis per study. In addition, it is essential to understand the limitations of the current version compared to the original and the need for future adjustments and considerations regarding its use in practice.

CONCLUSIONS

The Cochrane's RoB 2.0 tool has low adherence among Cochrane protocols and systematic reviews. Further efforts are necessary to facilitate the implementation of this new tool.

HIGHLIGHTS

• We conducted a meta-research study to assess the frequency of CSR and CSR protocols using (or planning to use) the RoB 2.0 tool to assess the risk of bias of the included RCTs.

• A total of 440 CSR and 536 CSR protocols were analyzed. Overall, 4.8% (21/440) of the CSR and 28.5% (153/536) of the CSR protocols used or planned to use the RoB 2.0.

• Although low, adherence is increasing over time. In 2019, 0% of CSR used RoB 2.0, compared to 24.1% in 2022. In CSR protocols, adherence increased from 6.9% in 2019 to 41.5% in 2022.

• The Cochrane's RoB 2.0 tool has low adherence among Cochrane protocols and systematic reviews. Further efforts are necessary to facilitate the implementation of this new tool.

• Funding: none.

REFERENCES

• ¹
  Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012. https://doi.org/10.1136/bmj.b4012
  » https://doi.org/10.1136/bmj.b4012
• ²
  Verhagen AP, Vet HC, Bie RA, Boers M, Brandt PA. The art of quality assessment of RCTs included in systematic reviews. J Clin Epidemiol. 2001;54(7):651-4. https://doi.org/10.1016/s0895-4356(00)00360-7
  » https://doi.org/10.1016/s0895-4356(00)00360-7
• ³
  Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions version 6.3 (updated February 2022). Cochrane; 2022.
• ⁴
  Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration; 2011.
• ⁵
  Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019; 366:l4898. https://doi.org/10.1136/bmj.l4898
  » https://doi.org/10.1136/bmj.l4898
• ⁶
  Minozzi S, Cinquini M, Gianola S, Gonzalez-Lorenzo M, Banzi R. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application. J Clin Epidemiol. 2020;126:37-44. https://doi.org/10.1016/j.jclinepi.2020.06.015
  » https://doi.org/10.1016/j.jclinepi.2020.06.015
• ⁷
  Clark B, Whitall J, Kwakkel G, Mehrholz J, Ewings S, Burridge J. The effect of time spent in rehabilitation on activity limitation and impairment after stroke. Cochrane Database Syst Rev. 2021;10(10):CD012612 https://doi.org/10.1002/14651858.CD012612.pub2
  » https://doi.org/10.1002/14651858.CD012612.pub2

Publication Dates

• Publication in this collection
  20 Feb 2023
• Date of issue
  2023

History

• Received
  30 Nov 2022
• Accepted
  15 Dec 2022