Detecting the extent of control over selection bias relating to oral health and otorhinolaryngology: cross-sectional study

ABSTRACT BACKGROUND: The authors of randomized controlled trials will usually claim that they have met the randomization process criterion. However, sequence generation schemes differ and some schemes that are claimed to be randomized are not genuinely randomized. Even less well understood, and often more difficult to ascertain, is whether the allocation was really concealed. OBJECTIVE: To detect the extent of control over selection bias, in a comparison between two Cochrane groups: oral health and otorhinolaryngology; and to describe the methods used to control for this bias. DESIGN AND SETTING: Cross-sectional study conducted in a public university in São Paulo, Brazil. METHODS: The risk of selection bias in 1,714 records indexed in Medline database up to 2018 was assessed, independent of language and access. Two dimensions implicated in the allocation were considered: generation of the allocation sequence; and allocation concealment. RESULTS: We included 420 randomized controlled trials and all of them were evaluated to detect selection bias. In the sample studied, only 28 properly controlled the selection bias. Lack of control over selection bias was present in 80% of the studies evaluated in both groups. CONCLUSION: The two groups were similar regarding control over selection bias. They are also similar to the methods used. The dimension of allocation concealment appears to be a limiting factor with regard to production of randomized controlled trials with low risk of selection bias. The quality of reporting in studies on oral health and otorhinolaryngology is suboptimal and needs to be improved, in line with other fields of healthcare.


INTRODUCTION
Randomized controlled trials (RCTs) are considered to be a powerful research design for evaluating the effects of healthcare interventions. They constituted one of the most important scientific advances during the 20 th century. Through using such trials, researchers have the assurance that the differences found between the groups evaluated truly result from the effectiveness of the intervention, given that the allocation is random, i.e. there is an equal distribution of prognoses between the groups. [1][2][3] The controls over the allocation implementation process include generation of a random allocation sequence and simultaneous allocation concealment. 4,5 It is fundamentally important that the investigators should not be capable of anticipating the allocation of the next participants. Absence of controls over the allocation process is a major barrier to internal validity, because this allows the researcher to predict the participants' allocation at the recruitment stage. Even when bias signals are minimal, systematic differences in prognosis can be expected between the groups that will be compared. In particular, selection bias may compromise any randomized experiment in which the enrollment of subjects is sequential and the administration of treatments is unmasked. [6][7][8][9] The authors of RCTs will usually claim that they have met the randomization process criterion, in the title of the article or in the abstract. However, sequence generation schemes differ, and some schemes that are claimed to be randomized are not genuinely randomized. [10][11][12][13] Even less well understood, and often more difficult to ascertain, is whether the allocation was really concealed. Allocation concealment is actually part of the randomization process and, while distinct from the method used to generate the randomized sequence, is essential to the success of randomization. 13 The idea of comparing data from different fields within healthcare is not new. [16][17][18] However, today, there is a tendency for each research group only to evaluate data from their own field. [19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38] True randomized controlled trials allow healthcare providers to make informed inferences about the validity of these trials.

OBJECTIVE
The aims of this study were to detect the extent of control over selection bias through comparing different fields within healthcare: otorhinolaryngology and oral health; and to describe the methods used to control for this bias.

METHODS
This was a cross-sectional study in which the risk of selection bias of randomized controlled trials indexed in the Medline database was assessed.
Primary studies with an RCT design in the fields of otorhinolaryngology (i.e. ear, nose and throat (ENT) diseases) and oral healthcare that were published in journals and indexed in the Medline database were eligible, independent of language and access. A random sample was taken from these studies. These two fields of healthcare were chosen because of the similar numbers of Cochrane systematic reviews that have been published, for reasons of their anatomical and functional proximity and because they share healthcare problems.
Since it was not possible to conduct a cohort study to evaluate the methods of randomized controlled trials, we chose to conduct two cross-over studies at two different time points: 2011-2013 and 2018-2020. Data from this second period have already begun to be collected and as soon as finalized will be compared and published.
Thus, initially, approximately 755 potential RCTs in the oral health group and 959 potential RCTs in the ENT group were retrieved.

Sample size
The sample calculation was performed taking a margin of error of 5%, a difference between groups of 30% and a confidence interval of 95%, in the light of our working hypothesis. It was found that analysis on approximately 200 articles in each field would be necessary. A total of 420 RCTs were included, comprising 214 on ENT and 206 on oral health.

Types of data and methods (variables and bias)
To evaluate the allocation process, the Cochrane Collaboration risk-of-bias tool was used. 9 Two dimensions implicated in the allocation were considered: generation of the allocation sequence; and allocation concealment.
The variable "risk of bias" was segmented into three possible responses: a) low risk of bias, meaning that the author controlled the entire allocation process, comprising random generation and allocation concealment, and therefore that this was an RCT with low risk of selection bias; b) uncertain risk of bias, meaning that the author did not report the allocation process comprising random generation and concealment of the allocation, or reported it in such a way that it was impossible to know for sure what the real situation of the allocation and concealment was; and c) high risk of bias, meaning that the author reported using an incorrect method for the allocation process.

Search methods for identifying studies (data sources)
To identify RCTs in the ENT group, an electronic search strategy recommended by the Cochrane Ear, Nose and Throat Disorders Group was used, with the following search terms: ear nose throat; general ENT and head-and-neck cancer; nose and adenoids, pharynx, larynx and upper respiratory tract infection (URTI); salivary glands; and skull base and neck. This was combined with the Cochrane highly sensitive search strategies for identifying randomized trials in Medline (Figure 1). The same approach was used for identifying RCTs in the oral health group (Figure 1).

Data extraction and management
To assess the risk of the RCTs, two independent reviewers classified the studies, to ensure that these trials could be replicated. The first was a medical researcher (CAF) who was an expert on methodology; and the second was a pharmacist who was an expert on epidemiology (VA). Disagreements were discussed with a third researcher, who was an expert on methodology, to establish a consensus (CASL).

Statistical processing of data
The statistical analyses were performed using the SPSS 19.0 software for Windows. The variable "risk of bias" was taken to be the  ordinal and each group was taken to be the categorical variable.
To ascertain differences between the groups within each dimension of the risk of bias, nonparametric test were used (Pearson chisquare and Mantel-Haenszel). The odds ratio (OR) was used to represent the chances between the groups evaluated. The significance level used in the tests was 5% (alpha = 0.05), and tests with P-values less than 5% (P < 0.05) were taken to be statistically significant.

RESULTS
The Cochrane strategy combined with the key words in the field of oral health recovered 755 records. From this universe, a sample of 206 RCTs was taken. All of these RCTs were evaluated to detect selection bias. In the sample studied, only 38 RCTs properly controlled selection bias (Figure 1).
The Cochrane strategy combined with the key words in the field of ENT recovered 959 records. From this universe, a sample of 214 RCTs was taken. All of these RCTs were evaluated to detect selection bias. In the sample studied, only 28 RCTs properly controlled selection bias (Figure 1).
The level of agreement between the reviewers was measured using the kappa statistical test. The result was satisfactory (kappa = 0.71).     Table 2. Classification of risk of bias in 420 studies regarding the dimension of allocation concealment in the Cochrane oral health and otorhinolaryngology (ear, nose and throat, ENT) groups of selection bias was almost 1.5 times greater in the oral healthcare group than in the ENT group.
Different methods were used for generating the allocation process. Most of the authors used unrestricted randomization to generate the allocation sequence, like computer programs or tables or lists of random numbers. Flipping a coin was used only in the oral health group. Central randomization was used only in the ENT group.
Some authors used restricted randomization, like randomization in blocks: this was seen both in the ENT group and in the oral health group. Only three authors used minimization in the ENT group and only two used minimization in the oral health group. Only one author used random allocation by drawing lots, and this was within oral health (Figure 1).
Most of the authors used sealed numbered envelopes to accomplish allocation concealment. Only two studies used central allocation to achieve concealment in the ENT group and one study in the oral health group (Figure 1).

DISCUSSION
Many study projects may control for large numbers of types of bias, but the means used for adequately applying the allocation process in order to control selection bias is precisely the feature that distinguishes RCTs from other types of study project. 9 In the present study, it was found that selection bias was possible in most of the studies evaluated in both fields. The dimension of allocation concealment appears to be a limiting factor with regard to production of RCTs with low risk of bias, thus representing the main barrier against production of RCTs with control over selection bias. Only 16% of the studies were truly RCTs or enabled control over selection bias.
The two groups were identical regarding control over selection bias and, thus, this appears not to be a condition relating only to the field of oral health. This result was unexpected, because some studies [16][17]19,23,27,[30][31][32]34 have shown that RCTs within oral health were of poor or inadequate quality. Similarly, Peters (2015) assessed the quality of reports and abstracts of RCTs within the literature relating to otorhinolaryngology and found that the quality of reporting of RCTs was suboptimal. This author showed that these articles did not report the allocation process sufficiently.
The only study comparing medicine with dentistry was by Sjögren,17 yet the scale that this author used to evaluate RCTs was the Jadad quality assessment scale. Moreover, the sample was too small: only 200 in each field. The allocation implementation process, which included generation of a random allocation sequence and allocation concealment, was not simultaneous.
There is a dynamic movement towards enhancement of reports and specifically the quality of RCTs, within all fields.  Some authors have shown that improvements in the methodology of published studies within the field of oral health have been achieved. Nonetheless, great concern remains regarding description and reporting of the methods used, in published RCTs. 27,32 There was similarity between the fields regarding the methods used in the allocation process. Simple random allocation is the easiest and most basic approach towards providing unpredictability of treatment assignment. Good methods of generating random allocation sequences include using a random-numbers table or a computer software program to generate the random sequence.
There are manual methods for achieving random allocation, such as tossing a coin, drawing lots or throwing dice. However, these manual methods in practice often become nonrandom, are difficult to implement and do not leave an audit trail. 31 To achieve allocation concealment, opaque sealed envelopes were used in both groups. This method is usually considered accept-  In relation to sequence and allocation concealment, Shulz 13,15 showed that deciphering does occur, most commonly because the method of allocation concealment was inadequate. Even though Altman 14 showed that carrying out allocation concealment is a very simple procedure that can be incorporated into the design of any trial, the data of the present study showed that the dimension of allocation concealment is a limiting factor with regard to production of RCTs with low risk of bias. Thus, allocation concealment is the main barrier against production of RCTs with control over selection bias.
The search for the present analysis was made only in the PubMed database, and other medical databases were excluded. However, PubMed is one of the largest and most widely available databases: it is accessed through the National Institutes of Health (NIH).

CONCLUSION
Efforts need to be made to focus on the allocation implementation process, using only a single step: generation of random allocation sequences and simultaneous implementation of the sequence in such a way that the allocation is concealed. The quality of reporting within the fields of oral health and ENT is suboptimal and needs to be improved so as to match the quality already attained in other fields of healthcare. It is of great importance to take these findings into account, in order to improve the level of evidence of future randomized controlled trials within all fields of healthcare.