Quality analysis of prior systematic reviews of carpal tunnel syndrome: an overview of the literature

ABSTRACT BACKGROUND: Carpal tunnel syndrome (CTS) is a common condition greatly affects patients’ quality of life and ability to work. Systematic reviews provide useful information for treatment and health decisions. OBJECTIVE: This study aimed to assess the methodological quality of previously published systematic reviews on the treatment of CTS. DESIGN AND SETTING: Overview of systematic reviews conducted at the Brazilian public higher education institution, São Paulo, Brazil METHODS: We searched the MEDLINE and Cochrane Library database for systematic reviews investigating the treatment of CTS in adults. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and measurement tool to assess systematic reviews (AMSTAR) were applied by two independent examiners. RESULTS: Fifty-five studies were included. Considering the stratification within the AMSTAR measurement tool, we found that more than 76% of the analyzed studies were “low” or “very low”. PRISMA scores were higher when meta-analysis was present (15.61 versus 10.40; P = 0.008), while AMSTAR scores were higher when studies performed meta-analysis (8.43 versus 5.59; P = 0.009) or when they included randomized controlled trials (7.95 versus 6.06; P = 0.043). The intra-observer correlation demonstrated perfect agreement (> 0.8), a Spearman’s correlation coefficient of 0.829, and an ICC of0.857. The inter-observer correlation indicated that AMSTAR was more reliable than PRISMA. CONCLUSION: Overall, systematic reviews of the treatment of CTS are of poor quality. Reviews with better-quality conducted meta-analysis and included randomized controlled trials. AMSTAR is a better tool than PRISMA because it has a better performance and should be recommended in future studies. REGISTRATION NUMBER IN PROSPERO: CRD42020172328 (https://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42020172328)


INTRODUCTION
Median nerve compression in carpal tunnel syndrome (CTS) affects 1-3 people per 1,000 according to studies in the United States. This syndrome leads to pain, decreased sensitivity, and hand strength, and has a significant detrimental economic impact. 1 The initial treatment of the condition is usually non-operative, and surgical treatment is reserved for cases in which nonsurgical treatment fails or when facing advanced disease. 2 In this context, the aims of CTS treatment include the achievement of more efficient resolution of symptoms and earlier return to work. In recent decades, many studies have been conducted to establish the best treatment for this disease. The advent of systematic reviews and modern methods of statistical evaluation is currently pushing research towards more reliable evidence. However, systematic reviews do not always follow the necessary methodological concepts, leading to imprecision and erroneous conclusions. 3 Recent studies have shown, both in hand surgery as a whole, 4 and specifically in carpal tunnel syndrome treatment, 5 that systematic reviews are often lacking in quality.
To identify poorly conducted systematic reviews, objective tools and questionnaires have been developed to improve the methodological robustness of reviews and to provide a parameter for data collection, analysis, and synthesis of the evidence achieved. These protocols [6][7][8][9] act as safeguards for systematic reviews, and numerous studies in the literature have supported their systematic usefulness.

OBJECTIVE
This study aimed to assess the methodological quality of previously published systematic reviews on the treatment of CTS, as well as to verify the reproducibility of the A Measurement Tool to Assess Systematic Reviews (AMSTAR) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) scores in this scenario, as no study in the literature has previously used these two tools for this purpose.

METHODS
The methodology of this review is registered in the PROSPERO data- and "systematic review" in the Cochrane Library platform with the filter "Other reviews" (Epistemonikos) ("carpal tunnel syndrome" AND "systematic review") AND

(Epistemonikos[filter])
The results of both search strategies were independently analyzed by two researchers (M.C.C. and G.L.O.), and any discrepancies and disagreements were resolved with the help of a senior third author (V.Y.M.). We selected the MEDLINE and the Cochrane Library databases for their worldwide audience and to include relevant research data.

Inclusion criteria
Systematic reviews (with or without meta-analysis) that included any studies (Randomized Clinical Trials or non-Randomized Clinical Trials) evaluating the treatment of CTS in an adult population (18 years or older).

Exclusion criteria
Reviews lacking a transparent literature search and strategy for their data approach, those that were diagnostic-focused, involved anesthetic procedures, or were clearly narrative.

Methodology evaluation (internal validity) and quality reports
The data from all evaluated studies were considered for the elaboration of a descriptive table presenting the various characteristics of the systematic reviews on the topic.
The following were included in the data analysis: journal impact factor (high impact versus low impact), performed a meta-analysis or not, number of institutions involved, total number of patients, total number of words, presence of conflicts of interest, country of origin of the study, citation of PRISMA, and inclusion or exclusion of randomized controlled trials.

Impact factor stratification
The impact factor is expressed as the average number of weighted is a tool comprising 27 items that aids in the formulation and analysis of systematic reviews and meta-analyses. For this analysis, we considered all 27 items and the sum of answers as the final score. Although the overall aim of PRISMA is to help ensure the transparency of systematic reviews, in this study, it was used as a tool in which the sum of its items denoted better quality in the studies, as has been performed in previous studies. 10,11 The acquisition of study data and application of the AMSTAR and PRISMA questionnaires were performed in duplicate. A senior author (V.Y.M.) mediated any cases of disagreement between the examiners.

Data analysis
We defined a priori subgroups for a comparative analysis of the quality of systematic reviews: high-impact journal (Q1) versus low impact (non-Q1), presence of meta-analysis versus non-metaanalysis, randomized controlled trials versus non-randomized clinical trials, statement of interest versus non-declaration, whether PRISMA was cited, country of origin, and number of words.
We defined a priori subgroups for a comparative analysis of the quality of systematic reviews, as follows: high-impact journals (Q1) versus low-impact journals (non-Q1), presence of meta-analysis versus its absence, systematic reviews of randomized clinical trials versus studies that did not employ them, presence of a declaration of interest versus its absence, whether PRISMA was mentioned, country of origin of the study, total number of words, total number of patients, and number of institutions involved.

Statistical analysis
Continuous

RESULTS
In this systematic review, we considered 55 studies.
The PRISMA flowchart, including the reasons for exclusion at each stage, is outlined in Figure 1.Studies characteristics are detailed in Table 1,  and quantitative data are presented in Table 2.
The mean values of the two examiners (Examiner E1 and Examiner E2) for the PRISMA and AMSTAR scores were compared with the following covariates: impact factor, conflict of interest, country of origin, meta-analysis, cite PRISMA, and design of the included studies.
Considering the stratification within the AMSTAR, 87% of the studies evaluated by E1 had "low" or "very low" quality, whereas for E2, this value was 76.4%. Thus, only 2.7% of the studies were classified as having "high" quality ( Table 3).
PRISMA resulted in the highest scores when the studies included meta-analysis (15.61 versus 10.40; P = 0.008). There were no differences in the other variables analyzed, as shown in Table 4. AMSTAR resulted in higher scores when the studies performed meta-analysis (8.43 versus 5.59; P = 0.009) or when they included randomized clinical trials (RCT) (7.95 versus 6.06; P = 0.043), as presented in Table 5.
Journals with the greatest impact did not influence most variables, except for the PRISMA citation statement. In publications that cited PRISMA, 47.6% were low-impact journals and 20.6% were high-impact journals. Among those that did not mention PRISMA, 52.4% were low-impact journals, whereas 79.4% were high-impact journals (P = 0.035), as shown in Table 6.
By assessing the correlation of the country of origin with the same qualitative covariates, we observed a positive correlation between Chinese studies and those that performed meta-analysis (100% in Chinese studies versus 51% in non-Chinese studies) (P = 0.022), as presented in Table 7.
We identified that the intraobserver correlation for E1 and E2 in the AMSTAR and PRISMA scores was above 0.8, with perfect agreement between the pairs, as presented in Table 8.
The inter-observer correlation between the two examiners, using the Blant-Altman model, showed that PRISMA has low reliability, unlike AMSTAR, as the values of the latter were closer to zero, as shown in Table 9.
Applying the Kappa coefficient to assess inter-observer agreement in AMSTAR, revealed substantial agreement (0.61-0.8) when grouping this tool into two variables: "low" or "medium/high" quality studies, as presented in Table 10.
Multivariate analysis using the linear regression model showed a greater impact factor for a journal when a study used meta-analysis, and further showed that multicenter studies have significantly increased PRISMA and AMSTAR scores, as presented in Table 11.

DISCUSSION
Systematic reviews on CTS are mostly of low quality. Several factors are related to better methodological quality, including study design, studies that mention PRISMA, and meta-analyses.
Factors such as conflicts of interest, country of origin, and multicenter studies did not have the same influence.
Similar studies have shown consistent results regarding the intra-observer correlation of the PRISMA and AMSTAR scores. In agreement with our study, these studies found the influence of the presence of meta-analysis on the score values. They also pointed out that there was no difference in the AMSTAR score in terms of the presence of conflicts of interest and impact factor. 11 Other studies have indicated that reviews including only RCTs have better AMSTAR scores, which is similar to the findings of our study. They also observed differences in the PRISMA results of studies that presented declared conflicts of interest. In our study, we did not observe this difference. 67 There have been relatively few studies on the quality of systematic reviews of specific hand and upper limb diseases in orthopedics. However, several of these studies have pointed out that the quality of systematic reviews in leading journals in orthopedics is suboptimal, 68-70 despite having substantially improved following publication of PRISMA. 71 Taking into account the same area of knowledge of hand surgery, an overview of the quality of systematic reviews of the treatment of fractures of the distal radius 9 also showed that studies only including randomized clinical trials and those that performed meta-analyses had better quality.
AMSTAR scores had greater inter-observer agreement than PRISMA scores, especially when dichotomously dividing the qualitative results into high-and low-quality studies. Our findings therefore suggest that AMSTAR is more robust, although improvements are still possible.

Eligibility
PRISMA has emerged as a guideline for systematic reviews with better technical quality, which differs from the AMSTAR scores.
We speculate that this is one explanation for the lower agreement between observers and the lower robustness of this score. In addition, AMSTAR generally presents more detailed items. [7][8][9]67 Observing the relationship between the same covariates and country of origin, we noted that Chinese studies performed meta-analyses more consistently: 100% of Chinese studies included in this study performed meta-analyses, while only 51% of non-Chinese studies performed meta-analyses in their systematic reviews, which supports the current trend of high-quality Chinese studies. 11 Studies citing PRISMA were more common in journals with a lower impact factor. Although this finding is not intuitive, many high-impact journals endorse PRISMA, and we inferred that many high-quality studies rely on the items in this questionnaire despite not explicitly quoting it (i.e. they have a high PRISMA score despite not mentioning it).
Systematic reviews on CTS have consistently revealed recurrent imperfections. Many lost points on PRISMA for presenting an incomplete, unstructured summary, not presenting a review protocol, not presenting a detailed search strategy, not presenting the data combination methods in detail, and not presenting the          similar systems allows for the organization and systematization of the main aspects related to the quality and reliability of information sources. This would further improve the refinement of the best currently available evidence for the treatment of carpal tunnel syndrome.

Limitations
The main limitation of this study was that the search for systematic reviews was published in all languages, except Mandarin.
We tried to minimize biases in the selection, application of questionnaires, and data analysis by carrying out our analysis with independent examiners, and any disagreements were concluded with reference to the senior author. Statistical analysis was conducted by an independent statistician with no conflicts of interest.

CONCLUSION
Our results suggest that published systematic reviews on the treatment of CTS are of low quality, and those that contain meta-analyses and include randomized clinical trials are generally of better quality.
The PRISMA and AMSTAR scores are effective tools for formulating and guiding systematic reviews, although AMSTAR performed better. The reproducibility of AMSTAR scores allows for the analysis of future studies on the treatment of CTS, which is useful for the preparation of other high-quality studies.    PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses; AMSTAR = A measurement tool to assess systematic reviews; Coef. = coefficient; RCT = randomized controlled trial; ANOVA = analysis of variance.