An alternative approach to deal with the absence of clinical trials . A proportional meta-analysis of case series studies

PURPOSE: Systematic reviews are criticized for frequently offering inconsistent evidences and absence of straightforward recommendations. Their value seems to be depreciated when the conclusions are uncertain. To describe an alternative approach of evaluating case series studies in health care when there is absence of clinical trials. METHODS: We provide illustrations from recent experiences. Proportional meta-analysis was performed on surgical outcomes: (a) case series studies, (b) use of cryoablation or radiofrequency ablation, and (c) patients with small renal cell carcinoma. The statistically significant difference between both interventions studied was defined if their combined 95% confidential interval (CI) did not overlap. RESULTS: As demonstrated by the example, this analysis is an alternative approach to provide some evidence of the intervention ́s effects under evaluation and plotting all available case series in the absence of clinical trials for the health field. CONCLUSIONS: Although we are leading to a low level of evidence to determine efficacy, effectiveness and safety of interventions this alternative approach can help surgeons, physicians and health professionals for a provisionally decision in health care along with their clinical expertise and the patient ́s wishes and circumstances in the absence of high-quality primary studies. It ́s not a replacement for the gold standard randomized clinical trial, but an alternative analysis for clinical research.


Introduction
Evidence-based medicine (EBM) is defined as the link between good scientific research and clinical practice 1 .One of EBM´s principal is to integrate patient's values and preferences with the current best evidence.In other words, EBM uses existing and available scientific evidence, with good internal and external validity, to apply its results to clinical practice.Systematic review is a type of study focused on a research question that tries to identify, appraise, and synthesize all the research evidence that is of high quality.However, this type of review is often criticized for frequently offering evidence that is inconsistent, along with there being an absence of straightforward recommendations given 1 .Their value seems to be depreciated when the conclusions are uncertain 2 .
A cross-sectional study evaluated the conclusions of 1.016 Cochrane systematic reviews of randomized controlled trials in terms of their recommendations for clinical practice and research.It was concluded that the majority (47.83%) of the analyzed reviews did not offer enough evidence for clinical practice (i.e., insufficient evidence), and the authors did asked for further research 3 .
In 2011, the authors reanalyzed the reviews to evaluate whether this percentage had significantly decreased, again they found that most of the Cochrane systematic reviews did not provide a consistent conclusion, of which the author´s review recommended additional studies 4 .The authors concluded that we should produce higher-quality primary studies, in mass, with participation from worldwide center's to cover all "insufficient evidence" scenarios for clinical practice seen in systematic reviews 5 .
Many health areas present an absence of evidence from level 1b (randomized controlled trials), according to the classification of levels of evidence from the Centre for Evidence Based Medicine 6 , that help to determine the effectiveness, efficacy and safety of different interventions, mainly, in surgical procedures.This seems to have an implication in the design of well-conducted clinical trials as some authors consider sham procedures unethical, since the patients are submitted to anesthesia, and may be exposed to risks.
Other authors defend that a procedure can be ethically justified if there is a relevant clinical question to be answered, or if the use of a control group with sham is methodologically necessary to test the study hypothesis and if the risk of the procedure with sham is minimal 7 .Besides that, there are ethical issues involving randomization in clinical trials evaluating surgical approaches and a participant's vulnerability 8 .
For these reasons the need to create strategies to deal with the absence of clinical trials is essential.Therefore, we describe an alternative approach called proportional meta-analysis of caseseries studies when there are no or insufficient clinical trials.

Methods
A previous study evaluating CA versus RFA showing absence of RCT was our first experience with the application of this alternative analysis.Please refer to El Dib 2012 9 for full report of the original study.

Steps to perform a proportional meta-analysis of case series studies from our recent experience
Search strategy A detailed search strategy was performed to identify all case series studies regarding cryoablation (CA) versus radiofrequency ablation (RFA) to treat renal cell carcinoma with no language restriction.The search strategy was run in the main electronic databases: Medline, Embase, ISI web of Science, and Lilacs using a comprehensive search strategy along with the use of MeSH and text words, including an exhaustive list of synonyms.
The bibliographic references in relevant review articles were also examined for eligible studies.

Inclusion criteria
The inclusion criteria consisted of: (a) case series studies, (b) use of cryoablation or radiofrequency ablation, (c) patients with small renal cell carcinoma regardless of tumor size, and (d) the studies specified a measure of clinical efficacy based on followup imaging.Any case series with incomplete data was excluded from the review.Please refer to El Dib 2012 9 for the definition of clinical efficacy.

Selection of studies and data extraction form
Two reviewers independently screened the titles identified by the literature search, extracted the data, and analyzed the results.A standard form was proposed to extract information such as the authors and year of publication; the number of participants; the mean age of patients; the ablative techniques (cryoablation or radiofrequency ablation), the number of tumors treated, the mean tumor size; and the duration of patients followed up along with the outcomes of interest.
Due to the nature of this type of study there is no tool to evaluate the methodological quality as we are dealing with a low level of evidence.

Statistical analysis
The outcome used as an example for this study was clinical efficacy defined as the percentage of tumours treated successfully by the procedure with its respectively 95% confidential intervals (CI).Due to the clear difference amongst the included studies and several uncontrolled variables, we suggested to use a randomeffect model 9 .StatsDirect was the software used to plot the studies into a meta-analysis.

Interpretation of the forest plots
Forest plots were presented to summarize the data.Each horizontal line on a forest plot represents a case series included in the meta-analysis.The length of the line corresponds to a 95% CI of the corresponding case series' effect estimate.The effect estimate is marked with a solid black square.The size of the square represents the weight that the corresponding study exerts in the meta-analysis.The pooled estimate is marked with an unfilled diamond at the bottom of the forest plot.Confidence intervals of pooled estimates are displayed as a horizontal line through the diamond; where the line might be contained within the diamond if the confidence interval is narrow.
The statistically significance difference between both interventions studied was defined if their combined 95% CIs did not overlap.We considered a p < 0.05 as statistically significant for the calculation of heterogeneity.
Funnel plots were performed by Egger tests to assess the possibility of publication bias as they are useful adjuncts to metaanalyses.
All the proportional meta-analysis of case series studies was performed using the StatsDirect software.

Interpretation of the proportional meta-analysis of case series studies in our example: pooled results
There was no statistically signifi cance difference

Discussion
According to the statistical analysis proposed in this paper we can conclude that there was no signifi cant difference between CA and RFA regarding clinical effi cacy as their CIs overlapped (Figure 5).
Our analysis also demonstrates that there is signifi cant heterogeneity in the clinical outcome of cryoablation (Figure 1), which was already expected to occur as we are dealing with case series studies.Reasons for this heterogeneity could be both of outcomes, and duration of follow-up.There were also methodological differences in the handling of withdrawals and losses to follow up.In addition, the funnel plot for the CA series suggests the possibility that publication bias may have occurred given the asymmetry (Figure 2).As for the RFA series, there was far less heterogeneity when it came to the clinical efficacy outcome (Figure 3).This suggests that the RFA series were far more consistent in patient selection and treatment protocol.

The alternative approach to deal with the absence clinical trials
Overall it is still desirable to describe the existing data, so that physicians and health professionals may have the state of current knowledge mapped.For this reason, a proportional metaanalysis of case series studies is suggested to be conducted with a comprehensive systematic search of uncontrolled studies (i.e., case series) when there are no clinical trials in the literature to answer intervention clinical questions.However, the investigators and policy makers should remain extremely aware of the results as there are many flaws in the internal validity of this type of studies.This is due to the occurrence of much bias in case series studies such as the issue of a no control group and the diversity of clinical and methodological issues.Furthermore, the evidence provided from the proportional meta-analysis of case series studies should be used until appropriate clinical trials are conducted.

Summary on how to perform a proportional metaanalysis of case series studies
How to perform a proportional meta-analysis of case  protocols, to supply all those systematic reviews that did not offer enough evidence for clinical practice' 5 .So until high-quality primary studies are been conducting we offer an alternative approach to deal with the absence of clinical trials in the health field, mainly, in those reviews that did not offer enough evidence for clinical practice (i.e., insufficient evidence), and that the authors did asked for further research.

Conclusions
We describe an alternative approach to evaluate case series studies in health care reviews called proportional metaanalysis of case series studies.An alternative approach to deal with the absence of clinical trials.A proportional meta-analysis of case series studies

Figure 2
Figure 2 represents an asymmetric funnel plot from the outcome of clinical efficacy of cryoablation's case series 20 11-30 by Egger test that indicates a relationship between treatment effect and study size.These results suggest the possibility of publication bias.

FIGURE 3 -
FIGURE 3 -Example of proportional meta-analysis of case series studies 31-41 regarding clinical effi cacy in radiofrequency ablation therapy.

Figure 4
Figure 4 represents the results from the funnel plot of case series studies 11 30-40 regarding the clinical effi cacy outcome in RFA intervention by Egger test.There is a symmetric inverted funnel shape that arises from a 'well-behaved' data set, in which publication bias is unlikely to occur.

FIGURE 4 -
FIGURE 4 -Example of funnel plot of case series studies 31-41 regarding the clinical effi cacy in RFA therapy by Egger test.
regarding clinical effi cacy between CA and RFA therapy as their CIs overlapped (Figures1, 3 and 5).

Figure 5
Figure 5 represents the combined overlapped CIs from CA and RFA therapies.

FIGURE 5 -
FIGURE 5 -Example of the combined overlapped CIs from CA and RFA therapies.
series studies in the absence of RCT to investigate the effectiveness and safety of health interventions can be summarized by the following: a) Perform a comprehensive search strategy to identify all case series studies with no language restriction including an exhaustive list of synonyms for the intervention, control group and clinical situation and run the search strategy in at least the following sources Medline, Embase and, ISI web of Science; b) Define the outcome to be analyzed; c) Extract data relevant to perform a descriptive Although we are leading to a low level of evidence to determine efficacy, effectiveness and safety of interventions, surgical procedures and prevention programmers this alternative analysis can help surgeons, physicians and health professionals for a provisionally decision along with their clinical expertise and the patient´s wishes and circumstances in the absence of high-quality primary studies and while clinical trials are ethically unacceptable or methodologically biased.It´s not a replacement for the gold standard randomized clinical trial, but an alternative for clinical research until well-conducted clinical trials are conducted.The health care professionals should weigh the benefits/risks profile of this approach and also take into consideration the patient's values and preferences.
tableand the proportional meta-analysis of case series studies such as authors and year of publication, number of participants, mean age of patients, information regarding the techniques under evaluation from both the intervention and control groups, duration of patients follow-up and the outcomes of interest; d) Plot the dichotomous data using StatsDirect software; e) Conduct an analysis of CIs and discuss the impact of the results making an allowance for the level of evidence.