Acessibilidade / Reportar erro

An alternative approach to deal with the absence of clinical trials: a proportional meta-analysis of case series studies

Abstract

PURPOSE: Systematic reviews are criticized for frequently offering inconsistent evidences and absence of straightforward recommendations. Their value seems to be depreciated when the conclusions are uncertain. To describe an alternative approach of evaluating case series studies in health care when there is absence of clinical trials. METHODS: We provide illustrations from recent experiences. Proportional meta-analysis was performed on surgical outcomes: (a) case series studies, (b) use of cryoablation or radiofrequency ablation, and (c) patients with small renal cell carcinoma. The statistically significant difference between both interventions studied was defined if their combined 95% confidential interval (CI) did not overlap. RESULTS: As demonstrated by the example, this analysis is an alternative approach to provide some evidence of the intervention´s effects under evaluation and plotting all available case series in the absence of clinical trials for the health field. CONCLUSIONS: Although we are leading to a low level of evidence to determine efficacy, effectiveness and safety of interventions this alternative approach can help surgeons, physicians and health professionals for a provisionally decision in health care along with their clinical expertise and the patient´s wishes and circumstances in the absence of high-quality primary studies. It´s not a replacement for the gold standard randomized clinical trial, but an alternative analysis for clinical research.

Review; Methodology; Clinical Trials as Topic; Meta-analysis; Time Series Studies


10 - ORIGINAL ARTICLE

SYSTEMATIC REVIEWS AND META-ANALYSIS

An alternative approach to deal with the absence of clinical trials. A proportional meta-analysis of case series studies1 1 Research performed at St. Joseph's Healthcare, McMaster Institute of Urology, McMaster University, Canada.

Regina El DibI; Paulo Nascimento JuniorII; Anil KapoorIII

IPhD, Assistant Professor, Anaesthesiology Department, Botucatu Medical School, Sao Paulo State University (UNESP), Brazil. Researcher Collaborator, St. Joseph's Healthcare, McMaster Institute of Urology, McMaster University, Canada. Conception, design, intellectual and scientific content of the study, manuscript writing, critical revision<

IIPhD, Associate Professor, Anaesthesiology Department, Botucatu Medical School, UNESP, Botucatu-SP, Brazil. Critical revision

IIIMD, Renal Transplant Program Surgical Director, Urologic Laparoscopy Program Director. Associate Professor of Surgery, St. Joseph's Healthcare, McMaster Institute of Urology, McMaster University, Hamilton, Canada., Scientific content of the study, manuscript writing

Correspondence Correspondence: Regina El Dib Departamento de Anestesiologia. Faculdade de Medicina de Botucatu-UNESP Distrito de Rubião Júnior, s/n 18618-190 Botucatu - SP Brasil Tel.: (55-11)99999-6647 eldib@fmb.unesp.br

ABSTRACT

PURPOSE: Systematic reviews are criticized for frequently offering inconsistent evidences and absence of straightforward recommendations. Their value seems to be depreciated when the conclusions are uncertain. To describe an alternative approach of evaluating case series studies in health care when there is absence of clinical trials.

METHODS: We provide illustrations from recent experiences. Proportional meta-analysis was performed on surgical outcomes: (a) case series studies, (b) use of cryoablation or radiofrequency ablation, and (c) patients with small renal cell carcinoma. The statistically significant difference between both interventions studied was defined if their combined 95% confidential interval (CI) did not overlap.

RESULTS: As demonstrated by the example, this analysis is an alternative approach to provide some evidence of the intervention´s effects under evaluation and plotting all available case series in the absence of clinical trials for the health field.

CONCLUSIONS: Although we are leading to a low level of evidence to determine efficacy, effectiveness and safety of interventions this alternative approach can help surgeons, physicians and health professionals for a provisionally decision in health care along with their clinical expertise and the patient´s wishes and circumstances in the absence of high-quality primary studies. It´s not a replacement for the gold standard randomized clinical trial, but an alternative analysis for clinical research.

Key words: Review. Methodology. Clinical Trials as Topic. Meta-analysis. Time Series Studies.

Introduction

Evidence-based medicine (EBM) is defined as the link between good scientific research and clinical practice1. One of EBM´s principal is to integrate patient's values and preferences with the current best evidence. In other words, EBM uses existing and available scientific evidence, with good internal and external validity, to apply its results to clinical practice.

Systematic review is a type of study focused on a research question that tries to identify, appraise, and synthesize all the research evidence that is of high quality. However, this type of review is often criticized for frequently offering evidence that is inconsistent, along with there being an absence of straightforward recommendations given1. Their value seems to be depreciated when the conclusions are uncertain2.

A cross-sectional study evaluated the conclusions of 1.016 Cochrane systematic reviews of randomized controlled trials in terms of their recommendations for clinical practice and research. It was concluded that the majority (47.83%) of the analyzed reviews did not offer enough evidence for clinical practice (i.e., insufficient evidence), and the authors did asked for further research3.

In 2011, the authors reanalyzed the reviews to evaluate whether this percentage had significantly decreased, again they found that most of the Cochrane systematic reviews did not provide a consistent conclusion, of which the author´s review recommended additional studies4. The authors concluded that we should produce higher-quality primary studies, in mass, with participation from worldwide center's to cover all "insufficient evidence" scenarios for clinical practice seen in systematic reviews5.

Many health areas present an absence of evidence from level 1b (randomized controlled trials), according to the classification of levels of evidence from the Centre for Evidence Based Medicine6, that help to determine the effectiveness, efficacy and safety of different interventions, mainly, in surgical procedures. This seems to have an implication in the design of well-conducted clinical trials as some authors consider sham procedures unethical, since the patients are submitted to anesthesia, and may be exposed to risks.

Other authors defend that a procedure can be ethically justified if there is a relevant clinical question to be answered, or if the use of a control group with sham is methodologically necessary to test the study hypothesis and if the risk of the procedure with sham is minimal7. Besides that, there are ethical issues involving randomization in clinical trials evaluating surgical approaches and a participant's vulnerability8.

For these reasons the need to create strategies to deal with the absence of clinical trials is essential. Therefore, we describe an alternative approach called proportional meta-analysis of case-series studies when there are no or insufficient clinical trials.

Methods

A previous study evaluating CA versus RFA showing absence of RCT was our first experience with the application of this alternative analysis. Please refer to El Dib 20129 for full report of the original study.

Steps to perform a proportional meta-analysis of case series studiesfrom our recent experience

Search strategy

A detailed search strategy was performed to identify all case series studies regarding cryoablation (CA) versus radiofrequency ablation (RFA) to treat renal cell carcinoma with no language restriction. The search strategy was run in the main electronic databases: Medline, Embase, ISI web of Science, and Lilacs using a comprehensive search strategy along with the use of MeSH and text words, including an exhaustive list of synonyms. The bibliographic references in relevant review articles were also examined for eligible studies.

Inclusion criteria

The inclusion criteria consisted of: (a) case series studies, (b) use of cryoablation or radiofrequency ablation, (c) patients with small renal cell carcinoma regardless of tumor size, and (d) the studies specified a measure of clinical efficacy based on follow-up imaging. Any case series with incomplete data was excluded from the review. Please refer to El Dib 20129 for the definition of clinical efficacy.

Selection of studies and data extraction form

Two reviewers independently screened the titles identified by the literature search, extracted the data, and analyzed the results. A standard form was proposed to extract information such as the authors and year of publication; the number of participants; the mean age of patients; the ablative techniques (cryoablation or radiofrequency ablation), the number of tumors treated, the mean tumor size; and the duration of patients followed up along with the outcomes of interest.

Due to the nature of this type of study there is no tool to evaluate the methodological quality as we are dealing with a low level of evidence.

Statistical analysis

The outcome used as an example for this study was clinical efficacy defined as the percentage of tumours treated successfully by the procedure with its respectively 95% confidential intervals (CI). Due to the clear difference amongst the included studies and several uncontrolled variables, we suggested to use a random-effect model9. StatsDirect was the software used to plot the studies into a meta-analysis.

Interpretation of the forest plots

Forest plots were presented to summarize the data. Each horizontal line on a forest plot represents a case series included in the meta-analysis. The length of the line corresponds to a 95% CI of the corresponding case series' effect estimate. The effect estimate is marked with a solid black square. The size of the square represents the weight that the corresponding study exerts in the meta-analysis. The pooled estimate is marked with an unfilled diamond at the bottom of the forest plot. Confidence intervals of pooled estimates are displayed as a horizontal line through the diamond; where the line might be contained within the diamond if the confidence interval is narrow.

The statistically significance difference between both interventions studied was defined if their combined 95% CIs did not overlap. We considered a p < 0.05 as statistically significant for the calculation of heterogeneity.

Funnel plots were performed by Egger tests to assess the possibility of publication bias as they are useful adjuncts to meta-analyses.

All the proportional meta-analysis of case series studies was performed using the StatsDirect software.

Results

Absence of RCTs on CA compared to RFA for renal cell carcinoma

The results of the search strategy showed that there was no RCT evaluating RFA versus CA for the treatment of renal tumors9. Hence, we found 31 case series studies (20 CA11-30, 11 RFA31-41) that met all inclusion criteria and were included in the meta-analysis of case series studies.

Pooled results

The pooled proportion of clinical efficacy was 89% [95% confidential interval (CI) 0.83 to 0.94) in cryoablation therapy from 2011-30 studies with a total of 457 cases. There was statistical significance regarding heterogeneity (I2 value)=70.6%) (p<0.0001) showing the inconsistency of clinical and methodological aspects between the studies included in the meta-analysis (Figure 1).


Figure 2 represents an asymmetric funnel plot from the outcome of clinical efficacy of cryoablation's case series 2011-30 by Egger test that indicates a relationship between treatment effect and study size. These results suggest the possibility of publication bias.


The pooled proportion of clinical efficacy was 90% [95% CI 0.86 to 0.93) in RFA therapy from 1131-41 studies with a total of 426 cases. There was a low level of heterogeneity (I2 value = 34.1%) (p=0.126) between the studies included in the meta-analysis (Figure 3).


Figure 4 represents the results from the funnel plot of case series studies 1130-40 regarding the clinical efficacy outcome in RFA intervention by Egger test. There is a symmetric inverted funnel shape that arises from a 'well-behaved' data set, in which publication bias is unlikely to occur.


Interpretation of the proportional meta-analysis of case series studies in our example: pooled results

There was no statistically significance difference regarding clinical efficacy between CA and RFA therapy as their CIs overlapped (Figures 1, 3 and 5 ).

Figure 5 represents the combined overlapped CIs from CA and RFA therapies.

Discussion

According to the statistical analysis proposed in this paper we can conclude that there was no significant difference between CA and RFA regarding clinical efficacy as their CIs overlapped (Figure 5 ).

Our analysis also demonstrates that there is significant heterogeneity in the clinical outcome of cryoablation (Figure 1), which was already expected to occur as we are dealing with case series studies. Reasons for this heterogeneity could be both clinical and methodological. The studies differed considerably in their patient selection, baseline disease severity, techniques (laparoscopically, percutaneously or open), management of outcomes, and duration of follow-up. There were also methodological differences in the handling of withdrawals and losses to follow up. In addition, the funnel plot for the CA series suggests the possibility that publication bias may have occurred given the asymmetry (Figure 2). As for the RFA series, there was far less heterogeneity when it came to the clinical efficacy outcome (Figure 3). This suggests that the RFA series were far more consistent in patient selection and treatment protocol.

The alternative approach to deal with the absence clinical trials

Overall it is still desirable to describe the existing data, so that physicians and health professionals may have the state of current knowledge mapped. For this reason, a proportional meta-analysis of case series studies is suggested to be conducted with a comprehensive systematic search of uncontrolled studies (i.e., case series) when there are no clinical trials in the literature to answer intervention clinical questions. However, the investigators and policy makers should remain extremely aware of the results as there are many flaws in the internal validity of this type of studies. This is due to the occurrence of much bias in case series studies such as the issue of a no control group and the diversity of clinical and methodological issues. Furthermore, the evidence provided from the proportional meta-analysis of case series studies should be used until appropriate clinical trials are conducted.

Summary on how to perform a proportional meta-analysis of case series studies

How to perform a proportional meta-analysis of case series studies in the absence of RCT to investigate the effectiveness and safety of health interventions can be summarized by the following:

a) Perform a comprehensive search strategy to identify all case series studies with no language restriction including an exhaustive list of synonyms for the intervention, control group and clinical situation and run the search strategy in at least the following sources Medline, Embase and, ISI web of Science;

b) Define the outcome to be analyzed;

c) Extract data relevant to perform a descriptive table and the proportional meta-analysis of case series studies such as authors and year of publication, number of participants, mean age of patients, information regarding the techniques under evaluation from both the intervention and control groups, duration of patients follow-up and the outcomes of interest;

d) Plot the dichotomous data using StatsDirect software;

e) Conduct an analysis of CIs and discuss the impact of the results making an allowance for the level of evidence.

The next appeal for the Evidence-Based Medicine age

Cochrane's appeal was that it aimed to represent the first step to establishing good evidence for decision making in health care when he said that we need to organize a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials42. However, the next appeal for the Evidence-Based Medicine age, according to El Dib's, is 'A great criticism of the evidence-based medicine age is that we have not produced enough higher-quality primary studies with worldwide centre participation and in accordance with predefined Cochrane's protocols, to supply all those systematic reviews that did not offer enough evidence for clinical practice'5. So until high-quality primary studies are been conducting we offer an alternative approach to deal with the absence of clinical trials in the health field, mainly, in those reviews that did not offer enough evidence for clinical practice (i.e., insufficient evidence), and that the authors did asked for further research.

Conclusions

We describe an alternative approach to evaluate case series studies in health care reviews called proportional meta-analysis of case series studies. Although we are leading to a low level of evidence to determine efficacy, effectiveness and safety of interventions, surgical procedures and prevention programmers this alternative analysis can help surgeons, physicians and health professionals for a provisionally decision along with their clinical expertise and the patient´s wishes and circumstances in the absence of high-quality primary studies and while clinical trials are ethically unacceptable or methodologically biased. It´s not a replacement for the gold standard randomized clinical trial, but an alternative for clinical research until well-conducted clinical trials are conducted. The health care professionals should weigh the benefits/risks profile of this approach and also take into consideration the patient's values and preferences.

Acknowledgement

To Camilla Tajzler for her support with the English grammar revisions.

Received: Aug 12, 2013

Review: Oct 14, 2013

Accepted: Nov 13, 2013

Conflict of interest: none

Financial source: none

  • 1. Browman GP. Essence of evidence-based medicine: a case report. J Clin Oncol. 1999;17:1969-73.
  • 2. Egger M, Smith G.D, Phillips NA. Meta-analysis: principles and procedures. BMJ. 1997;315:1533-7.
  • 3. El Dib RP, Atallah NA, Andriolo RB. Mapping the Cochrane evidence for decision-making in health care. J Eval Clin Pract. 2007;13:689-92.
  • 4. Villas Boas PJ, Spagnuolo RS, Kamegasawa A, Braz LG, Polachini do Valle A, Jorge EC, Yoo HH, Cataneo AJ, Corrêa I, Fukushima FB, do Nascimento P, Módolo NS, Teixeira MS, de Oliveira Vidal EI, Daher SR, El Dib R Systematic reviews showed insufficient evidence for clinical practice in 2004: what about in 2011? The next appeal for the Evidence-Based Medicine age. J Eval Clin Pract. 2013;19:633-7.
  • 5. El Dib RP. Anestesia baseada em evidências. In: Volquind D, Albuquerque MAC, Pires OC, Vianna PTG. Curso de Educação à Distância em Anestesiologia. 12ed. São Paulo: Segmentofarma; 2012. p.1-5.
  • 6. Centre for Evidence Based Medicine (CEBM). Levels of evidence. Available fromhttp://www.cebm.net/index.aspx?o=1025
  • 7. Macklin R. The ethical problems with sham surgery in clinical research. N Engl J Med. 1999;341:992-6.
  • 8. Horng S, Miller FG. Ethical framework for the use of sham procedures in clinical trials. Crit Care Med. 2003;31:S126-30.
  • 9. El Dib R, Touma NJ, Kapoor A. Cryoablation versus radiofrequency ablation for the treatment of renal cell carcinoma: a meta-analysis of case series studies. BJU Int. 2012;110(4):510-6.
  • 10. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7:177-88.
  • 11. Bassignani MJ, Moore Y, Watson L, Theodorescu D. Pilot experience with real-time ultrasound guided percutaneous renal mass cryoablation. J Urol. 2004;171:1620-3.
  • 12. Bachmann A, Sulser T, Jayet C, Wyler S, Ruszat R, Reich O, Gasser TC, Siebels M, Stief CG, Casella R. Retroperitoneoscopy-assisted cryoablation of renal tumors using multiple 1.5 mm ultrathin ryoprobes: a preliminary report. Eur Urol. 2005;47:474-9.
  • 13. Beemster P, Phoa S, Wijkstra H, de la Rosette J, Laguna P. Follow-up of renal masses after cryosurgery using computed tomography; enhancement patterns and cryolesion size. BJU Int. 2008;101:1237-42.
  • 14. Byrd GF, Lawatsch EJ, Mesrobian HG, Begun F, Langenstroer P. Laparoscopic cryoablation of renal angiomyolipoma. J Urol. 2006;176:1512-6.
  • 15. Carvalhal EF, Gill IS, Meraney AM, Desai MM, Schweizer DK, Sung GT. Laparoscopic renal cryoablation: impact on renal function and blood pressure. Urology. 2001;58:357-61.
  • 16. Cestari A, Guazzoni G, dell'Acqua V, Nava L, Cardone G, Balconi G, Naspro R, Montorsi F, Rigatti P. Laparoscopic cryoablation of solid renal masses: intermediate term followup. J Urol. 2004;172:1267-70.
  • 17. Georgiades CS, Hong K, Bizzell C, Geschwind JF, Rodriguez R. Safety and efficacy of CT-guided percutaneous cryoablation for renal cell carcinoma. J Vasc Interv Radiol. 2008;19:1302-10.
  • 18. Gill IS, Novick AC, Meraney AM, Chen RN, Hobart MG, Sung GT, Hale J, Schweizer DK, Remer EM. Laparoscopic renal cryoablation in 32 patients. Urology. 2000;56:748-53.
  • 19. Gupta A, Allaf ME, Kavoussi LR, Jarrett TW, Chan DY, Su LM, Solomon SB. Computerized tomography guided percutaneous renal cryoablation with the patient under conscious sedation: initial clinical experience. J Urol. 2006;175:447-52.
  • 20. Lawatsch EJ, Langenstroer P, Byrd GF, See WA, Quiroz FA, Begun FP. Intermediate results of laparoscopic cryoablation in 59 patients at the Medical College of Wisconsin. J Urol. 2006;175:1225-9.
  • 21. Lee DI, McGinnis DE, Feld R, Strup SE. Retroperitoneal laparoscopic cryoablation of small renal tumors: intermediate results. Urology. 2003;61:83-8.
  • 22. Miki K, Shimomura T, Yamada H, Kishimoto K, Ohishi Y, Harada J, Egawa S. Percutaneous cryoablation of renal cell carcinoma guided by horizontal open magnetic resonance imaging. Int J Urol. 2006;13:880-4.
  • 23. Moon TD, Lee FT Jr, Hedican SP, Lowry P, Nakada SY. Laparoscopic cryoablation under sonographic guidance for the treatment of small renal tumors. J Endourol. 2004;18:436-40.
  • 24. Polascik TJ, Nosnik I, Mayes JM, Mouraviev V. Short term clinical outcome after laparoscopic cryoablation of the renal tumor < 3.5 cm. Technol Cancer Res Treat. 2007;6:621-4.
  • 25. Nadler RB, Kim SC, Rubenstein JN, Yap RL, Campbell SC, User HM. Laparoscopic renal cryosurgery: the Northwestern experience. J Urol. 2003;170:1121-5.
  • 26. Rukstalis DB, Khorsandi M, Garcia FU, Hoenig DM, Cohen JK. Clinical experience with open renal cryoablation. Urology. 2001;57:34-9.
  • 27. Shingleton WB, Sewell PE Jr. Percutaneous renal tumor cryoablation with magnetic resonance imaging guidance. J Urol. 2001;165:773-6.
  • 28. Silverman SG, Tuncali K, vanSonnenberg E, Morrison PR, Shankar S, Ramaiya N, Richie JP. Renal tumors: MR imaging-guided percutaneous cryotherapy--initial experience in 23 patients. Radiology. 2005;236:716-24.
  • 29. Weld KJ, Figenshau RS, Venkatesh R, Bhayani SB, Ames CD, Clayman RV, Landman J. Laparoscopic cryoablation for small renal masses: three-year follow-up. Urology. 2007;69:448-51.
  • 30. Wyler SF, Sulser T, Ruszat R, Weltzien B, Forster TH, Provenzano M, Gasser TC, Bachmann A. Intermediate-term results of retroperitoneoscopy-assisted cryotherapy for small renal tumours using multiple ultrathin cryoprobes. Eur Urol. 2007;51:971-9.
  • 31. Breen DJ, Rutherford EE, Stedman B, Roy-Choudhury SH, Cast JE, Hayes MC, Smart CJ. Management of renal tumors by image-guided radiofrequency ablation: experience in 105 tumors. Cardiovasc Intervent Radiol. 2007;3:936-42.
  • 32. DiMarco DS, Farrell MA, Zincke H, Gettman MT, Charbonneau JW, Chow GK. Radiofrequency ablation of renal tumors. J Urol. 2004;171(Suppl):129, A482
  • 33. Gervais DA, McGovern FJ, Arellano RS, McDougal WS, Mueller PR. Renal cell carcinoma: clinical experience and technical success with radio-frequency ablation of 42 tumors. Radiology. 2003;226:417-24.
  • 34. Klingler HC, Marberger M, Mauermann J, Remzi M, Susani M. 'Skipping' is still a problem with radiofrequency ablation of small renal tumours. BJU Int. 2007;99:998-1001.
  • 35. Lewin JS, Nour SG, Connell CF, Sulman A, Duerk JL, Resnick MI, Haaga JR. Phase II clinical trial of interactive MR imaging-guided interstitial radiofrequency thermal ablation of primary kidney tumors: initial experience. Radiology. 2004;232:835-45.
  • 36. Memarsadeghi M, Schmook T, Remzi M, Weber M, Pötscher G, Lammer J, Kettenbach J. Percutaneous radiofrequency ablation of renal tumors: midterm results in 16 patients. Eur J Radiol. 2006;59:183-9.
  • 37. Pavlovich CP, Walther M, Choyke PL, Pautler SE, Chang R, Linehan WM, Wood BJ. Percutaneous radio frequency ablation of small renal tumors: initial results. J Urol. 2002;167:10-5.
  • 38. Raman JD, Thomas J, Lucas SM, Bensalah K, Lotan Y, Trimmer C, Cadeddu JA. Radiofrequency ablation for T1a tumors in a solitary kidney: promising intermediate oncologic and renal function outcomes. Can J Urol. 2008;15:3980-5.
  • 39. Watanabe F, Kawasaki T, Hotaka Y, Ishiyama M, Fuwa S, Nagata M, Saida Y. Radiofrequency ablation for the treatment of renal cell carcinoma: initial experience. Radiat Med. 2008;26:1-5.
  • 40. Wingo MS, Leveillee RJ. Central and deep renal tumors can be effectively ablated: radiofrequency ablation outcomes with fiberoptic peripheral temperature monitoring. J Endourol. 2008;22:1261-7.
  • 41. Zagoria RJ, Traver MA, Werle DM, Perini M, Hayasaka S, Clark PE. Oncologic efficacy of CT-guided percutaneous radiofrequency ablation of renal cell carcinomas. Am J Roentgenol. 2007;189:429-36.
  • 42. Cochrane AL. 1931-1971: a critical review, with particular reference to the medical profession. In: Medicines for the year 2000. London: Office of Health Economics; 1979. p.1-11.
  • Correspondence:
    Regina El Dib
    Departamento de Anestesiologia. Faculdade de Medicina de Botucatu-UNESP
    Distrito de Rubião Júnior, s/n
    18618-190 Botucatu - SP Brasil
    Tel.: (55-11)99999-6647
  • 1
    Research performed at St. Joseph's Healthcare, McMaster Institute of Urology, McMaster University, Canada.
  • Publication Dates

    • Publication in this collection
      04 Dec 2013
    • Date of issue
      Dec 2013

    History

    • Received
      12 Aug 2013
    • Accepted
      13 Nov 2013
    • Reviewed
      14 Oct 2013
    Sociedade Brasileira para o Desenvolvimento da Pesquisa em Cirurgia https://actacirbras.com.br/ - São Paulo - SP - Brazil
    E-mail: actacirbras@gmail.com