Hierarchy of evidence referring to the central nervous system in a high-impact radiation oncology journal: a 10-year assessment. Descriptive critical appraisal study

ABSTRACT CONTEXT AND OBJECTIVE: To the best of our knowledge, there has been no systematic assessment of the classification of scientific production within the scope of radiation oncology relating to central nervous system tumors. The aim of this study was to systematically assess the status of evidence relating to the central nervous system and to evaluate the geographic origins and major content of these published data. DESIGN AND SETTING: Descriptive critical appraisal study conducted at a private hospital in São Paulo, Brazil. METHODS: We evaluated all of the central nervous system studies published in the journal Radiotherapy & Oncology between 2003 and 2012. The studies identified were classified according to their methodological design and level of evidence. Information regarding the geographical location of the study, the institutions and authors involved in the publication, main condition or disease investigated and time of publication was also obtained. RESULTS: We identified 3,004 studies published over the 10-year period. Of these, 125 (4.2%) were considered eligible, and 66% of them were case series. Systematic reviews and randomized clinical trials accounted for approximately 10% of all the published papers. We observed an increase in high-quality evidence and a decrease in low-quality published papers over this period (P = 0.036). The inter-rater reliability demonstrated significant agreement between observers in terms of the level of evidence. CONCLUSIONS: Increases in high-level evidence and in the total number of central nervous system papers were clearly demonstrated, although the overall number of such studies remained relatively small.


INTRODUCTION
Evidence-based medicine has become essential to clinical and research actions since it was formally proposed in 1990. 1 The importance of evidence-based medicine concepts was highlighted in an article published in the British Medical Journal in 2007, in which the editors described the emergence of evidence-based medicine as one of the 15 most important milestones since the foundation of the British Medical Journal (1870). 2,3Henceforth, critical evaluation of evidence has become an important tool for assessing research quality and progress.Clinical research can be classified into levels of evidence, which are based on evaluating and interpreting evidence.The level of evidence is closely related to the likelihood that a piece of research will produce valid and reliable results.
Radiotherapy is no different in this regard.The pursuit of the best evidence is changing and is beginning to follow the trends reported in the 1990s. 4As an example, conducting a quick Medline search associating "randomized trials" and "radiation oncology", 211, 144, 27 and 5 studies for the years 2012, 1996, 1981 and 1970 are identified, respectively.This finding demonstrates the evolution and intensification of research applied to radiotherapy, with a 40-fold increase in publications, over this time period.Moreover, high-quality studies play a fundamental role in medical journals.From a broader perspective, the methodological quality and level of evidence of published articles are important determinants of how many times an article is cited, which therefore affects the impact factor of that journal and can also play a major role in the clinical transfer of knowledge. 5,6This has become an essential aspect of the evaluation of scientific journals. 6 2003, prominent journals began to use evidence hierarchies to rank the published studies. 7,8As a result, evidence-based medicine concepts were adopted by the conferences and symposia of the main specialties.Following this paradigm, great efforts have been applied within radiation oncology to follow the evidencebased medicine trend.Nevertheless, to date, there has been no systematic assessment of the quality of scientific production in several areas of radiation oncology. 4,5

OBJECTIVE
The aim of this study was to identify central nervous system studies published in Radiotherapy & Oncology (Elsevier Ireland) over the last decade (2003-2012), classify the type of study and evidence levels according to evidence-based medicine criteria and observe the inter-rater agreement in the classification of the studies included.

METHODS
Using electronic databases, two researchers independently evaluated all studies published in all editions of the major European radiation oncology-specific journal (Radiotherapy & Oncology, Elsevier Ireland, accessed at http://www.thegreenjournal.com) between 2003 and 2012.This journal was chosen because it is important in the field of radiation oncology field; it is indexed in at least one major international database; and it is, so far, the radiation oncology journal with the highest impact factor.We conducted a descriptive critical appraisal study.
Studies in this journal were initially screened based on their titles and were classified as eligible, potentially eligible or not eligible.The sole inclusion criterion was that they needed to be clinical studies relating to the central nervous system that were published between 2003 and 2012.Thus, presence of the following topics in the title counted for this initial screening: metastatic central nervous system; low-grade glioma; high-grade glioma; pediatrics and central nervous system (medulloblastoma, ependymoma or astrocytoma); central nervous system lymphoma; benign tumors (meningioma, schwannoma or arterial venous malformations); spinal cord, orbital and skull-base tumors; and experimental central nervous system studies.After this initial screening, the selected studies (eligible and potentially eligible) were first reassessed using their abstracts and then by using their full texts.All studies relating only to dosimetry were excluded.
A third evaluator resolved any disagreements.
The studies thus identified were assessed by two examiners and were subsequently classified according to the methodological design: 1. systematic reviews; 2. randomized or non-randomized clinical trials; 3. cohort studies; 4. case-control studies; 5. case series; and 6. basic science studies.The studies were also classified according to their level of evidence using the guidelines of the Oxford Centre for Evidence-based Medicine: systematic reviews of randomized clinical trials, level I; randomized clinical trials, level II; cohort and case-control studies, level III; case series, level IV; and narrative reviews and other designs, level V.This is a widely used classification method that has been adapted for use within the radiation oncology literature. 9This categorization was done after reading the full texts of the eligible studies.
For all the studies ultimately included, we also obtained information regarding the geographical location at which the study was performed, institutions/departments and authors involved in the publication, main condition studied, main disease investigated and time of publication.We also examined the productivity relating to radiotherapy for the central nervous system in each department over the 10-year period covered by this analysis.The following parameters were stratified using the following parameters: time of publication (period

Statistical analysis
The assumption of normal distribution in the sample was assessed using the Kolmogorov-Smirnov test.Cohen's kappa test was used to assess reliability and to evaluate the internal consistency of the inter-rater classifications.The magnitude of agreement was determined based on the proposal of Landis and Koch: I. < 0, no agreement; II.0 to 0.20, slight agreement; III.0.21 to 0.40, fair agreement; IV. 0.41 to 0.60, moderate agreement; V. 0.61 to 0.80, significant agreement; and VI.0.81 to 1.00 almost perfect agreement. 10,11The chi-square test was used to evaluate the proportions of papers at evidence levels I, II and III between the two periods.We considered P-values from two-sided tests < 0.05 to be statistically significant.

RESULTS
We identified 3,004 studies published over the 10-year period evaluated.Of these, 135 were initially selected (central nervous system disease), from which 10 were then excluded.Thus, 125 studies (4.2%) were considered eligible and were included in this analysis (Figure 1).There was an average of 300.4 publications per year during the study period (which included an average of 13.5 publications per year relating to the central nervous system).We noted an absolute increase in the number of published papers of 33% overall and 41% in relation to the central nervous system, from period 1 to period 2 (Table 1).
Table 2 shows the distribution of the central nervous system studies according to the geographical location at which they were conducted.European studies accounted for more than 60% of the published data over the entire period and, in comparison with the rest of the world, this difference was statistically significant (P = 0.0306).
Stratification according to disease classification showed that the majority (74%) of the studies were related to central nervous system metastasis, followed by high-grade gliomas and benign tumors (Table 2).
We also noted that the average numbers of authors and departments involved in the studies were 6.86 and 3.29, respectively; 67% of the first authors were radiation oncologists (Table 2)  Among these studies, 66.5% were case series (prospective and retrospective; number published (n) = 81 articles); 8% were prospective controlled studies (not randomized) (n = 10); 1% were cohort studies (n = 1); 1% were case-control studies (n = 1); 6% were cross-sectional studies (n = 8); and 3% were review articles (n = 5).Systematic reviews (n = 5) and randomized clinical trials (n = 7) accounted for approximately 10% of all the published papers.Other studies, which included case reports, were responsible for 5% of the publications (n = 7).In analyzing the level of evidence according to year, we observed that there were greater numbers of published papers with evidence levels I, II and II and lower numbers with evidence levels IV and V in period 2 (2008-2012) than in period 1 (2003-2007) (P = 0.036). 9e Scientific Journal Rankings index showed average values of 1.38 and 1.89 for periods 1 and 2, respectively.This higher index for period 2 presented a tendency towards a statistically significant difference in relation to period 1 (P = 0.0528).The inter-rater reliability for the classification of study type according to the kappa statistic demonstrated significant agreement between the observers (kappa = 0.69).

DISCUSSION
In this 10-year single-journal analysis, we found that the studies published within the scope of the central nervous system increased in quality and number, although significant representation in the journal Radiotherapy & Oncology is still lacking (< 4.5% of published papers).We also found that case series (retrospective and prospective) represented the majority of central nervous system papers published in this journal.Furthermore, level of evidence was found to be a reproducible tool, and secondary tumor (metastasis) research was well represented in this journal.The major strength of our data is that they represent, to the best of our knowledge, an original study with a representative period of evaluation in a single journal.
Moreover, this analysis was based on formal and systematic data-gathering and evaluation, and our results present sequential assessment, including formal statistical analysis and interreliability analysis based on Cohen's kappa test.
In the United States, according to the national database, primary central nervous system tumors account for less than 3% of all diagnosed neoplasms. 12Similarly, primary and metastatic central nervous system tumors, together with benign central nervous system diseases, represent vast opportunities for treatment improvements, with implementation of new markers and prognostic factors.This is an important field for radiotherapy research, including newer approaches using stereotactic radiosurgery.In the field of central nervous system tumors, radiotherapy plays a major role in the management of almost all types of malignant brain tumors.Moreover, a high level of evidence can play a major role in treatment decisions.
Among patients diagnosed with cancer (all anatomical sites), approximately 54% of all of them will require some radiation treatment during their lifetime, and 12% will require re-irradiation. 13Based on evidence-based guidelines, the central nervous   1. Frequencies of the hierarchy of evidence, grouped according to the period and region of origin system shows a highly recommended overall optimal radiotherapy utilization rate (approximately 92-93%). 13In a comprehensive analysis in which the objective was to estimate the ideal proportion of patients with newly diagnosed central nervous system neoplasms who could benefit from external-beam radiotherapy, most of the recommendations were based on evidence level III. 146][17][18] Based on our analysis, most of the studies presented low-level evidence, such as prospective and retrospective case series (66%).Higher evidence levels such as systematic reviews and randomized clinical trials represented only approximately 10% of all central nervous system published papers.In comparison with case series, randomized trials involve greater numbers of ethical issues and higher costs.The incidence of primary central nervous system tumors, combined with the tendency to treat them in large specialized centers, may explain the findings of this study.It also needs to be taken into consideration that the journal Radiotherapy & Oncology accepts physical contributions, dosimetry studies, molecular biology assays and other types of non-formal clinical publications.In addition to the important role of such articles with regard to development of radiation oncology, they are classified as presenting evidence levels IV or V according to the Oxford Criteria. 9Nevertheless, there was an increase in the evidence level of published central nervous system articles (in our data) over the years, particularly in the more recent period.
In a manner similar to our study, Yarascavitch et al.

(at
McMaster University, Hamilton, Canada) quantified the level of evidence in 660 eligible articles in the neurosurgical literature in order to determine the changes over time and the predictive factors for higher-level evidence. 19Levels I and II accounted for only 1 in 10 neurosurgical clinical papers in top journals, and papers with larger sample sizes were significantly associated with higher level of evidence.These authors concluded that there is a need for better evidence in papers published within this field and that patient management and the publication of prospective studies may be improved by education and the adoption of level of evidence.[22][23][24] Similarly, it is important to note that level of evidence can be correlated with journal impact factor and that increasing numbers of studies with high-level evidence have been observed in palliative and orthopedic settings. 22,25,26 The limitations of the current analysis lie in the fact that central nervous system articles may not be well represented in the journal chosen for analysis here because other radiation oncology journals that were not included in the electronic search also publish articles relating to the central nervous system.In addition, specific journals and higher-impact journals may account for significant numbers of published papers relating to the central nervous system.These were not assessed in the present analysis but may have had an impact on the data presented.Furthermore, a wider search of the literature might lead to a more optimistic outlook regarding the proportion of high-quality studies.
In this study, a training workshop on manuscript classification was conducted initially.The Oxford system of levels of evidence seemed to be a feasible instrument for evaluating studies, with a significant degree of consistency. 9These findings and those in other studies emphasize the importance of specific training for individuals who are responsible for determinations relating to the quality of evidence. 28Finally, our study represents a possible landmark for future studies and other evidence-based assessments on the central nervous system within the field of radiation oncology research.Moreover, the present study may result in new research opportunities, such as assessment of the internal and external validity of other study features and evaluations on high-impact and specialized journals.In particular, it would be interesting to evaluate how radiation oncologists manage and comprehend evidence-based medicine.

1 :
2003-2007; and period 2: 2008-2012); geographical location; level of evidence; and Scientific Journal Rankings index (www.scimagojr.com).This index measures the impact that a single published paper has, and hence the scientific influence of an average article in a journal.It thus expresses the extent to which an average journal article is central to the global scientific discussion.
, and 85% of the first authors only had a single article published in Radiotherapy & Oncology as the first author.Twelve institutions (University Hospital of Heidelberg, Institute of Southern Switzerland, University Hospital Zurich, University Hospital Groningen, Tata Memorial Hospital, University of Wisconsin Medical School, McGill University Health Center, ''S.Maria" Hospital, VU University Medical Center, St. Jude Children's Research Hospital, Ludwig-Maximilians-University Munich and San Raffaele Scientific Institute) (13.3% of the total)were responsible for 40 published papers (32% of the total) and 78 institutions were responsible for the other 68% of publications over this 10-year period.

Table 2 .
Central nervous system papers published, according to region, diagnosis and first author over the 10-year period This finding emphasizes that there is an urgent need to expand the data relating to evidence-based oncology.Regarding the origin of the articles included in this study, in comparison with the rest of the world, Europe showed the largest number of published papers within the field of the central nervous system (60%).Because during their four years of training.We consider that this sort of analysis is important for future studies and for knowledge of referral centers for future postgraduate training.