Critical analysis of scientific publications of the Revista Brasileira de Ortopedia from 2006 to 2010

Abstracts

OBJECTIVE:

Profile analysis of articles from scientific journals is rare in our country. The aim of this study was to perform an analysis of publications of the Revista Brasileira de Ortopedia (RBO), to specify the designs of the studies and their level of evidence.

METHODS:

All articles published in RBO from January 2006 to December 2010 were classified according to the design of the study. The clinical studies were further stratified according to the level of evidence, in agreement with the norm of the journal. The studies classified as randomized and controlled clinical trials (RCTs) had their quality assessed by the system proposed by Jadad.

RESULTS:

In this period, there were 376 articles published in RBO. Clinical studies represented most of the papers, with 60.64% of the total. Case series represented 61.4% of the clinical studies. Thirteen RCTs were published, accounting for 3.46% of the total, and 5.7% of the clinical studies. The analysis of the quality of the RCTs showed that 5 (38.46%) were considered high quality, while 8 (61.54%) were of low quality. Among the studies in which the level of evidence does not apply (non-clinical), non-systematic reviews (46 articles) and basic research (40 articles) have prevailed, representing 12.23% and 10.64% respectively of the total.

CONCLUSIONS:

Case series were the most prevalent (37.23%) studies published in RBO between 2006 and 2010, while RCTs accounted for 3.46% of the articles. The majority of RCTs (61.54%) were considered low quality, and only 1.32% of the clinical studies were classified as level I evidence.

Methodology ; Bibliometrics ; Periodicals ; Orthopedics

OBJETIVO

: Fazer uma análise das publicações da Revista Brasileira de Ortopedia (RBO), visando a avaliar os tipos e o nível de evidência científica dos estudos.

MÉTODOS

: Todos os artigos científicos publicados na RBO de janeiro de 2006 a dezembro de 2010 foram classificados de acordo com o tipo de estudo. Os trabalhos clínicos foram ainda estratificados conforme o nível de evidência clínica, de acordo com a norma editorial da revista. Os estudos classificados como ensaios clínicos controlados e randomizados (ECCR) tiveram sua qualidade avaliada pelo sistema proposto por Jadad.

RESULTADOS

: Nesse período, foram publicados na RBO 376 artigos. Os estudos clínicos corresponderam à maioria das publicações, com 60,64% do total, sendo que desses 61,4% eram séries de casos. Foram publicados 13 ECCR, correspondendo a 3,46% do total dos artigos publicados e a 5,7% dos estudos clínicos. A análise da qualidade dos ECCR mostrou que cinco (38,46%) foram considerados de alta qualidade, enquanto oito (61,54%) foram de baixa qualidade. Dentre os estudos nos quais o nível de evidência clínica não se aplica (não clínicos) prevaleceram os artigos de revisão não sistemática (46) e os de pesquisa básica (40), que representaram, respectivamente, 12,23% e 10,64% do total de publicações.

CONCLUSÕES

: As publicações do tipo série de casos foram as mais prevalentes (37,23%), enquanto os ECCR corresponderam a 3,46% do total. A maioria dos ECCR (61,54%) foi considerada de baixa qualidade. Apenas 1,32% dos estudos clínicos foi classificado como nível I de evidência.

Metodologia ; Bibliometria ; Publicações periódicas ; Ortopedia

Introduction

Evidence-based medicine has gained prominence in current medical literature. This clinical practice is aimed at combining clinical experience with statistical results.¹ Using evidence-based medicine, physicians can better understand treatment outcomes and use the information to better treat patients and reduce costs.²

Randomized controlled clinical trials (RCTs) provide the most evidence of all clinical research models. However, some medical areas have characteristics that make it difficult to conduct RCTs with maximum accuracy.³ Orthopedics is a surgical discipline that presents its own obstacles for carrying out high-quality clinical trials.^4,5 Therefore, the majority of research in this field is observational and produces data that may be inconclusive and bring into doubt its credibility.^6,7

Assessment papers published in scientific journals are rare in our field, but they are useful tools to measure scientific productivity in Brazil.

This study aimed to perform a quantitative analysis of the publications in the Revista Brasileira de Ortopedia (RBO) from 2006 to 2010 to evaluate the types of study designs and levels of scientific evidence produced by the studies.

Material and Methods

All scientific papers published in RBO from January 2006 to December 2010 were evaluated. Papers were classified into 14 categories based on the type of study. Clinical papers were divided into the following seven categories: technique description, case report, case series, cohort, RCT and systematic review. Non-clinical papers were divided into seven other categories: update papers or non-systematic reviews, accuracy analysis of diagnostic methods, epidemiological studies, basic research, biomechanical testing, anatomic studies and others.

Clinical studies were further stratified according to the level of clinical evidence produced in the study; studies were classified from I to V as per the magazine's editorial rule.⁸

Three participants in this study classified papers independently. If there were any disagreements regarding papers, it was re-reviewed as a group and the evaluation given by at least two reviewers was considered to be the final classification.

The quality of the studies classified as RCTs was evaluated according to the system developed by Jadad et al.⁹ Studies are assessed based on whether they are randomized, whether they are double-blind, whether they describe reasons for patient dropout, and whether the randomization and blinding are appropriate. Good quality papers are those that have positive responses for three or more of the above criteria. When the paper did not specify the items of the Jadad system in its methodology or when these items were not implicit in the text, they were considered to be missing.

Statistical Analysis

A descriptive analysis (absolute numbers and percentages) of the results was performed using Excel® (Microsoft).

Results

During the period studied, 376 papers were published in RBO. The distribution according to the type of study is shown in Table 1.

Table 1
Distribution of types of studies

Type of Study	Definition	n	%
Case series	Description of intervention in a series of subjects without bias control	140	37.23
Non-systematic review	Non-systematic review of a series of articles	46	12.23
Basic research	Analysis of animal, tissue, cellular or molecular models	40	10,64
Case report	Description of a rare condition or unexpected intervention result	33	8.78
Cohort	Outcome evaluation regarding exposure	31	8,24
Anatomic study	Description of anatomic characteristics	20	5,32
Biomechanical study	Evaluation of the resistance of tissues	16	4,26
Randomized and controlled clinical study	Randomized comparative experimental study	13	3,46
Accuracy of the diagnostic method	Evaluation of diagnostic techniques and their reproducibility	12	3,19
Epidemiological study	Description of the characteristics of a population	12	3,19
Case-control	Search for causal factors for certain outcomes	5	1,33
Technique description	New technique description	3	0,80
Systematic review	Systematic review of a series of articles	3	0,80
Others	Do not fit into the previous categories	2	0,53
Total		376	100,00

Clinical studies accounted for 60.64% of all reports. Case series were also prevalent and represented 37.23% of all publications and 61.40% of published clinical research. Thirteen RCTs were published, comprising 3.46% of all papers and 5.7% of clinical studies that were published. The distribution of the different types of clinical studies and their evidence levels can be observed in Tables 2 and 3, respectively.

Table 2
Distribution of types of clinical studies.

Clinical studies	n	%
Systematic review	3	1,32
Randomized and controlled clinical trial	13	5,70
Case-control	5	2,19
Cohort	31	13,60
Case series	140	61,40
Case report	33	14,47
Technique description	3	1,32
Total	228	100,00

Table 3
Distribution of clinical studies regarding the level of evidence.

Level of evidence	n	%
I	3	1,32%
II	12	5,26%
III	37	16,23%
IV	140	61,40%
V	36	15,79%
Total	228	100,00%

Quality analysis of RCTs showed that five trials (38.46%) were high quality, while eight (61.54%) were low quality. Based on Jadad's classification, the most-commonly omitted criteria were appropriate blinding and the description of the study as double-blind. Both were present in only 23.08% of the published studies (Table 4). Furthermore, only three RCTs were considered to be studies with level I evidence.

Table 4
Evaluation of the randomized and controlled clinical trial according to the Jadad classification.

	n	%
Randomized	13	100,00%
Double-blind	3	23,08%
Description of patient dropout	7	53,85%
Appropriate randomization	6	46,15%
Appropriate blinding	3	23,08%
High-quality papers	5	38,46%
Low-quality papers	8	61,54%
Total of papers	13	100,00%

Among the studies in which the level of clinical evidence is not applicable, non-systematic review papers (46) and basic research papers (40) were prevalent, which represent 12.23% and 10.64% of the total number of publications, respectively. Table 5 shows the detailed distribution of non-clinical papers.

Table 5
Distribution of non-clinical studies.

Non-clinical studies	n	%
Non-systematic review	46	31,08%
Basic research	40	27,03%
Anatomic study	20	13,51%
Biomechanical study	16	10,81%
Accuracy of the diagnostic method	12	8,11%
Epidemiological study	12	8,11%
Others	2	1,35%
Total	148	100,00%

Discussion

The percentage of studies in the RBO from January 2006 to December 2010 classified as having level I clinical evidence is lower than that found in international literature. Hanzlik et al.¹⁰, who analyzed papers published in the Journal of Bone and Joint Surgery (American Volume) in 2005, found that 21% of all studies had level I clinical evidence. Obremskey et al.¹¹ evaluated publications from eight scientific magazines in the first semester of 2003 and found that 11.3% were level I studies. Cashin et al.¹², who studied publications from the ^{Journal of Pediatric Orthopaedics}, found that level I studies made up 3% of all papers. For the period assessed, we observed that 1.32% of published studies had this level of evidence.

Furthermore, 5.26% of the studies were classified as having level II evidence, a rate comparable to that found by Cashin et al.¹² (5%) but lower than that found by Hanzlik et al.¹⁰ and Obremskey et al.¹¹ (15% and 20.7%, respectively). The percentage of papers with level III evidence (16.23%) was higher than that found by Obremskey et al.¹¹ (9.9%), similar to that found by Hanzlik et al.10 (16%), and lower than that observed by Cashin et al.¹² (24.1%). If we sum the level-II and level-III studies, the results from the RBO (21.49%) are lower than those found in the literature, where they are approximately 30%.^10-12

The two studies used here to compare the levels of scientific evidence used slightly different classifications, in which the case reports were classified together with series cases as level IV.^11,12 Summing these two types of studies, we observed that the percentage found in RBO (77.19%) is higher than that found by Cashin et al.¹² (58%) and Obremskey et al.¹¹ (58.1%). Hanzlik et al.,¹⁰ who, like us, graded case reports as having level V evidence, did not find occurrences of this type of study in the Journal of Bone and Joint Surgery (American Volume) in 2005. The percentage of case reports in RBO is 15.79%. When appropriately elaborated, case series play a significant role in surgical specialties and are an important to generating hypotheses for high-level studies,¹³ but they must be carefully considered when used as a guide for decision-making. Case reports, on the other hand, are important in describing rare diseases or unusual procedural complications, but they present the lowest level of clinical evidence among studies.

We hypothesize that the lower rates of studies with better levels of clinical evidence in the RBO can be at least partially explained by the fact that Brazilian authors search for journals with higher impact factors. The two most important national orthopedic magazines, RBO and Acta Ortopédica Brasileira (AOB), are not indexed by Pubmed and the articles published in these two magazines reach considerably fewer readers. This produces a bias, as Brazilian authors try to publish their best papers in international magazines.

Regarding the quality of RCTs, most fail to appropriately describe their methodology. According to the Jadad scale, only 38.46% were considered to be high quality. Appropriate blinding and a description of the study as double-blind were present in only 23.08% of the studies. Similar findings have also been reported in international literature. Dulai et al.¹⁴, who assessed studies by means of the Detsky Scale, noted that only 19% of RCTs satisfied the threshold for a satisfactory level of quality. Failures in blinding were also described by Poolman et al.¹⁵, who found that the description of the appropriate blinding of patients and treatment providers were only present in 19% and 6% of RCTs, respectively. Blinding is important to minimize bias and obtain a high-quality study and presents additional difficulties in trials involving surgical procedures, as in the case of orthopedics.⁴ Depending on the type of study, the appropriate blinding of the surgeon or the patient is not possible.⁴

During the evaluation of the studies, when raters did not agree upon their classifications, they performed a group review. Even after a thorough analysis, some papers were difficult to classify. Description and methodology failures were common, possibly due to the author's lack of knowledge regarding some methodological principles. The type of study and its level of evidence are needed when submitting an article to RBO, although this information is not published. We believe that the publication of this information, similar to international publications, would be interesting for readers. However, as many orthopedists are not familiar with the classification system and also because of the risk of evaluation bias (the author may have a tendency to overvalue the scientific level of his/her study), this information should be reviewed by the editorial board of the magazine.

De Moraes et al.¹⁶ assessed 22 RCTs published in RBO and AOB from 2000 to 2009. The authors did not find significant differences in quality or types of publications when comparing the first five-year period with the second one. Hanzlik et al.¹⁰, who assessed the percentage of studies with level I evidence in the Journal of Bone and Joint Surgery (American Volume), showed a significant evolution in the types of orthopedics publications over time. In 1975, orthopedics papers comprised only 4% of all publications in the journal, but by 2005, this number had increased to 21% of all publications in the journal. We believe that the lack of changes found by De Moraes et al.¹⁶ is because a shorter period was assessed and that the analysis was performed in two periods of five years (2000-2004 and 2005-2009). In our study, we did not analyze historical trends because we considered five years to be too short to do so.

The limitations of this work are that only one magazine (RBO) was analyzed over a shorter period compared to other studies.^10,16 However, unlike researchers who only assessed clinical studies,^10-12,16 we provided information about the distribution of different types of articles published in the magazine, In addition, we provided data and information about all Jadad classifications of RCTs, which is unlike the study by de Moraes et al.,¹⁶ who only provided mean values of the scoring system.

Critical analyses of the publications in journals are essential to obtain a better understanding of the national scientific scenario. Furthermore, it these analyses are useful to promote changes in editorial policies and can improve the quality of the studies.

Conclusion

Case series were the most prevalent (37.23% of the total number of publications) studies, while RCTs accounted for 3.46% of the total number of publications. 61.54% of RCTs were considered to be low quality, and only 1.32% of the clinical studies were classified as having level I evidence.

REFERENCES

• ¹
  Sackett D, Rosenberg W, Gray J, Haynes R, Richardson W. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312(13):71-2.
• ²
  Carter MJ. Evidence-based medicine: an overview of key concepts. Ostomy Wound Management. 2010;56(4):68-85.
• ³
  Carr A. Evidence-based orthopaedic surgery: what type of research will best improve clinical practice? J Bone Joint Surg Br. 2005;87(12):1593-4.
• ⁴
  Malavolta EA, Demange MK, Gobbi RG, Imamura M, Fregni F. Ensaios clínicos controlados e randomizados na Ortopedia: dificuldades e limitações. Rev Bras Ortop. 2011;46(4):452-9.
• ⁵
  Chaudhry H, Mundi R, Singh I, Einhorn TA, Bhandari M. How good is the orthopaedic literature? Indian J Orthop. 2008;42(2):144-9.
• ⁶
  Hartz A, Marsh JL. Methodologic issues in observational studies. Clin Orthop Relat Res. 2003;413:33-42.
• ⁷
  Parsons NR, Hiskens R, Price CL, Achten J, Costa ML. A systematic survey of the quality of research reporting in general orthopaedic journals. J Bone Joint Surg Br. 2011;93(9):1154-9.
• ⁸
  Camanho GL. Nivel de evidência. [Editorial]. Rev Bras Ortop. 2009;44(6).
• ⁹
  Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Controlled clinical trials. 1996;17(1):1-12.
• ¹⁰
  Hanzlik S, Mahabir RC, Baynosa RC, Khiabani KT. Levels of evidence in research published in The Journal of Bone and Joint Surgery (American Volume) over the last thirty years. J Bone Joint Surg Am. 2009;91(2):425-8.
• ¹¹
  Obremskey WT, Pappas N, Attallah-Wasif E, Tornetta P, Bhandari M. Level of evidence in orthopaedic journals. J Bone Joint Surg Am. 2005;87(12):2632-8.
• ¹²
  Cashin MS, Kelley SP, Douziech JR, Varghese RA, Hamilton QP, Mulpuri K. The levels of evidence in pediatric orthopaedic journals: where are we now? J Pediatr Orthop. 2011;31(6):721-5.
• ¹³
  Brighton B, Bhandari M, Tornetta P, Felson DT. Hierarchy of evidence: from case reports to randomized controlled trials. Clin Orthop Relat Res. 2003;(413):19-24.
• ¹⁴
  Dulai SK, Epi M, Slobogean ÞBLT, Beauchamp ÞRD, Mulpuri K, Ortho MS, et al. A Quality Assessment of Randomized Clinical Trials in Pediatric Orthopaedics. Clinical Trials. 2007;27(5):573-81.
• ¹⁵
  Poolman RW, Struijs PA, Krips R, Sierevelt IN, Lutz KH, Bhandari M. Does a "Level I Evidence" rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC medical research methodology. BMC Med Res Methodol. 2006;6:44.
• ¹⁶
  De Moraes VY, Moreira CD, Jun M, Tamaoki S. Ensaios clínicos randomizados na ortopedia e traumatologia: avaliação sistemática da evidência nacional. Trauma. 2010;45(6):601-5.

Publication Dates

• Publication in this collection
  May/June 2013

History

• Received
  31 May 2012
• Accepted
  20 June 2012