Hierarchy of evidence relating to hand surgery in Brazilian orthopedic journals

A hierarquia das evidências em cirurgia da mão nas revistas ortopédicas nacionais

ORIGINAL ARTICLE

Hierarchy of evidence relating to hand surgery in Brazilian orthopedic journals

A hierarquia das evidências em cirurgia da mão nas revistas ortopédicas nacionais

Vinícius Ynoe de Moraes^I; João Carlos Belloti^II; Fábio Ynoe de Moraes^III; José Antonio Galbiatti^IV; Evandro Pereira Palácio^V; João Baptista Gomes dos Santos^II; Flávio Faloppa^VI

^IMD. Orthopedic Surgeon, Universidade Federal de São Paulo - Escola Paulista de Medicina (Unifesp-EPM), São Paulo, Brazil

^IIMD, PhD. Professor in the Department of Orthopedics and Traumatology, Universidade Federal de São Paulo (Unifesp), São Paulo, Brazil

^IIIMedical student, Faculdade Estadual de Medicina de Marília (Famema), Marília, São Paulo, Brazil

^IVMD, PhD. Professor in the Department of Orthopedics and Traumatology, Faculdade Estadual de Medicina de Marília (Famema), Marília, São Paulo, Brazil

^VMD. Orthopedic Surgeon, Faculdade Estadual de Medicina de Marília (Famema), Marília, São Paulo, Brazil

^VIMD, PhD. Chairman of Orthopedics and Traumatology, Universidade Federal de São Paulo (Unifesp), São Paulo, Brazil

^{Address for correspondence}

ABSTRACT

CONTEXT AND OBJECTIVE: There is no systematic assessment of the quality of scientific production in the specialty of hand surgery in our setting. This study aimed to systematically assess the status of evidence generation relating to hand surgery and to evaluate the reproducibility of the classification method based on an evidence pyramid.

DESIGN AND SETTING: Secondary study conducted at Universidade Federal de São Paulo (Unifesp) and Faculdade Estadual de Medicina de Marília (Famema).

METHODS: Two researchers independently conducted an electronic database search for hand surgery studies published between 2000 and 2009 in the two main Brazilian orthopedic journals (Acta Ortopédica Brasileira and Revista Brasileira de Ortopedia). The studies identified were subsequently classified according to methodological design (systematic review of the literature, randomized clinical trial, cohort study, case-control study, case series and other studies) and evidence level (I to V).

RESULTS: A total of 1,150 articles were evaluated, and 83 (7.2%) were included in the final analysis. Studies with evidence level IV (case series) accounted for 41 (49%) of the published papers. Studies with evidence level V (other studies) accounted for 12 (14.5%) of the papers. Only two studies (2.4%) were ranked as level I or II. The inter-rater reproducibility was excellent (k = 0.94).

CONCLUSIONS: Hand surgery articles corresponded to less than one tenth of Brazilian orthopedic production. Studies with evidence level IV were the commonest type. The reproducibility of the classification stratified by evidence level was almost perfect.

Key-words: Orthopedics. Hand. Epidemiologic methods. Research design. Evidence-based medicine

RESUMO

CONTEXTO E OBJETIVO: Não há uma avaliação sistemática da qualidade da produção científica na especialidade de cirurgia da mão em nosso meio. Este estudo objetiva analisar o status da geração de evidências em cirurgia da mão, por meio de avaliação sistematizada, além de avaliar a reprodutibilidade da classificação baseada na pirâmide das evidências.

TIPO DE ESTUDO E LOCAL: Estudo secundário realizado pela Universidade Federal de São Paulo (Unifesp) e Faculdade Estadual de Medicina de Marília (Famema).

MÉTODOS: De forma independente, dois pesquisadores promoveram uma busca eletrônica dos trabalhos envolvendo cirurgia da mão, no período de 2000-2009, para duas principais revistas na literatura ortopédica nacional (Acta Ortopédica Brasileira e Revista Brasileira de Ortopedia). Os trabalhos identificados foram subsequentemente classificados pelo tipo de delineamento metodológico (revisões sistemáticas da literatura, estudos clínicos randomizados, estudos de coorte, estudos caso-controle, séries de casos e outros estudos) e nível de evidência (I a V).

RESULTADOS: Foram avaliados 1.150 artigos, sendo 83 (7,2%) considerados para a análise final. Estudos com nível de evidência IV (séries de caso) correspondem a 41 (49%) das publicações. Estudos nível de evidência V (outros estudos), corresponderam a 12 (14,5%) da amostra. Apenas dois estudos (2,4%) foram identificados como nível I ou II. A reprodutibilidade interobservadores foi excelente (k = 0.94).

CONCLUSÕES: Os artigos de cirurgia da mão correspondem a menos de um décimo da produção ortopédica nacional. Estudos nível de evidência IV foram os mais encontrados na literatura nacional. A classificação estratificada por níveis de evidência apresenta reprodutibilidade quase perfeita.

Palavras-chave: Ortopedia. Mãos. Métodos epidemiológicos. Projetos de pesquisa. Medicina baseada em evidências

INTRODUCTION

The notion of evidence-based medicine (EBM) was introduced in 1991 and has been increasingly accepted by the international scientific community.¹ In 2003, prominent journals began to use evidence hierarchies to rank the published studies. As a result, EBM concepts were adopted by the main specialty conferences and symposia.^2,3 The importance and acceptance of the EBM concepts were measured in an article published in the British Medical Journal (BMJ) in 2007, in which the editors described the emergence of EBM as one of the 15 most important milestones for progress in the practice of medicine since the founding of BMJ (1870) along with other milestones such as the discoveries of antimicrobials, anesthetics and DNA.^4,5

Since then, critical analysis of the orthopedic literature has become indispensable for orthopedists and hand surgeons who require evidence from studies of good methodological quality. In addition, many steps have been taken to disseminate the EBM concepts that apply to the particular characteristics of orthopedics,^6,7 along with critical appraisal of the methodological quality of published studies.^3,8-13

In accordance with this paradigm, efforts have been undertaken within hand surgery to follow the EBM movement.^14-16 However, there is a lack of data on the quality of the evidence produced to date, particularly with regard to literature published within specific countries.

OBJECTIVES

This study aimed to: identify hand surgery studies published over the last decade (2000-2009) in the Brazilian orthopedics journals Acta Ortopédica Brasileira (AOB) and Revista Brasileira de Ortopedia (RBO); classify the types of study and evidence levels according to evidence-based medicine criteria; and observe the inter-rater agreement in the classification of the studies included.

METHODS

Using an electronic database, two researchers independently evaluated all studies published in all editions of the AOB and RBO between 2000 and 2009. These two journals were chosen because they were national journals with an orthopedics scope and journals that were indexed in at least one international bibliographic database. The studies were initially screened based on their titles and were classified as eligible, potentially eligible and not eligible. The initial inclusion criteria included the presence of the following themes in the titles: hand and wrist fractures; peripheral nerve lesions and vascular lesions in the upper limbs; nail bed lesions; brachial plexus lesions; muscle tendon lesions; upper limb skin coverage; microsurgery; upper limb pain syndromes; upper limb congenital malformations; and anatomical and experimental studies. After this initial screening, the eligible and potentially eligible studies were again screened: first using the abstracts and then, the full-text articles. Any disagreements were resolved by a third evaluator.

After screening, the studies were assessed by the two examiners, who subsequently categorized them in terms of study type and study evidence level, in accordance with a widely used classification method that has been adapted to the orthopedic literature.^3,17 The categorization was conducted after reading the full texts of the eligible studies. We took systematic reviews of randomized clinical trials to represent evidence level I; randomized clinical trials, level II; cohort and case-control studies, level III; case series, level IV; and narrative reviews and other designs, level V.^18,19

For all the studies ultimately included, we obtained information regarding the journal (AOB or RBO); geographic location of the study (South, Southeast, or North-Northeast-Midwest); strength of the effect estimate in the study;²⁰ and publication period (2000-2004 or 2005-2009). Case reports and studies that were not primarily clinical studies (e.g. biomechanical, anatomical, histological and molecular studies) were classified as "other studies".

Statistical analysis

The assumption of normal distribution in the sample was assessed using the Kolmogorov-Smirnov test. Cohen's kappa test was used to assess reliability and evaluate the internal consistency of the inter-rater classifications. The interpretation of the agreement magnitude was performed based on the proposal of Landis and Koch:²¹ I. < 0, representing no agreement; II. 0 to 0.20, slight agreement; III. 0.21 to 0.40, fair agreement; IV. 0.41 to 0.60, moderate agreement; V. 0.61 to 0.80, significant agreement; and VI. 0.81 to 1.00, almost perfect agreement. The chi-square test was used to evaluate the proportions between the two periods. We considered P-values < 0.05 to be statistically significant.

RESULTS

We identified 1,150 studies. Of these, 83 (7.2%) were considered eligible for this present study (Graph 1; ^{Appendix 1}). Over half of these studies from both journals were case series (Table 1; Graph 2). The AOB (Table 2) published 21 (25%) of the studies (Table 2). The inter-rater reliability for the classification of study type according to the kappa statistic demonstrated almost perfect agreement (kappa = 0.94). The proportions of hand surgery studies grouped according to journal showed a statistically significant difference, such that a larger proportion of these studies were published in the RBO (Table 2; chi-square test, P = 0.02). The total number of hand surgery papers published decreased between 2000-2004 and 2004-2009 (Table 3; chi-square test, P = 0.01). Among the studies classified as "other," the majority were individual case reports (Graph 3).

DISCUSSION

This analysis on the available literature demonstrated that a large number of hand surgery studies, particularly the case series, presented a low level of methodological evidence. High-impact orthopedic journals have also been shown to be populated with studies presenting low levels of methodological evidence.⁹ The high prevalence of published case series suggests the following hypotheses: these studies are low-cost studies that require little planning and prior knowledge; and they may be part of the routine of a healthcare group.

Therefore, the published studies were conducted as a byproduct of healthcare processes and without the goal of scientific research per se. These practices may be attributed to the lack of a scientific research tradition that promotes adequate methodological quality. Regional aspects appear to be a factor; most of the Brazilian scientific research is concentrated in the South-Southeast axis.

To place our results within the context of other orthopedic journals, we note that a study²² that evaluated articles published in the Journal of Bone and Joint Surgery (an American journal) found a large proportion of evidence level I and II published papers (36%), and that this proportion had increased over the preceding 30 years ago, from a proportion of 9%. Our study found that the frequency of level I and II published papers was 3%, thus suggesting the following hypotheses: 1) Higher-level studies may be more likely to be published in the most widely-indexed international journals; and 2) There is a real deficit of studies of higher methodological quality in Brazil.

The low proportion of hand surgery studies reveals a need to stimulate more scientific research in hand surgery centers with a focus on the pursuit of better evidence. The large numbers of case series and anatomical and biomechanical studies demonstrate that the search for new knowledge is intermittent and focused on surgical practice. It is worrisome that the proportion of studies published over recent years has decreased. However, this finding may demonstrate some degree of scientific maturity among the community regarding studies of higher methodological quality, which take a longer time to complete and publish.

One limitation of the present study is the possibility that hand surgery articles are not well represented in the two journals chosen for this investigation, because other Brazilian journals that were not included in the electronic search may also have published articles on hand surgery. In addition, we did not search the international literature for Brazilian authors; such a search might lead to a more optimistic outlook regarding the proportion of high-quality studies. A previous evaluation on the reliability of the study type classification demonstrated that the evidence pyramid developed by Sackett et al.¹⁹ seemed to be a feasible instrument for evaluating studies, with adequate external validity.

CONCLUSIONS

Hand surgery studies comprise less than one-tenth of Brazilian orthopedic research publish between 2000 and 2009 Among these studies, half of them present evidence level IV (case series). In our analysis, the inter-rater reliability of the evidence level classification was almost perfect.

Date of first submission: July 8, 2010

Last received: December 16, 2010

Accepted: December 17, 2010

Sources of funding: None.

Conflict of interest: None.

• 1. Evidence-Based Medicine Working Group. Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992;268(17):2420-5.
• 2. Bhandari M, Richards RR, Sprague S, Schemitsch EH. The quality of reporting of randomized trials in the Journal of Bone and Joint Surgery from 1988 through 2000. J Bone Joint Surg Am. 2002;84-A(3):388-96.
• 3. Bhandari M, Swiontkowski MF, Einhorn TA, et al. Interobserver agreement in the application of levels of evidence to scientific papers in the American volume of the Journal of Bone and Joint Surgery. J Bone Joint Surg Am. 2004;86-A(8):1717-20.
• 4. Hoppe DJ, Bhandari M. Evidence-based orthopaedics: a brief history. Indian J Orthop. 2008;42(2):104-10.
• 5. Watts G. Let's pension off the "major breakthrough". BMJ. 2007;334 Suppl 1:s4.
• 6. Atallah AN. The Cochrane Collaboration: shared evidence for improving decision-making in human health. Sao Paulo Med J. 1999;117(5):183-4.
• 7. Poolman RW, Kerkhoffs GM, Struijs PA, Bhandari M, International Evidence-Based Orthopedic Surgery Working Group. Don't be misled by the orthopedic literature: tips for critical appraisal. Acta Orthop. 2007;78(2):162-71.
• 8. Dijkman BG, Abouali JA, Kooistra BW, et al. Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity? J Bone Joint Surg Am. 2010;92(1):48-57.
• 9. Kiter E, Karatosun V, Günal I. Do orthopaedic journals provide high-quality evidence for clinical practice? Arch Orthop Trauma Surg. 2003;123(2-3):82-5.
• 10. Siebelt M, Siebelt T, Pilot P, et al. Citation analysis of orthopaedic literature; 18 major orthopaedic journals compared for Impact Factor and SCImago. BMC Musculoskelet Disord. 2010;11:4.
• 11. Amatuzzi MLL. Análise da evolução qualitativa de publicações em Ortopedia e Traumatologia: comparação entre a Revista Brasileira de Ortopedia e Journal of Bone and Joint Surgery [Analysis of qualitative evolution from publications in Orthopedics and traumatology]. Rev Bras Ortop. 2004;39(9):527-35.
• 12. Bhandari M, Guyatt GH, Lochner H, Sprague S, Tornetta P 3rd. Application of the Consolidated Standards of Reporting Trials (CONSORT) in the Fracture Care Literature. J Bone Joint Surg Am. 2002;84-A(3):485-9.
• 13. Riera R. Designs of studies published in two Brazilian journals of orthopedics and sports medicine, recently indexed in the ISI Web of Science. Sao Paulo Med J. 2009;127(6):355-8.
• 14. Gummesson C, Atroshi I, Ekdahl C. The quality of reporting and outcome measures in randomized clinical trials related to upper-extremity disorders. J Hand Surg Am. 2004;29(4):727-34; discussion 735-7.
• 15. Martinoff R, Kreder H. Finding evidence: evidence-based practice. Hand Clin. 2009;25(1):15-27, v.
• 16. Lozano-Calderón S, Anthony S, Ring D. The quality and strength of evidence for etiology: example of carpal tunnel syndrome. J Hand Surg Am. 2008;33(4):525-38.
• 17. Moher D, Pham B, Jones A, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998;352(9128):609-13.
• 18. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions 4.2.4. Chichester: John Wiley & Sons; 2005.
• 19. Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-Based Medicine: how to practice and teach EBM. 2^nd ed. London: Churchill Livingstone; 2000.
• 20. Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85-A(1):1-3.
• 21. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74.
• 22. Hanzlik S, Mahabir RC, Baynosa RC, Khiabani KT. Levels of evidence in research published in The Journal of the Bone and Joint Surgery (American Volume) over the last thirty years. J Bone Joint Surg Am. 2009;91(2):425-8.

Appendix 1

Publication Dates

• Publication in this collection
  13 May 2011
• Date of issue
  Mar 2011

History

• Reviewed
  16 Dec 2010
• Received
  08 July 2010
• Accepted
  17 Dec 2010