Design and level of evidence of studies published in two Brazilian medical journals recently indexed in the ISI Web of Science database

Desenho e nível de evidência dos estudos publicados em dois periódicos brasileiros recentemente indexados na Web of Science -ISI

ORIGINAL ARTICLE

Design and level of evidence of studies published in two Brazilian medical journals recently indexed in the ISI Web of Science database

Desenho e nível de evidência dos estudos publicados em dois periódicos brasileiros recentemente indexados na Web of Science -ISI

Maria Regina Torloni^I; Rachel Riera^II

^IMD, PhD. Gynecologist and obstetrician in the Department of Obstetrics, and research assistant in the Brazilian Cochrane Center, Universidade Federal de São Paulo - Escola Paulista de Medicina (Unifesp-EPM), São Paulo, Brazil

^IIMD, MSc. Research assistant in the Brazilian Cochrane Center, and physician in the Discipline of Emergency Medicine and Evidence-Based Medicine, Universidade Federal de São Paulo - Escola Paulista de Medicina (Unifesp-EPM), São Paulo, Brazil

^{Address for correspondence}

ABSTRACT

CONTEXT AND OBJECTIVES: The level of evidence and methodological quality of articles published in medical journals are important aids for clinicians in decision-making and also affect journals' impact factor. Although systematic reviews (SR) are considered to represent the highest level of evidence, their methodological quality is not homogeneous and they need to be as carefully assessed as other types of study. This study aimed to assess the design and level of evidence of articles published in 2007, in two recently indexed Brazilian journals (Clinics and Revista da Associação Médica Brasileira), and to evaluate the methodological quality of the SRs.

DESIGN AND SETTING: Descriptive study developed in the Brazilian Cochrane Center, Universidade Federal de São Paulo.

METHODS: All 289 published articles were classified according to types of study design and level of evidence. The SRs were critically appraised by two evaluators using the AMSTAR tool.

RESULTS: The most frequent design types were cross-sectional studies (39.9%), case reports (15.8%), experimental studies (10.8%) and narrative reviews (7.4%). According to the Oxford criteria, 25.6% of the articles were classified as level 4 or 5 evidence, while 2.8% were level 1. SRs represented only 2% of the published articles and their methodological quality scores were low.

CONCLUSIONS: The main design types among the published papers were observational and experimental studies and narrative reviews. SRs accounted for a small proportion of the articles and had low methodological scores. Brazilian medical journals need to encourage publication of greater numbers of clinically relevant papers of high methodological quality.

Key words: Journal impact factor. Periodicals as topic. Journal article [Publication type]. Research design. Review [Publication type].

RESUMO

CONTEXTO E OBJETIVOS: O nível de evidência e a qualidade metodológica dos artigos publicados em revistas médicas são importantes para auxiliar os clínicos a tomarem decisões e também afetam o fator de impacto dos periódicos. Apesar de as revisões sistemáticas (RS) serem consideradas o maior nível de evidência, sua qualidade metodológica não é homogênea e elas precisam ser avaliadas tão criteriosamente como outros tipos de estudos. O objetivo deste estudo foi avaliar os desenhos e nível de evidência dos artigos publicados em 2007 em duas revistas brasileiras recentemente indexadas (Clinics e Revista da Associação Médica Brasileira) e avaliar a qualidade metodológica das RS.

TIPO DE ESTUDO E LOCAL: Estudo descritivo desenvolvido no Centro Cochrane do Brasil, Universidade Federal de São Paulo.

MÉTODOS: Todos os 289 artigos publicados foram classificados quanto ao tipo de desenho e nível de evidência. As RS foram analisadas criticamente por dois revisores usando o instrumento AMSTAR.

RESULTADOS: Os tipos de desenhos mais frequentes foram os transversais (39,9%), relatos de casos (15,8%), estudos experimentais (10,8%) e revisões narrativas (7,4%). Segundo os critérios de Oxford, 25,6% dos artigos eram de nível 4 ou 5 e 2,8% de nível 1. As RS representaram apenas 2% do total dos artigos publicados e tiveram notas de qualidade metodológica baixas.

CONCLUSÕES: Os principais tipos de desenhos dos estudos publicados foram observacionais, experimentais e revisões narrativas. As RS representaram uma pequena parcela dos artigos e tiveram baixa pontuação metodológica. Revistas médicas brasileiras devem incentivar a publicação de mais artigos clinicamente relevantes com alta qualidade metodológica.

Palavras-chave: Fator de impacto de revistas. Publicações periódicas como assunto. Artigo de revista. Projetos de pesquisa. Revisão.

INTRODUCTION

Evidence-based medicine consists of conscientious, explicit and judicious use of the current best evidence in making decisions about care for individual patients.¹ Busy clinicians, especially in developing countries, frequently rely on medical journals to help them find reliable evidence to answer questions that come up in their daily practices. In this context, medical journals can play an important educational role by offering high-level evidence to help and support these clinicians in their decision-making processes.

Systematic reviews (SRs) are currently considered to be the highest level of evidence in the hierarchy of studies.² SRs involve an exhaustive review of the literature addressing a clearly defined question, using systematic, transparent and explicit methodology to identify, select and critically evaluate all the relevant studies. After relevant data from the primary studies available have been collected, extracted and analyzed, a synthesis of the findings is then produced in a clear and objective manner. Conducting a SR is a complex task and flaws are possible in this process, thus leading to variations in the quality of published SRs. Therefore, readers and users of SRs should take a critical viewpoint and look carefully at the methodological quality of the published papers available.

From an editorial perspective, the methodological quality and level of evidence of the articles published in a journal are important determinants of how often an article is cited, and they consequently affect the impact factor (IF) of that journal. Eight Brazilian medical journals were recently included in the Journal Citation Reports (JCR) database³ and will have their impact factor available from 2010 onwards. Two recently published articles^4,5 have assessed the relevant methodological aspects of articles published in 2007 in four of these journals (Acta Ortopédica Brasileira, Revista Brasileira de Medicina do Esporte, Arquivos Brasileiros de Cardiologia and Revista Brasileira de Cirurgia Cardiovascular). As a continuation of this series, the present article will analyze the articles published in two additional journals (Clinics and Revista da Associação Médica Brasileira) during the same year, looking specifically at the number and quality of published systematic reviews.

OBJECTIVES

This review aimed to analyze the types of articles published in 2007 in two recently indexed Brazilian medical journals. Specifically, we sought 1) to assess the distribution of articles according to type of study design and level of evidence, and 2) to evaluate the methodological quality of published systematic reviews.

METHODS

Through a manual search, the two authors retrieved all articles published in 2007 in Clinics and Revista da Associação Médica Brasileira (RAMB). All manuscripts were read and classified according to their study design into one of eight main categories: 1) Primary clinical studies: case reports and case series, case-control studies, cohort studies, cross-sectional studies, trials (randomized, quasi-randomized and non-randomized) and diagnostic accuracy studies; 2) Primary experimental studies on animal models or corpses; 3) Integrative studies on the literature: narrative reviews and systematic reviews; 4) Technical notes describing surgical procedures or anatomical findings; 5) Development, translation and validation of scales, clinical measurements and questionnaires; 6) Clinical guidelines; 7) Continuing medical education quizzes; and 8) Others: editorials, commentaries (on other published studies), opinions, reflections on practice and attitudes and letters to the editor. Each article was classified after reading the title, abstract (when available) and methods sections.

Each article was classified according to the system of the Oxford Centre for Evidence-based Medicine Levels of Evidence.⁶ This classification system ranks the validity of the evidence into a hierarchy, such that level 1 is the highest level and level 5 is the lowest. Articles categorized in groups 4-8 (non-evidence literature) were excluded from this analysis.

To achieve the second objective, the full texts of all the systematic reviews published in these two journals in 2007 were retrieved, read and evaluated by two independent raters (the authors) using the AMSTAR tool (a measurement tool to assess systematic reviews), which consists of 11 items that are rated as 0 or 1.⁷ This tool has good face and content validity for measuring the methodological quality of systematic reviews.⁸ The AMSTAR scores of each rater were compared and differences were discussed until a consensus was reached.

RESULTS

In 2007, Clinics and RAMB each published five issues, containing 127 articles and 162 articles, respectively. Thus, a total of 289 articles were evaluated. Table 1 presents the distribution and relative frequency of each type of article published in the two journals. Cross-sectional studies and case reports were the most frequent design type in both journals, representing over 50% of all the clinical articles published during that year. Out of the 289 articles published in 2007, four were randomized (or quasi-randomized) trials^9-12 and four were systematic reviews,^13-16 thus representing less than 2% of the total number of published papers in both of these journals. After exclusion of six clinical guidelines^17-22 and 80 non-evidence articles (technical notes, clinical measurement validation studies, continuing medical education, quizzes and others) the distribution of the remaining 203 articles is presented in Table 2. According to the Oxford criteria, 25.6% of the articles were classified as level 4 or 5 evidence, while 2.8% were level 1.

Table 3 presents the AMSTAR scores of the four systematic reviews^13-16 published during 2007 in the two journals. The overall scores ranged from 1 to 6, with a mean score of 4.0 (standard deviation: 2.2).

DISCUSSION

The most frequent study designs of the articles published in Clinics and RAMB in 2007 were cross-sectional studies, case reports, experimental studies and narrative reviews, which together represented about three-quarters of all the clinical papers published. The number of systematic reviews published in these two journals represented only 2% of all the clinical papers published. Their methodological quality, as assessed by AMSTAR, was low.

With rare exceptions, case reports, experimental studies or narrative reviews are insufficient to justify or support healthcare decisions regarding treatment, diagnosis or prevention. However, we found that a relatively large number of such articles were published in the two journals. The journal Clinics contained a large number of experimental studies (n = 14), mostly using rat models, but also one study using a pig model (analyzing respiratory dysfunction in relation to sepsis)²³ and three experimental studies on the knees of human cadavers.^24-26 For both journals, the most common clinical questions addressed in the articles related to the prevalence, frequency or associations between variables, which were answered using cross-sectional studies. There were only 25 cohort studies (8.6% of all the articles), and a single case-control study.²⁷ The popularity of cross-sectional studies is probably due to the fact that they are methodologically easier to conduct, require less time and cost less than other types of observational studies (cohort and case-control studies) or clinical trials.²⁸ These practical considerations are important factors in determining what type of study to conduct, especially in settings with limited resources.

According to the quality assessment tool used, the few systematic reviews published in the two journals were of relatively low quality. To reduce bias and subjectivity, it is recommended that all steps of a systematic review be conducted by two independent investigators. However, one of the published reviews was apparently performed by a single reviewer (the author)¹⁶ and, although the other three reviews had more than one author,^13-15 they did not report whether the study selection and data extraction had been done in duplicate. Similarly, none of the four reviews provided all the necessary details on the literature search, such as the names of the electronic databases, the years and key words used, and whether searches were complemented by consulting current contents, reviews, textbooks, specialized registers or experts in the particular field of study, or by reviewing the references in the studies found. Additionally, none of the reviews provided a complete list of the studies included and excluded, nor did they clearly acknowledge potential sources of support, either for the systematic review or for the studies included.

Ideally, all published medical articles should be relevant, have good methodological quality and offer a high level of evidence. Although this may be the sincere aim of the editors of most journals, the reality for the reviewers and editorial teams of medical journals is that they have to struggle through piles of submitted manuscripts every week, trying to sort and select for publication the best possible studies from among a plethora of less-than-perfect texts submitted by hopeful authors. Nevertheless, in keeping with the goal of offering high-level evidence for clinicians, and in order to increase their impact factor, Brazilian medical journals should be encouraged to try to increase the numbers and methodological quality of the systematic reviews published in their issues.

CONCLUSIONS

In 2007, the most common study designs among the articles published in Clinics and RAMB were cross-sectional studies, case reports, experimental studies and narrative reviews. The numbers and methodological quality of the systematic reviews published in these two journals were low. Publication of greater numbers of clinically relevant and high-quality studies should be encouraged among Brazilian journals.

Date of first submission: June 14, 2010

Last received: June 14, 2010

Accepted: July 19 , 2010

Sources of funding: None

Conflict of interest: None

Brazilian Cochrane Center, São Paulo, Brazil

• 1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312(7023):71-2.
• 2. Guyatt GH, Sackett DL, Sinclair JC, et al. Users' guides to the medical literature. IX. A method for grading health care recommendations. Evidence-Based Medicine Working Group. JAMA. 1995;274(22):1800-4.
• 3. Thomson Reuters. Introducing the impact factor. Available from: http://thomsonreuters.com/products_services/science/academic/impact_factor/ Accessed in 2010 (Jul 15).
• 4. Macedo CR, Silva DL, Puga ME. Methodological adequacy of articles published in two open-access Brazilian cardiology periodicals. Sao Paulo Med J. 2010;128(2):85-9.
• 5. Riera R. Designs of studies published in two Brazilian journals of orthopedics and sports medicine, recently indexed in the ISI Web of Science. Sao Paulo Med J. 2009;127(6):355-8.
• 6. Centre for Evidence Based Medicine. Oxford Centre for Evidence-based Medicine - Level of Evidence. Available from: http://www.cebm.net/index.aspx?o=1025 Accessed in 2010 (Jul 15).
• 7. Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.
• 8. Shea BJ, Bouter LM, Peterson J, et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One. 2007;2(12):e1350.
• 9. Cabrita HB, Croci AT, Camargo OP, Lima AL. Prospective study of the treatment of infected hip arthroplasties with or without the use of an antibiotic-loaded cement spacer. Clinics (Sao Paulo). 2007;62(2):99-108.
• 10. Costa MN, Marcondes AM, Sakano E, Kara-José N. Endoscopic study of the intranasal ostium in external dacryocystorhinostomy postoperative. Influence of saline solution and 5-fluorouracil. Clinics (Sao Paulo). 2007;62(1):41-6.
• 11. Moro ET, Nakamura G, Martino RS, et al. Efeito do propofol associado à efedrina no tempo de latência do cisatracúrio [Effect of propofol in combination with ephedrine upon the onset time of cisatracurium]. Rev Assoc Med Bras. 2007;53(3):272-5.
• 12. Sakaki MH, Crocci AT, Zumiotti AV. Comparative study of the locked intramedullary nail and Ender pins in the treatment of tibial diaphyseal fractures. Clinics (Sao Paulo). 2007;62(4):455-64.
• 13. Couto JM, Castilho EA, Menezes PR. Chemonucleolysis in lumbar disc herniation: a meta-analysis. Clinics (Sao Paulo). 2007;62(2):175-80.
• 14. Faria LS, Arneiro AHA, Troster EJ. Ventilação de alta frequência em crianças e adolescentes com síndrome do desconforto respiratório agudo (impacto sobre o uso de ecmo) [High-frequency ventilation in children and adolescents with acute respiratory distress syndrome (impact on the use of ecmo)]. Rev Assoc Med Bras. 2007;53(3):223-8.
• 15. Holtz JV, Grandi LM, Ecclissato PR, Bernardo WM. Há benefício no uso da ranitidina em pacientes hospitalizados, sob menor regime de estresse, na profilaxia da úlcera péptica ou da hemorragia digestiva? [Is there any benefit in the use of ranitidine in hospitalized patients, under a minor regimen of stress, in the prophylaxis of peptic ulcer or gastrointestinal bleeding?]. Rev Assoc Med Bras. 2007;53(6):476-7.
• 16. Raimondi AM. Metanálise sobre o uso de glicocorticóide pré-operatório para redução do risco de complicações após esofagectomia por carcinoma do esôfago [Use of preoperative glucocorticoid to reduce risk of complications after esophagectomy for carcinoma of the esophagus: meta-analysis]. Rev Assoc Med Bras. 2007;53(2):158-61.
• 17. Sociedade Brasileira de Nefrologia e Urologia. Transplante renal: complicações cirúrgicas [Renal transplantation: surgical complications]. Rev Assoc Med Bras. 2007;53(3):203-5.
• 18. Sociedade Brasileira de Cirurgia de Cabeça e Pescoço e Colégio Brasileiro de Radiologia. Tumores congênitos do pescoço [Congenital neck neoplasms]. Rev Assoc Med Bras. 2007;53(4):288-90.
• 19. Sociedade Brasileira de Coloproctologia. Diagnóstico da hemorróida [Diagnosis of hemorrhoids]. Rev Assoc Med Bras. 2007;53(1):10-1.
• 20. Federação Brasileira das Associaçoes de Ginecologia e Obstetrícia. Assistência pré-natal parte II [Prenatal care - part II]. Rev Assoc Med Bras. 2007;53(6):479-81.
• 21. Assistência pré-natal: parte I [Prenatal care - part I]. Rev Assoc Med Bras. 2007;53(5):385-6.
• ²²
  Disfunção erétil: tratamento com drogas inibidoras da fosfodiesterase tipo 5 [Erectile dysfunction: treatment with type 5 phosphodiesterase inhibitors]. Rev Assoc Med Bras. 2007;53(2):102-3.
• 23. de Azevedo LC, Park M, Noritomi DT, Maciel AT, Brunialti MK, Salomão R. Characterization of an animal model of severe sepsis associated with respiratory dysfunction. Clinics (Sao Paulo). 2007;62(4):491-8.
• 24. Albuquerque RF, Amatuzzi MM, Pacheco AP, Angelini FJ, Campos JO. Positioning of the femoral tunnel for arthroscopic reconstruction of the anterior cruciate ligament: comparative study of 2 techniques. Clinics (Sao Paulo). 2007;62(5):613-8.
• 25. Albuquerque RF, Sasaki SU, Amatuzzi MM, Angelini FJ. Anterior cruciate ligament reconstruction with double bundle versus single bundle: experimental study. Clinics (Sao Paulo). 2007;62(3):335-44.
• 26. Garbelotti Júnior SA, Pelozo Júnior O, Caldana RP, Ramalho Jr A, Smith RL. Experimental evaluation of 3-dimensional kinematic behavior of the cruciate ligaments. Clinics (Sao Paulo). 2007;62(5):619-26.
• 27. Vilela R, Jácomo AD, Tresoldi AT. Risk factors for central venous catheter-related infections in pediatric intensive care. Clinics (Sao Paulo). 2007;62(5):537-44.
• 28. Fletcher RH, Fletcher SW, Wagner EH. Epidemiologia clínica: bases científicas da conduta médica. 3Ş ed. Porto Alegre: Artes Médicas; 1996.

Publication Dates

• Publication in this collection
  22 Nov 2010
• Date of issue
  July 2010

History

• Reviewed
  14 June 2010
• Received
  14 June 2010
• Accepted
  19 July 2010