Print version ISSN 1517-8692
Rev Bras Med Esporte vol.17 no.5 São Paulo Sept./Oct. 2011
EXERCISE AND SPORTS MEDICINE CLINIC
Claudio Gil Soares de AraújoI,II; Aline SardinhaIII,IV
IGraduate Program in Physical Education, Gama Filho University. Rio de Janeiro, RJ - Brazil
IIExercise Medicine Clinic, CLINIMEX. Rio de Janeiro, RJ - Brazil
IIIPanic and Breathing Laboratory, Psychiatry Institute,Federal University of Rio de Janeiro. Rio de Janeiro, RJ - Brazil
IVNational Institute of Science and Technology Translational Medicine (CNPq), RJ - Brazil
BACKGROUND: there has been an increasing interest and need in evaluating the quality of scientific papers written by researchers. Therefore, the most important measurements used are the number of citations, the average number of citations per article published in indexed journals and the H-index. Those bibliometric indicators present, however, limitations when assessing the potential impact of the publications of a given researcher, particularly among the most productive ones.
OBJECTIVE: To propose and to demonstrate the use of a new scienciometric strategy the H-index of the citing articles -, that allows better discriminationof the impact of a given researcher to the body of knowledge in the respective research area.
METHODS: Research data Web of Science - from 13 of the most productive Brazilian PhD researchers in the field of exercise and sports sciences were analyzed, including: number of publications, number of citations, average number of citations per article, H-index and the proposed H-index of citing articles (the H-index extracted from the ranking of articles that cited the papers published by a given researcher). Data from four other researchers Brazilian and foreign were used as reference for comparisons.
RESULTS: The researcher's H-index and the H-index of the citing articles were associated (r=.92; p>.01). However, when the six most productive authors with H-index above 7 were analyzed, the association between the two indexes dissapeared (r=.35 ; p=.49).
CONCLUSIONS:The H-index of the citing articles can be useful to discriminate the scientific production of authors with a high number of published papers. It is suggested that this algorithm should be adopted by Brazilian and foreign financing and scientific production evaluation agencies.
Keywords: scientometrics, citations, research, evaluation, impact factor.
The production of scientific knowledge is considered fruitful when the results and innovations brought by the research are able to reach the scientific community and general public as well. The habitual way of making the research results and new proposals and scientific analyses available is through the publishing of articles in specialized vehicles, commonly termed scientific journals. Generally, such vehicles are characterized by rigid editorial policy, supported by academically skilled and impartial editors and revisors, with the aim to select and allow only the publishing of methodologically valid and scientifically relevant articles. From this point, a process in which the researchers try to have their contributions in excellence and with high credit journals published is established. However, what does define the researcher or a scientific journal quality? In order to try to answer this question, a need to generate objective assessment criteria emerges.
Science metrics or scientometrics is a field in the information sciences which has become increasingly popular among the scientific providing agencies, the higher education institutions and the researchers of many different knowledge areas(1-3). In fact, as reported by Bergstrom(4), the analysis of bibliography citations has already been reported in 1927, but it was only through a proposal made by the researcher Eugene Garfield and published in the Science journal in 1955(5), that this bibliography metrics has become the basic reference(6). Thus, the inclusion of information or data of a given article published in the text of a new article, and consequently its citation on the bibliography reference list of this new article is currently the main element present in the most used algorithms of scientometrics(6,7).
From the increasing information systematization and more recently the use of powerful mathematical tools, it has become objectively possible to find, quantify and value the number of times a given scientific article is subsequently cited. As a natural consequence of this metrics, the impact factor of the journals and their multiple variations, representing in a simplistic analysis, the mean of the citations of the published articles by a given journal in the two years following its publishing,has appeared (8). The discussion on the positive side and the flaws or limitations of the impact factor dates from long ago(9-14), but still remains on the spotlight, as proved by many relevant and fairly recent articles(15-22). In this context, it is relevant to highlight that the citations of the articles published by Brazilian authors in national journals have already been analyzed for many years based on the data available by the ISI system (Institute of Scientific Information)(1,23-28).
Thus, it is interesting to observe that many health-related Brazilian journals, specially the most qualified ones, sometime on the last years, have published some articles on this issue, either to present a point of view(2,29-36) or to report the first factor impact calculated by the Web of Science(37-39). The issue of the impact factor has taken an even greater meaning in Brazil, since it has been used and extremely valued by the Brazilian governmental providing agencies and especially, in the evaluation of scientific production of the professors linked to the post-graduation programs by CAPES (Coordination for the Advancement of Higher Education Teaching)(40). However, this practice has been suitably questioned by the main editors of Brazilian scientific journals, notably those which act in the health field(21,33,41,42)and even by the lay press(43).
Although some of the main indicators of scientific production have shown to be sufficiently consistent and valid, able not only to resist to massive criticism for over 50 years, butalso become progressively notorious and increasingly used, it is natural that other indicators of scientific production have come out in the last years. Among the new indicators, theH-index proposed by the physicist Hirsch in 2005(44) and its subsequent variations(45-47), is perhaps the one which has attracted the most attention in the academic field. The H-index, of less intuitive comprehension, also makes use of citations obtained by the published articles without establishing a temporal limit though, being hence the citations obtained from its publishing or availability on the site of the journal (publish ahead) counted. The H-index consists of the number of published articles which received citations with a number higher ore equal to it(29,48,49) and can be applied both for an individual and group of researchers(29,50,51), as well as journals(46). For example, a researcher who has an H-index 10, implies that he/she has published at least 10 articles which received 10 or more citations from its publishing or availability (publish ahead). With conceptualization very close to the central tendency measurement known as median, the H-index is not influenced by extremes, as occurs for instance with the mean of citations per published article. Since it occurs with its basic measure or fundamental unit the citation , the H-index should be contextualized by area or subarea of knowledge for results comparison or interpretation(20,47).
Although there are positive points in the metrics described above, the analysis remains being primarily quantitative, without a more qualitative component which allows valuing the academic merit or highlight. In fact, it is relevant not only that a published article becomes cited, but also especially who is citing it and in which journal the article is being cited. One of the most recent alternatives for this issue is the proposition and analysis of the Eigenfactor(4) (www.eigenfactor.org, for further information), which incorporates a quality pondering based on one algorithm; however, its high mathematical complexity makes it extremely difficult to simply read data and minimize its wider application.
In this same logic of metric analysis, it can hence be verified the potential multiplying and disseminating factor of a given scientific production by the number of citations the articles cited are obtaining. This metrics can be defined as the H-index of the citing articles and can be obtained in a relatively simple and fast manner through the analysis and decreasing order of data made available on the Web of Science Thomson-Reuters.
In practice, we cansee that some researchers accumulate an important number of citations. However, there are those whose articles are frequently cited by exponent pairs and in high impact journals, and others whose the respective articles are cited in its great majority in journals of lower impact or less rigid editorial policy. Thus, it may be appropriate to identify the scientific metrics which allows evaluating not only the quantitative aspect of the citations but also which aggregates a valuing qualitative component.
The aim of the present article is to propose and demonstrate the discrimination potential of a new bibliography metrics the H-index of the citing articles through the results of data analysis of a selected sample of Brazilian researchers in the physical exercise and sports field.
Seventeen authors were selected for analysis of the citations and data of scientific production. Out of the 15 Brazilian authors, 13 primarily research on the biological field of the physical exercise, with Physiology and/or Exercise and Sports Medicine themes. These 13 PhD researchers (10 of which are beholders of CNPqproductivity in research scholarships levels 1A and 2), range in age and career time, from senior advisors, and are probably some of the most productive in the country in the field, to young novice Doctors. The data of four other authors of the health field two foreigners and two Brazilians were obtained to complement the analyses and allow the establishment of some reference points for comparisons.
On December, 2010 several metrics relevant to the study were researched on the data base Web of Science (Thomson-Reuters), accessed through the CAPES journal link, including: number of listed articles, number of citations, number of citations of the most cited article, mean of citations per listed article, H-index of the researcher, number of articles published in the 2006-2010quinquennium, number of articles published in 2010 and number of citations in 2010. Using tools from the system itself, a list of articles which cited the researcher's articles was carefully identified and compiled and subsequently ordered from the highest to the lowest number of citations, allowing hence visual H-index of the citing articles; that is, the H-number of articles which received at least H-citations. Special care should be taken to avoid that occasional synonyms would artificially inflate the data. Some complementary data on the undergraduation field, higher education institution of current relationship and status as CNPq scholarship beholder were obtained from the analysis of the resumes available on the Lattes Platform.
The statistical analyses were limited to conventional descriptive procedures and the determination of Pearson correlation coefficients. Whenever appropriate, the significance criterion of 5% of probability was applied. The Prism 5.04 software (GraphPad, USA) was used for figures calculations and designing.
Table1 illustrates the results of the study for the 13 Brazilian researchers in the physical exercise and sports field. The profile and volume of scientific production, a result of conscious choice of the researchers, substantially ranged from seven to 199 articles published and listed on the Web of Science, including from two to 1,106 citations in the entire professional career. The H-index of the researchers also ranged in 20 times, going from one to 20.
For 12 out of the 13 researchers, the H-index of the citing articles is higher than the H-index of the researcher him/herself, and can be up to three to four times higher. Figure 1 presents that there is a tendency of behavior similarity in the data of the H-indices of the researcher and the citing articles, reflected by a correlation coefficient of 0.92 (p < 0.001); however, when an analysis is performed using only the data of the six researchers with the highest scientific production (Figure 2) and ,therefore, with higher H-indices, it is observed that the association between the two metrics of the H-index disappears (r = 0.35; p = 0.49), indicating that for the researchers with H-index higher than 7, the H-index of the citing articles may be extremely variable in practice, ranging between 12 and 32. Table 2 presents the associations observed between the several scientific metrics and the H-index of the citing articles for the 13 Brazilian PhD researchers of the physical exercise and sports field who were analyzed in the present study.
Looking back at the data in table 1 it is possible to observe that a specific foreign researcher very active in the Sports Cardiology field certainly one of the most worldwide productive in terms of scientific publishing has 830 indexed articles on the Web of Science, 20% of which in the last five years, totalizing 33,523 citations. The most cited article by this author has 622 citations and his articles have mean citations of a little more than 40 times and his H-index is 97. For this researcher, due to limitations intrinsic to the database, it is not possible, through the access available by the CAPES journals link, to safely calculate his H-index of the citing articles. On the other hand, for a Canadian researcher publishing on Neurology themes and with article publishing concentration in the last five years, we identified 243 articles on the Web of Science and 2,528 citations and found H-index of 30 and H-index of citing articles of 56. Analyzing the obtained data of two Brazilian researchers from another health field, a CNPq productivity in research scholarship beholder and a PhD student the production of both is almost all concentrated in the last five years , it is observed that the scientific productivity metrics seems to be distinct from those observed in the exercise and sports field, with H-indices fairly high, both of the researcher and the post-graduation student 12 and 2, respectively, as well as of the citing articles 47 and 2, respectively.
It seems increasingly evident that the scientific production metrics is necessary and relevant(8,21). However, it is still important to consider the many limitations of the currently existing indicators as well as the broadening of horizons in the search for new information or indicators which are able to be obtained from the available data, especially within the Brazilian reality(21,42,52). The H-index of the citing articles proposal is inserted in this context.
It is relevant and interesting to observe that the stratification of the scientific journals by the CAPES Medicine field in the system known as Qualis, primarily considers the impact factor of the journal. In the area 21, representing Physical Education, Physiotherapy, Phonoaudilology and Occupational Therapy, in a broader and probably suitable position, the journals are stratified bothby the impact factor and by the H-index,but keeping a fairly inaccurate differentiation concerning whether or not a journal is from the field.
From the professional point of view, more specifically in the health field, the true impact of an article can be analyzed by the effective incorporation of its proposals, methods, results and conclusions to practice. The biggest example of this incorporation to the professional practice in Exercise and Sports Medicine may be observed in the protocols and nominated scales, such as the Cooper test, the Wingate test, the Astrandnomogram, the Bruce protocol, the Faulkner equation, the Borg scale, the Tanner stage. These examples and many others may have been originally published in original articles which did not necessarily reach a high number of citations, but which the subsequent articles that used them were widely cited. More over, it should be highlighted that these tests were progressively incorporated in the main textbooks of the knowledge field and, more importantly, inserted in the professional practice.
In practice, the logic of the process works in the following way: the journals tend to positively restrict the number of references an original article can present (the rule is less restrictive with review articles). Thus, the authors have to carefully select which articles should be cited to support his/her studies and data analyses and naturally tend to choose those which present higher quality and scientific importance. This is especially valid in the articles submitted to journals with stricter editorial policy, in which there is very critical evaluation of the literature used to support the article by the revisers and associated editors. As a result of this careful and selective process, the main articles with the best possible citations are published in the best journals and will be hence read by the main researchers and professionals around the world. These subjects on their turn, when devising their future studies or speak about their area of expertise, will include data from these new articles and from those which have been used in the citations and so forth, resulting in a progressive and differentiated exposition and high potential of influence from the original articles. In other words, the true potential which a scientific article presents to increment the knowledge on the field is determined by how important who cites it is, and where it is cited.
Naturally, the only metrics which does not significantly associate with the H-index of the citing articles is the number of published articles in 2010, since there was not sufficient time for the articles published in 2010 to be cited in articles and these articles would be cited. Therefore, it is natural that the articles published in the last quinquennium also present lower potential to be reflected in the H-index of the citing articles. Conversely, the correlation with the H-index of the researchers is excellent and it is very good with the mean of citations per article and with the number of citations in the ISI, reflecting the tendency of these measurements to be very similar when a broader group of researchers, mixing higher and lower experience or production, is analyzed.
Considering the scientific articles themselves, the simple quantification of the number of citations which an article receives does not allow evaluating or discriminating the multiplying or impact potential in the knowledge status on a given theme or topic. Therefore, the metrics with the H-index presents many advantages, beginning with the best characterization of the central tendency, not biased by the extreme values(16,29,44,53,54), but also for better reflecting the collection of the scientific production of a specific researcher. In our proposal, the H-index of the citing articles, the valuing of the scientific production andof its potential impact can be even much better evaluated and discriminated, when more experienced and productive researchers are considered. The results presented here indicate that when only the data of more experienced and productive researchers were analyzed, that is, with H-index above 7, the association between the two metrics H-indices and publishing of the researchers and the H-index of the citing articles of the publishing was not significant anymore. As occurs with many other bibliography indicators, the metrics proposed here the H-index of the citing articles should be contextualized to the knowledge field which is being analyzed, since apparently, distinct fields tend to present different result profiles(48). Comparative analysis with researchers from other fields or foreign ones with high scientific production also let us identify that, in rare situations, the number of citations may be so high that it makes the determination of the H-index of citing articles impossible. However, at least concerning researchers with H-indices up to 30, with at least 300 published articles and a number of citations lower than three thousand, it still seems possible to evaluate this production by the H-index of citing articles. Thus, almost all the Brazilian researchers in the health field will be able to be assessed by this metrics.
To sum up, the present article, when demonstrating the discriminating potential of the H-index of the citing articles in the differentiation of the scientific production of senior researchers corroborates its use as an additional indicator within the evaluation processes. Finally, it is suggested that this metrics proposed in the present study H-index of the citing articles is incorporated in the analysis of the scientific production of the researchers by the main providing agencies as well as higher education institutions and made available by the specific systems of the Web of Science-ISI, the Scopus, the Academic Google and the Lattes Platform.
The authors thank the financial support by CNPq and FAPERJ through research productivity and PhD scholarships.
1. Barreto ML. Crescimento e tendência da produção científica em epidemiologia no Brasil. Rev Saude Publica 2006;40:79-85. [ Links ]
2. Coura JR, de CB Willcox L. Fator de impacto, produção científica e qualidade das revistas médicas brasileiras. Memórias do Instituto Oswaldo Cruz 2003;98. [ Links ]
3. Marziale MHP, Mendes IAC. O fator de impacto das publicações científicas. Revista Latino-Americana de Enfermagem 2002;10:466-7. [ Links ]
4. Bergstrom C. Eigenfactor Measuring the value and prestige of scholarly journals. C&RL News (American Library Association) 2007;68:1-4. [ Links ]
5. Garfield E. Citation indexes for science; a new dimension in documentation through association of ideas. Science 1955;122:108-11. [ Links ]
6. Garfield E. "Science Citation Index"--A New Dimension in Indexing. Science 1964;144:649-54. [ Links ]
7. Garfield E. Journal impact factor: a brief review. Cmaj 1999;161:979-80. [ Links ]
8. Garfield E. The history and meaning of the journal impact factor. Jama 2006;295:90-3. [ Links ]
9. Garfield E. How can impact factors be improved? Bmj 1996;313:411-3. [ Links ]
10. Garfield E. Impact factors, and why they won't go away. Nature 2001;411:522. [ Links ]
11. Editors. The impact factor game. It is time to find a better way to assess the scientific literature. PLoS Med 2006;3:e291. [ Links ]
12. Falagas ME, Kouranos VD, Arencibia-Jorge R, Karageorgopoulos DE. Comparison of SCImago journal rank indicator with journal impact factor. Faseb J 2008;22:2623-8. [ Links ]
13. McVeigh ME, Mann SJ. The journal impact factor denominator: defining citable (counted) items. Jama 2009;302:1107-9. [ Links ]
14. Dempsey JA. Impact factor and its role in academic promotion: a statement adopted by the International Respiratory Journal Editors Roundtable. J Appl Physiol 2009;107:1005. [ Links ]
15. Aizpuru F. A vueltas con el factor de impacto. Enferm Clin 2011;21:64-5. [ Links ]
16. Alexandrov GA. The meaning of the 'impact factor' in the case of an open-access journal. Carbon Balance Manag 2011;6:1. [ Links ]
17. Barrett J. Is impact a factor? Cytotherapy 2011;13:261. [ Links ]
18. Barrio Cantalejo IM. El culto a los numeros y el factor de impacto. Enferm Clin 2011;21:60-1. [ Links ]
19. Hirschtick RE. A piece of my mind. Impact factor. Jama 2011;305:230. [ Links ]
20. Labadie RF, Fitzpatrick JM. Can Otolaryngology Compete with Larger Fields Regarding Impact Factor?: Is Percentile-Based Impact Factor a Solution? Otolaryngol Head Neck Surg 2011. [ Links ]
21. Rocha-e-Silva M. Impact factor, Scimago Indexes and the Brazilian journal rating system: where do we go from here? Clinics (Sao Paulo) 2010;65:351-5. [ Links ]
22. Yngve A, Tseng M, McNeill G, Hodge A, Haapala I. Is the emperor nude? Impact factor or health impact factor? Public Health Nutr 2011;14:753. [ Links ]
23. Morel RLM, Morel CM. Um estudo sobre a produção científica brasileira, segundo os dados do Institute for Scientific Information (ISI). Ciência da Informação 1977;6:99-109. [ Links ]
24. Rousseau R. Indicadores bibliométricos e econométricos para a avaliação de instituições científicas. Ci Inf Brasília 1998;27:149-58. [ Links ]
25. Vilhena V, Crestana MF. Produção científica: critérios de avaliação de impacto. Revista da Associação Médica Brasileira 2002;48:20-1. [ Links ]
26. Meneghini R, Packer AL. Articles with authors affiliated to Brazilian institutions published from 1994 to 2003 with 100 or more citations: II - identification of thematic nuclei of excellence in Brazilian science. An Acad Bras Cienc 2006;78:855-83. [ Links ]
27. Packer AL, Meneghini R. Articles with authors affiliated to Brazilian institutions published from 1994 to 2003 with 100 or more citations: I - the weight of international collaboration and the role of the networks. An Acad Bras Cienc 2006;78:841-53. [ Links ]
28. Strehl L. O fator de impacto do ISI e a avaliação da produção científica: aspectos conceituais e metodológicos. Ciência da Informação 2006;34:19-27. [ Links ]
29. Thomaz PG, Assad RS, Moreira LF. Using the impact factor and H index to assess researchers and publications. Arq Bras Cardiol 2011;96:90-3. [ Links ]
30. Bianco AC. Fator de impacto. Boletim do editor? Arq Bras Endocrinol Metabol 2004;48:335-6. [ Links ]
31. Coelho PM, Antunes CM, Costa HM, Kroon EG, Sousa Lima MC, Linardi PM. The use and misuse of the "impact factor" as a parameter for evaluation of scientific publication quality: a proposal to rationalize its application. Braz J Med Biol Res 2003;36:1605-12. [ Links ]
32. Metze K. Bureaucrats, researchers, editors, and the impact factor: a vicious circle that is detrimental to science. Clinics (Sao Paulo) 2010;65:937-40. [ Links ]
33. Petroianu A. Critérios quantitativos para analisar o valor da publicação de artigos científicos. Rev Assoc Med Bras 2003;49:173-6. [ Links ]
34. Ruiz MA, Greco OT, Braile DM. Fator de impacto: importância e influência no meio editorial, acadêmico e científico. Revista Brasileira de Cirurgia Cardiovascular 2009;24:273-8. [ Links ]
35. Bressan RA, Mari JJ, Mercadante M, et al. RBP is the Latin American medical journal with the highest Impact Factor. Rev Bras Psiquiatr 2008;30:179-82. [ Links ]
36. Lafer B, Fleck MP, Kieling C, et al. RBP increases its impact factor. Rev Bras Psiquiatr 2009;31:196. [ Links ]
37. Moreira LFP. Os arquivos e a publicação de seu primeiro fator de impacto. Arq Bras Cardiol 2010;95:1-2. [ Links ]
38. Rocha-e-Silva M. Clinics has an impact factor. Clinics (Sao Paulo) 2010;65:567-8. [ Links ]
39. Bressan RA, Miguel EC, Mari JJ, Rohde LA, Mercadante MT. We have reached the ISI! Rev Bras Psiquiatr 2005;27:170-1. [ Links ]
40. Barros AJD. Produção científica em saúde coletiva: perfil dos periódicos e avaliação pela Capes. Rev Saude Publica 2006;40:43-9. [ Links ]
41. Leite JP. O novo QUALIS e a avaliação dos Programas de Pós-Graduação na área médica: mitos e realidade. Rev Bras Psiquiatr 2010;32:103-5. [ Links ]
42. Classification of journals in the QUALIS System of CAPES--urgent need of changing the criteria! Clinics (Sao Paulo) 2010;65:121-3. [ Links ]
43. Escobar E. Ranking coloca revistas científicas brasileiras em'risco de extinção'. O Estado de São Paulo 2009 06/07/2009;Sect. A-13. [ Links ]
44. Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A 2005;102:16569-72. [ Links ]
45. Bartneck C, Kokkelmans S. Detecting h-index manipulation through self-citation analysis. Scientometrics 2011;87:85-98. [ Links ]
46. Braun T, Glanzel W, Schubert A. A Hirsch-type index for journals. Scientometrics 2006;69:169-73. [ Links ]
47. Iglesias JE, Pecharromán C. Scaling the h-index for different scientific ISI fields. Scientometrics 2007;73:303-20. [ Links ]
48. Kellner AW, Ponciano LC. H-index in the Brazilian Academy of Sciences: comments and concerns. An Acad Bras Cienc 2008;80:771-81. [ Links ]
49. RegoI S. Índice H, autoria e integridade na produção científica. Rev Bras Educ Med 2010;34:189-90. [ Links ]
50. Hirsch JE. Does the H index have predictive power? Proc Natl Acad Sci U S A 2007;104:19193-8. [ Links ]
51. Pereira JC, Bronhara B. H-index of Collective Health professors in Brazil. Rev Saude Publica 2011;45:599-606. [ Links ]
52. Ferreira CA, Loureiro CA, Saconato H, Atallah A. Validity of Qualis database as a predictor of evidence hierarchy and risk of bias in randomized controlled trials: a case study in dentistry. Clinics (Sao Paulo) 2011;66:337-42. [ Links ]
53. Mugnaini R, Packer AL, Meneghini R. Comparison of scientists of the Brazilian Academy of Sciences and of the National Academy of Sciences of the USA on the basis of the h-index. Braz J Med Biol Res 2008;41:258-62. [ Links ]
54. Healy NA, Glynn RW, Scutaru C, Groneberg D, Kerin MJ, Sweeney KJ. The h index and the identification of global benchmarks for breast cancer research output. Breast Cancer Res Treat 2011;127:845-51. [ Links ]
Mailing address: All authors have declared there is not any potential conflict of interests concerning this article.
Rua Siqueira Campos, 93/101 22031-070 Rio de Janeiro, RJ, Brazil
All authors have declared there is not any potential conflict of interests concerning this article.