Hirsch's index: a case study conducted at the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, Universidade de São Paulo

Braz J Med Biol Res, November 2007, Volume 40(11) 1529-1536

Hirsch's index: a case study conducted at the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, Universidade de São Paulo

N. Torro-Alves¹, R.D. Herculano², C.A.S. Terçariol^2,3, O. Kinouchi Filho² and C.F.O. Graeff⁴

¹Departamento de Psicologia e Educação, ²Departamento de Física e Matemática, Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, Universidade de São Paulo, Ribeirão Preto, SP, Brasil

³Centro Universitário Barão de Mauá, Ribeirão Preto, SP, Brasil

⁴Departamento de Física, Faculdade de Ciências, Universidade Estadual Paulista, Bauru, SP, Brasil

^{Correspondence and Footnotes}

Abstract

An analysis of scientific bibliographic productivity using the Hirsch h-index, information from the Institute of Scientific Information database and the Curriculum Lattes (CNPq, Brazil) was performed at the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, Universidade de São Paulo (FFCLRP-USP) that has four departments in natural, biological and social sciences. Bibliometric evaluations of undergraduate programs showed a better performance of the departments of Chemistry (P < 0.001) and Biology (P < 0.001) when compared to the departments of Physics and Mathematics and Psychology and Education. We also analyzed the scientific output of the six graduate programs of FFCLRP: Psychology, Psychobiology, Chemistry, Physics Applied to Medicine and Biology, Comparative Biology, and Entomology. The graduate program in Psychology presented a lower h-index (P < 0.001) and had fewer papers indexed by the ISI web of science (P < 0.001) when compared to the other graduate programs. The poorer performance of the Psychology program may be associated with the limited coverage by the Thompson Institute of Scientific Information database.

Key words: H-index, Scientometrics, Science in Brazil

Introduction

In order to evaluate an individual's scientific research output, a new scalar h-index has been proposed (1). According to this index, a scientist with a certain h-index has h papers with at least h citations each. This index has the advantage of evaluating scientific output because it combines both productivity and impact of the research. The necessary data for assessing the h-index are available in the Thompson Institute of Scientific Information (ISI) Web of Science and can be purchased. However, the applicability of the h-index to different areas of knowledge has not been well established.

Many studies have shown that the h-index, like other bibliometric indexes, is sensitive to the research field (2). An interesting way of dealing with such differences is to analyze the relative impact of the publications in their field of knowledge by calculating the ratio between a given citation rate and the average rate of the field, so that different research fields can be compared.

The Hirsch index has been received with great interest by the scientific community (3-7). Currently, many international research funding agencies employ the h-index as a criterion for evaluating researchers, as done in Brazil by the CNPq (8). In fact, the problem of scientific productivity quantification is critical, since it influences the distribution of financial support for the scientific community. This problem is even more complicated when analyzing different research areas.

In this context, in the present study, we analyzed the scientific research output of professors, from assistant to full professors, linked to the undergraduate and graduate programs of the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto (FFCLRP) and we discuss the applicability of the h-index in an institutional context. This is a special institution in the Brazilian university system. It belongs to the most important University in Brazil and Latin America (9), Universidade de São Paulo (USP). In addition, it is the only Faculty in USP that encompasses four important areas of knowledge (fields of concentration), covered by the departments of 1) Chemistry, 2) Physics and Mathematics, 3) Biology, and 4) Psychology and Education. These departments offer a total of eight undergraduate courses and six graduate programs for M.S. and Ph.D. students (Table 1).

Moreover, the Departments of FFCLRP represent the typical academic organization of Brazil and Latin America and could be taken as a model for analyzing other Brazilian research institutions.

Material and Methods

In order to evaluate research produced by FFCLRP, analyses of three indicators of scientific productivity and research output were conducted for each professor. The indexes analyzed for each professor were: 1) total number of papers (indexed in the Curriculum Lattes database, a National Curriculum Vitae (CV) database), 2) the number of papers indexed by the Thomson ISI Web of Science database, and 3) the h-index (1).

Quantifying the total number of papers in Curriculum Lattes

The total number of papers of each professor linked to FFCLRP was calculated using information available in their CV Lattes. The Lattes System is a Brazilian database that includes CVs of researchers linked to Brazilian academic institutions. This system was initially developed for the use of the National Council for Scientific and Technological Development (8), which is a research funding agency linked to the Ministry of Science and Technology. In the current study, we used this information for two main reasons. First, all researchers in Brazil are required to maintain their Curriculum Lattes updated for USP and for research funding agencies. Second, these CVs are unrestrictedly available on the internet. According to the latest statistics of the Lattes Platform, until October, 2005, there were 77,649 researchers and 19,470 research groups from 335 different research institutions registered in the system.

The survey was conducted by typing the names of professors of FFCLRP in the tag field "Search for researchers (simple search)" available in the Curriculum Lattes database (10).

Quantifying the number of papers indexed by the Thompson ISI Web of Science database and calculating the individual h-index

The number of papers indexed by the Thompson ISI Web of Science database for each professor was quantified. The search was conducted using the field tags and Boolean options available in the "Advanced Search" of ISI (11). For each author, we looked for papers published in Brazil and abroad. After the search in the database, outputs were sorted according to the number of citations of each paper and the individual h-index for each professor was calculated. These analyses were performed between December 2005 and January 2006.

Results

The scientific output of professors linked to undergraduate and graduate programs of the FFCLRP was calculated. Thus, two distinct data analyses were performed: one focusing on differences between undergraduate programs and the other focusing on differences between graduate programs.

Evaluating the scientific research output of professors linked to undergraduate programs of FFCLRP

The total number of papers (Curriculum Lattes), number of papers indexed by ISI and respective h-indexes were quantified for each professor linked to the undergraduate programs. The Departments of Chemistry, Biology, Physics and Mathematics, and Psychology and Education have 37, 31, 51, and 53 professors linked to programs of undergraduate studies, respectively.

The mean values of scientific output were then submitted to between-within ANOVA according to the following model: 4 "departments of undergraduate studies" (Chemistry, Biology, Physics and Mathematics, Psychology and Education) vs 3 "indexes of productivity" (total number of papers, number of papers indexed by ISI, h-index). The variable "departments of undergraduate studies" was considered to be as a between-subject factor and the variable "indexes of productivity" was considered to be a within-subject factor.

Statistical analysis showed a significant main effect of the factor "department of undergraduate studies" (F3,166 = 12.007, P < 0.001). An a posteriori comparison test (HSD Tukey) indicated that the Departments of Biology and Chemistry presented higher values for the indexes quantified. In general, professors of the Departments of Biology and Chemistry published more papers, had more articles indexed by the Thompson ISI Web of Science and presented, on average, a higher h-index when compared to professors of the Departments of Psychology and Education and Physics and Mathematics (Figure 1A). The results of multiple comparisons between departments of the FFCLRP are shown in Table 2.

In a second statistical analysis, the ratio between the number of papers indexed by ISI and the total number of papers in the Curriculum Lattes was calculated for each professor of FFCLRP and these data were submitted to one-way ANOVA that showed a significant difference between departments (F3,166 = 25.318, P < 0.001). An a posteriori comparison test (HSD Tukey) revealed that the ratio between number of papers indexed in ISI and Curriculum Lattes was lower for professors of the Department of Psychology and Education when compared to other departments (Table 3).

The two-tailed Pearson correlation test was applied in order to determine the interactions between the indexes analyzed in the study and the results indicated that the three indexes were correlated (Table 4).

Analysis of the research output of professors linked to the graduate programs of FFCLRP

The mean total number of papers, number of papers indexed by ISI and h-index were calculated for each professor linked to the graduate programs of FFCLRP. The graduate programs of "Chemistry", "Physics Applied to Medicine and Biology", "Entomology", "Comparative Biology", "Psychology" and "Psychobiology" have 45, 15, 20, 22, 32, and 16 professors, respectively.

Furthermore, the mean values of individual scientific output were submitted to between-within ANOVA according to the model: 6 "programs of graduate studies" (Chemistry, Physics Applied to Medicine and Biology, Entomology, Comparative Biology, Psychology, Psychobiology) vs 3 "indexes of productivity" (total number of papers, number of papers indexed by ISI, h-index). The variable "programs of graduate studies" was considered to be a between-subject factor and the variable "indexes of productivity" was considered to be a within-subject factor.

Data analysis revealed differences between graduate programs (F5,146 = 6.043, P < 0.001). An a posteriori comparison test (HSD Tukey) indicated that, in general, professors of the graduate program in Psychology presented a lower h-index and had fewer papers indexed by ISI web of science when compared to professors of other graduate programs (Figure 1B). Concerning the total number of papers in Curriculum Lattes, we noticed that professors of the Department of Entomology published, in general, more papers than professors of other graduate programs. Except for the graduate program in Psychology, no statistically significant differences between graduate programs were found regarding mean h-index and number of papers indexed by ISI. The results of these post hoc comparisons between graduate programs are shown in Table 5.

Ratios between number of papers indexed in ISI and total number of papers in Curriculum Lattes were submitted to one-way ANOVA, which showed a significant difference between graduate programs (F5,146 = 25.829, P < 0.001). An a posteriori comparisons test (HSD Tukey) indicated that the Psychology graduate program has a smaller percentage of papers indexed by ISI Web of Science when compared to other graduate programs (Table 3).

As observed in the first set of comparisons, the Pearson two-tailed correlation test indicated a statistically significant correlation between the three indexes (Table 4).

Figure 1
. Means and standard errors of h-index values, number of papers indexed by the Thompson ISI Web of Science, and total number of papers in the Curriculum Lattes of professors linked to the departments (A) and graduate programs (B) of the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto (FFCLRP), USP.

Table 1.
List of undergraduate and graduate programs belonging to the departments of Psychology and Education, Chemistry, Physics and Mathematics, and Biology of the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, Universidade de São Paulo, USP.

Table 2
. P values of the hypothesis test concerning equality between Departments of the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, USP.

Table 3.
Ratio between number of papers indexed by the Thompson Institute of Scientific Information (ISI) and the total number of papers in Curriculum Lattes for departments and graduate programs of the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, USP.

Table 4.
Pearson's correlation coefficients between number of papers indexed by the Thompson Institute of Scientific Information (ISI), total number of papers in Curriculum Lattes and the h-index for departments and graduate programs of the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, USP.

Table 5.
P values of the hypothesis test concerning equality between graduate programs of the Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto, USP.

Discussion

Data analysis revealed interesting differences between the graduate and undergraduate programs of FFCLRP, showing that the h-index and the other two measures of productivity investigated, i.e., total number of papers in Curriculum Lattes and number of papers indexed by Thompson ISI Web of Science, are sensitive to the field of knowledge and to the particular characteristics of the research group.

The analysis of the undergraduate programs of FFCLRP showed that the Departments of Biology and Chemistry presented higher scores for the indexes analyzed when compared to the Departments of Psychology and Education, and Physics and Mathematics.

The graduate programs in Psychobiology, Chemistry, Physics Applied to Medicine and Biology, Comparative Biology, and Entomology presented closely similar results concerning the assessment of the three indexes. Nevertheless, the graduate program in Psychology presented a lower h-index average and had fewer papers indexed by ISI when compared to the other programs. A possible reason for this is that most psychology papers are published in journals not indexed by Thompson ISI Web of Science, and therefore they are not considered in the calculation of the h-index. Moreover, when checking the number of journals of different research fields available in the ISI Journal Citation Reports (11), we noticed that Psychology has few journals indexed by ISI (60), especially when compared to Chemistry (591), Physics (644), and Biological areas (1362).

Differences in performance were greater when undergraduate programs were compared. For graduate programs, with the exception of psychology, the performances were closely similar concerning the three indexes evaluated (average h-index, number of papers in ISI and papers in Curriculum Lattes). This suggests that these bibliometric indicators are more adequate to evaluate research-oriented fields or groups of researchers (graduate programs) than non-research-oriented fields or professionals (undergraduate programs).

Interestingly, the analysis showed that all measures of productivity for graduate or undergraduate programs were strongly correlated, although they refer to different aspects of the scientific research output. This observation agrees with recent publication by Van Raan (12), in which he suggests that the h-index should not be considered superior to other assessment forms, since it correlates with other bibliometric indicators as well as with peer opinion.

Recent papers have indicated some limitations of the h-index, such as differences in the distribution of citations throughout an author's career (13). Some groups of researchers have their citation impact more concentrated in a few papers, whereas other groups have their citation impact more distributed along the time in many papers. In general, the latter groups tend to have higher h-index values, but this does not necessarily imply differences in the quality of the research. Some corrections have been proposed in order to minimize such problems. For example, Hirsh (1) suggested the use of the `m' parameter to take into account the seniority time from the first paper published to the time of evaluation. Anyway, these differences are probably reflecting intrinsic characteristics of the research area. We cannot overlook the fact that, in a similar way, the inferior h-index value obtained for the graduate program in Psychology can be a result of this field's particularities. Most psychology papers in the present study are published in journals not indexed by ISI and therefore the impact of papers published by professors of Psychology cannot be appropriately assessed on the basis of data collected from the ISI Web of Science. In this respect, the analysis of the Curriculum Lattes was especially important since it showed that a lower h-index average was not necessarily related to low scientific productivity (number of papers), but rather to a small percentage of papers indexed by Thompson ISI Web of Science. The scientific productivity of graduate programs was closely similar and the graduate program in psychology only publishes fewer papers than the Entomology program (Figure 1B).

Currently, the coverage by ISI Web of Science is still a great problem, mainly because only a small percentage of papers published worldwide is included. It is known that the Thompson ISI Web of Science database encompasses less than 5% of all journals in the world, surely a very limited coverage of scientific journals. Thus, the assessment of the h-index through ISI Web of Science may be improper for evaluating areas of knowledge that publish in journals not included in the Thompson ISI Web of Science database.

In summary, the present study indicated that most graduate programs of FFCLRP obtained similar results in the assessment of the h-index, the number of papers indexed by Thompson ISI Web of Science and the total number of papers in Curriculum Lattes. However, concerning these indexes, we found a worse performance of the graduate program in Psychology, which can be associated with the characteristics of the field of concentration and the limited coverage by the Thompson ISI Web of Science database. The h-index, like other bibliometric indicators, is sensitive to the field of knowledge and must not be used as the only factor for assessing an individual's scientific research output.

The bibliometric indicators seem to be more appropriate for comparing graduate research programs than undergraduate non-research-oriented programs. Nevertheless, particularities of the research fields must be taken into account during such evaluations. Pereira et al. (14), when analyzing the Brazilian scientific production, highlighted that "less conspicuous fields might have their assessment jeopardized". Therefore, it would be interesting to develop new strategies for evaluating some research fields, such as psychology or other disciplines from the social sciences, which are generally not properly assessed in terms of their scientific contribution.

References

1. Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A 2005; 102: 16569-16572.

2. Batista PD, Campiteli MG, Kinouchi O, Martinez AS. Is it possible to compare researchers with different scientific interests? Scientometrics 2006; 68: 179-189.

3. Roediger HL. The h-index in Science: A new measure of scholarly contribution. Observer 19 (4). http://www.psychologicalscience.org/observer/getArticle.cfm?id=1971.

4. Glänzel W. On the h-index - A mathematical approach to a new measure of publication activity and citation impact. Scientometrics 2006; 67: 315-321.

5. Bornmann L, Daniel HD. Does the h-index for ranking of scientists really work? Scientometrics 2005; 65: 391-392.

6. Popov SB. A parameter to quantify dynamics of a researcher's scientific activity. http://arxiv.org/abs/physics/0508113.

7. Ball P. Index aims for fair ranking of scientists. Nature 2005; 436: 900.

8. CNPq. The National Council for Scientific and Technological Development, Plataforma Lattes. http://buscatextual.cnpq.br/buscatextual/index.jsp. Accessed January, 2006.

9. Times Higher Education Supplement. World University Rankings. http://www.thes.co.uk. Accessed May, 2006.

10. CNPq. The National Council for Scientific and Technological Development. http://www.cnpq.br. Accessed May, 2006.

11. ISI (Thompson ISI Web of Science). ISI Web of Knowledge. http://portal.isiknowledge.com/portal.cgi?DestApp=WOS&Func=Frame. Accessed August, 2007.

12. van Raan AFJ. Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics 2006; 67: 491-502.

13. Harzing AW. Reflections on the h-index. Harzing.com - Research in International and Cross-cultural Management. http://www.harzing.com/pop_hindex.htm. Accessed January, 2007.

14. Pereira JCR, Escuder MML, Zanetta DMT. Brazilian Sciences and Government Funding at the State of São Paulo. Scientometrics 1998; 43: 177-188.

Address for correspondence: N. Torro-Alves, Programa de Pós-graduação em Psicobiologia, FFCLRP, USP, Av. Bandeirantes, 3900, Bloco D, 14040-901 Ribeirão Preto, SP, Brasil. Fax: [phone]. E-mail: [e-mail] or [e-mail]

Research supported by CAPES. Received November 22, 2006. Accepted August 16, 2007.

Correspondence and Footnotes

Publication Dates

• Publication in this collection
  09 Nov 2007
• Date of issue
  Nov 2007

History

• Accepted
  16 Aug 2007
• Received
  22 Nov 2006