The use and misuse of the "impact factor" as a parameter for evaluation of scientific publication quality: a proposal to rationalize its application

Braz J Med Biol Res, December 2003, Volume 36(12) 1605-1612 (Concepts and Comments)

The use and misuse of the "impact factor" as a parameter for evaluation of scientific publication quality: a proposal to rationalize its application^*

P.M.Z. Coelho^1,5, C.M.F. Antunes^2,5, H.M.A. Costa^2†, E.G. Kroon³, M.C. Sousa Lima⁴ and P.M. Linardi²

¹Centro de Pesquisas René Rachou, FIOCUZ, Belo Horizonte, MG, Brasil

Departamentos de ²Parasitologia and ³Microbiologia, and ⁴Biblioteca Setorial, Instituto de Ciências Biológicas, Universidade Federal de Minas Gerais, Belo Horizonte, MG, Brasil

⁵Santa Casa de Misericórdia de Belo Horizonte, Belo Horizonte, MG, Brasil

^Text

References

^{Correspondence and Footnotes}

Abstract

We present a critical analysis of the generalized use of the "impact factor". By means of the Kruskal-Wallis test, it was shown that it is not possible to compare distinct disciplines using the impact factor without adjustments. After assigning the median journal the value of one (1.000), the impact factor value for each journal was calculated by the rule of three. The adjusted values were homogeneous, thus permitting comparison among distinct disciplines.

Key words: Impact factor, Multidisciplinary adjustment, Citation index, Bibliometric evaluation

Introduction

The use of the "impact factor" Journal Citation Reports (JCR) from the Science Citation Index (SCI) published by the Institute of Scientific Information (ISI) (1) as an indicator of scientific journal excellence is becoming increasingly generalized. In Brazil, this index is being used as a parameter in judgments carried out by financing agencies supporting scientific research and evaluating graduate programs and by educational and research institutions when selecting their scientific personnel (2).

Recently, some aspects related to the indiscriminate use of the "impact factor" have been discussed elsewhere, such as its use without the evaluation of its reach, as well as its limitations for comparing scientific production among the various areas of knowledge (3,4). As a practical result of the suggestions presented in the cited papers, a well-known international journal introduced changes in its editorial policy for improving its "impact factor" (5).

As the need for further discussions about essential points of the "impact factor" still exists, issues meriting considerations are addressed in the present article. A new proposal with the objective of establishing a more practical and rational use for this index is presented in order to prevent distortions in judgments where its use is appropriate.

The source publications

The SCI, as per its own definition, is a calendar-year index to scientific literature published by the ISI. The source publications are only the journals processed for the SCI. The first aspect to be considered is the fact that a private commercial enterprise is the most important organ for assessing the quality of scientific production. Among the criteria for inclusion in the ISI, an annual fee is required, which represents a restraint for the inclusion of scientific journals from developing countries. This is reflected by the small number (57) of Latin-American scientific journals cited by ISI: Brazil (15), Mexico (14), Chile (9), Argentina (7), Venezuela (4), Colombia, Costa-Rica, Cuba, Ecuador, Jamaica, Uruguay and Trinidad-Tobago (1 each), from a universe of nearly 16,000 journals within 160 areas of knowledge published by 46 countries. The second aspect is that there is a large body of scientists and of scientific institutions that do not participate in the process of establishing criteria for judging scientific production. Our suggestion would be to create an organ with this objective, supported by the United Nations, with the participation of agencies financing scientific research and of scientific societies.

Policy of support and encouragement of Brazilian scientific journals

The present tendency, due to the reasons discussed above, is for Brazilian scientists to publish their papers only in journals with higher "impact factor" values. This tendency could lead Brazilian scientific journals to ostracism within a short period of time. It is common knowledge that, in general, scientists send their papers to different international journals and, only after running out of all the possibilities for publication abroad, will send an article to an indexed Brazilian journal. This fact prevents Brazilian journals from reaching the desired excellence level. It is important to consider publications in our journals indexed by SCI or in our journals with possibilities to attain this indexing status. This would be taken into account in judgments carried out by financing agencies and educational/research institutions, acting as a stimulus to scientists to send first hand their qualified articles or at least a proportion of their papers to these journals. For the areas of Biology, Medicine and Public Health, the "impact factor" of SciELO (Scientific Electronic Library on Line), which includes 106 Brazilian scientific journals (about 21% of all Brazilian journals), could be used together with the SCI. SciELO follows criteria similar to those of SCI, does not charge an annual fee and uses a continuous evaluation process. The SciELO program is coordinated by the Biblioteca Regional de Medicina (BIREME) and Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP).

Proposal for a more rational use of the "impact factor" as an instrument for intra- and inter-area comparison of scientific knowledge

At present, until the appearance of adequate court(s) to establish criteria for evaluating scientific production, it is worth quoting the recommendations of caution expressed by the authors of the SCI: "Caution!" Caution is advisable in comparing journals, especially journals from different disciplines. The journal literature varies in its importance as a means of disseminating information in different fields. Wide citation may be necessary practice in one field, but a redundancy in another discipline because of other means of dissemination. Citation practices differ from one field to another. The difference may be complicated by a difference in the cited half-life of journal literature in different fields, as well as the size of the extant citable literature. Rapid obsolescence may characterize one field but not another. Thus, for example, it would be foolish to conclude merely on the basis of citation counts that the journal of the American Chemical Society is a "better" journal than Annals of Mathematics, or to hypothesize, without a great deal of study, which serves its own field "better". To enable the JCR user to analyze more carefully these data within subject groupings, a breakdown of journals by subject categories with impact factors and cited half-life indicators is provided in the "Subject Category Listing" (6).

To overcome these limitations, an adjustment is proposed in order to harmonize discrepancies. The adjustment is not an ideal solution, but may prevent distortions, errors or injustices in judgments in which scientific production - evaluated by the "impact factor" - constitutes the principal element in the decision process.

To illustrate, characteristics of the distribution of the "impact factor" of scientific journals from three areas, Parasitology, Microbiology and Immunology, are presented in Table 1. These areas are considered jointly (Biology 3) during the process of evaluation of Graduate Programs conducted by the Coordenação do Aperfeiçoamento de Pessoal de Nível Superior - CAPES (Brazilian Ministry of Education). The analysis of these distributions (Shapiro-Francia H test for normality) has revealed that, with the exception of Parasitology journals, they are not normal. In these situations, if the data are not transformed, the median should be the measure of central tendency (or location) to be used to characterize or compare distributions. The Kruskal-Wallis test for equality of populations showed a highly significant difference (P = 0.0006) between their medians. In other words, these distributions are not comparable. By transforming the value of the median to 1.000, adjusted "impact factors" can be generated by a simple calculation. For example, in the area of Parasitology/Tropical Medicine, the adjusted value for the Parasitology "impact factor" would be:

The use and misuse of the "impact factor" as a parameter for evaluation of scientific publication quality: a proposal to rationalize its application. P.M.Z. Coelho, C.M.F. Antunes, H.M.A. Costa, E.G. Kroon, M.C. Sousa Lima and P.M. Linardi. Brazilian Journal of Medical and Biological Research, 36 (12): 1605, 2003.

 

where aif = adjusted impact factor; m = median, and if = impact factor; (for values, see Table 2).

The use and misuse of the "impact factor" as a parameter for evaluation of scientific publication quality: a proposal to rationalize its application. P.M.Z. Coelho, C.M.F. Antunes, H.M.A. Costa, E.G. Kroon, M.C. Sousa Lima and P.M. Linardi. Brazilian Journal of Medical and Biological Research, 36 (12): 1605, 2003.

 

The original (from JCR) and adjusted "impact factor" values can be seen in Tables 2, 3 and 4. Comparison of the distribution of the adjusted values showed that 1) they are not normal (Shapiro-Francia H test for normality) and 2) their medians do not differ significantly (Kruskal-Wallis test, P = 0.772).

The use and misuse of the "impact factor" as a parameter for evaluation of scientific publication quality: a proposal to rationalize its application. P.M.Z. Coelho, C.M.F. Antunes, H.M.A. Costa, E.G. Kroon, M.C. Sousa Lima and P.M. Linardi. Brazilian Journal of Medical and Biological Research, 36 (12): 1605, 2003.

 

The use and misuse of the "impact factor" as a parameter for evaluation of scientific publication quality: a proposal to rationalize its application. P.M.Z. Coelho, C.M.F. Antunes, H.M.A. Costa, E.G. Kroon, M.C. Sousa Lima and P.M. Linardi. Brazilian Journal of Medical and Biological Research, 36 (12): 1605, 2003.

Since review journals do not publish original data, we suggest that these journals be not included in the calculation of the median. However, because they publish invited articles, they should be considered in the judgments, attesting to the investigator's qualification and prestige.

The procedure to be used for evaluation of the merit of papers belonging to the same area would be straightforward: the proportion of publications above or below the median (original "impact factor"), taken as the reference value. To compare scientific production among different areas of knowledge, the adjusted "impact factor" should be used. Since they are proportionally adjusted, the bias which would be introduced when areas with a smaller number of journals (and consequently a smaller number of citers) are compared is prevented.

A more stringent criterion could be used for classifying Graduate Programs: as a suggestion, to be considered as level 6 and 7 (national and international excellence), a specified proportion of the scientific production of these programs would have to be published in journals with an adjusted impact factor above the 75th and the 90th percentiles, respectively (above 1.500 and 2.000, Table 1). The criterion currently adopted by the Biology 3 CAPES Committee, to publish a specified proportion of scientific articles in journals with an original "impact factor" above 4.000, introduces distortions, as pointed out earlier.

One aspect to be considered is that, due to competition, in fields with larger numbers of investigators it would be more difficult for authors to have their papers published in journals with higher "impact factors". However, looking at Table 4 (Immunology), it can be seen that even after adjustment these journals maintain higher "impact factors" compared to other fields. This would compensate for the apparent handicap that the adjustment might have caused to the median level values. Exceptions to this procedure would be journals included in more than one area with different "impact factors"; in these cases, the higher value would be the choice, acknowledging the journal's capacity of being included in more than one area.

Final considerations

The judgment of scientific production quality is a difficult process and controversies regarding its limitations and applications will always be present. In addition to the "impact factor", other indexes, such as the half-life of scientific papers, can also be used to evaluate publications (3).

Presently, an initiative, under the leadership of Drs. Varmus, Brown and Eisen, aiming at creating a "new system based on-line peer review pre-print publishing" is being implemented. It is named Public Library of Science (PLOS); additional information can be obtained through the Internet (www.publiclibraryofsciencemag.org). The Science Now, a subsidiary journal of Science, published an interview with Dr. Eisen, who announced the publication of "PLOS Biology", discussing the policy of "free-of-charge access for readers". This system is intended to be faster compared to the traditional printed journals. However, the financial dependence of scientific societies on their journals led their Editorial Boards to oppose the idea. Scientists, even those who endorsed this initiative, are still publishing their papers in the traditional scientific journals, probably afraid of losing prestige with the absence of publications in these journals. An additional problem, especially for developing countries, is the publication costs (ca. US$ 1,500/article) (7).

It is expected that this article can contribute to the discussion of the "impact factor", raising questions and motivating different viewpoints to be expressed with the intention of rationalizing its use in the complex process of evaluating scientific production quality.

Acknowledgments

We wish to thank Jane Rodrigues Guirado, librarian at the Central Library/UFMG, for the data related to the titles of national journals processed by IBICT.

Address for correspondence: P.M.Z. Coelho, Centro de Pesquisas René Rachou, FIOCRUZ, Av. Augusto de Lima, 1715, 30190-002 Belo Horizonte, MG, Brasil. E-mail: [e-mail]

Research supported by CNPq, FAPEMIG, and Pronex. P.M.Z. Coelho, C.M.F. Antunes, E.G. Kroon and P.M. Linardi are recipients of CNPq Productivity fellowships. Received July 17, 2003. Accepted September 19, 2003.

^*The calculation of the 2001 impact factor for a journal is as follows: number of citations in 2001 to papers published in 1999 plus 2000 divided by the number of papers published by the journal in 1999 plus 2000. Thus, if a journal published 150 and 160 papers in 1999 and 2000, respectively, and these papers were cited 151 times in 2001 the journal's 2001 impact factor would be 151/310 or 0.487.

• 1. Science Citation Index (1997). Annual Guide and List of Source Publications ISI (Institute for Scientific Information), Philadelphia, PA, USA, 1-149.
• 2. Conselho Nacional de Pesquisa (1997). Brasil: Indicadores Nacionais de Ciência e Tecnologia 1990-1996 Ministério da Ciência e Tecnologia, Brasília, DF, Brazil.
• 3. Linardi PM, Coelho PMZ & Costa HMA (1996). The "impact factor" as a criterion for the quality of scientific production is a relative, not absolute, measure. Brazilian Journal of Medical and Biological Research, 29: 555-561.
• 4. Coura JR & Willcox LCB (2003). Impact factor, scientific production and quality of Brazilian medical journals. Memórias do Instituto Oswaldo Cruz, 98: 293-297.
• 5. Forrest MC (1997). Impact factor abuse. Journal of Chemotherapy, 9: 3-4.
• 6. SCI Journal Citation Reports (1988). Journal Ranking ISI (Institute for Scientific Information), Philadelphia, PA, USA, 1633.
• 7. Marshall E (1999). NIH weighs bold plan for online pre-print publishing. Science, 283: 1610-1611.

 

Correspondence and Footnotes

^†

In memoriam of Helio Martins de Araújo Costa, Professor of Parasitology deceased in January, 1999, dear colleague and friend, a great scientist.

Publication Dates

• Publication in this collection
  17 Nov 2003
• Date of issue
  Dec 2003

History

• Accepted
  19 Sept 2003
• Received
  17 July 2003