The use and misuse of the “ impact factor ” as a parameter for evaluation of scientific publication quality : a proposal to rationalize its application *

We present a critical analysis of the generalized use of the “impact factor”. By means of the Kruskal-Wallis test, it was shown that it is not possible to compare distinct disciplines using the “impact factor” without adjustments. After assigning the median journal the value of one (1.000), the “impact factor” value for each journal was calculated by the rule of three. The adjusted values were homogeneous, thus permitting comparison among distinct disciplines. Correspondence


Introduction
The use of the "impact factor" Journal Citation Reports (JCR) from the Science Citation Index (SCI) published by the Institute of Scientific Information (ISI) (1) as an indicator of scientific journal excellence is becoming increasingly generalized.In Brazil, this index is being used as a parameter in judgments carried out by financing agencies supporting scientific research and evaluating graduate programs and by educational and research institutions when selecting their scientific personnel (2).
Recently, some aspects related to the indiscriminate use of the "impact factor" have been discussed elsewhere, such as its use without the evaluation of its reach, as well as its limitations for comparing scientific production among the various areas of knowledge (3,4).As a practical result of the suggestions presented in the cited papers, a wellknown international journal introduced changes in its editorial policy for improving its "impact factor" (5).
As the need for further discussions about essential points of the "impact factor" still exists, issues meriting considerations are ad-dressed in the present article.A new proposal with the objective of establishing a more practical and rational use for this index is presented in order to prevent distortions in judgments where its use is appropriate.

The source publications
The SCI, as per its own definition, is a calendar-year index to scientific literature published by the ISI.The source publications are only the journals processed for the SCI.The first aspect to be considered is the fact that a private commercial enterprise is the most important organ for assessing the quality of scientific production.Only a small number (57) of Latin-American scientific journals are cited by ISI: Brazil (15), Mexico (14), Chile (9), Argentina (7), Venezuela (4), Colombia, Costa-Rica, Cuba, Ecuador, Jamaica, Uruguay and Trinidad-Tobago (1 each), from a universe of nearly 16,000 journals within 160 areas of knowledge published by 46 countries.The second aspect is that there is a large body of scientists and of scientific institutions that do not participate in the process of establishing criteria for judging scientific production.Our suggestion would be to create an organ with this objective, supported by the United Nations, with the participation of agencies financing scientific research and of scientific societies.

Policy of support and encouragement of Brazilian scientific journals
The present tendency, due to the reasons discussed above, is for Brazilian scientists to publish their papers only in journals with higher "impact factor" values.This tendency could lead Brazilian scientific journals to ostracism within a short period of time.It is common knowledge that, in general, scientists send their papers to different international journals and, only after running out of all the possibilities for publication abroad, will send an article to an indexed Brazilian journal.This fact prevents Brazilian journals from reaching the desired excellence level.It is important to consider publications in our journals indexed by SCI or in our journals with possibilities to attain this indexing status.This would be taken into account in judgments carried out by financing agencies and educational/research institutions, acting as a stimulus to scientists to send first hand their qualified articles or at least a proportion of their papers to these journals.For the areas of Biology, Medicine and Public Health, the "impact factor" of SciELO (Scientific Electronic Library on Line), which includes 106 Brazilian scientific journals (about 21% of all Brazilian journals), could be used together with the SCI.SciELO follows criteria similar to those of SCI and uses a continuous evaluation process.The SciELO program is coordinated by the Biblioteca Regional de Medicina (BIREME) and Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP).

Proposal for a more rational use of the "impact factor" as an instrument for intra-and inter-area comparison of scientific knowledge
At present, until the appearance of adequate court(s) to establish criteria for evaluating scientific production, it is worth quoting the recommendations of caution expressed by the authors of the SCI: "Caution!"Caution is advisable in comparing journals, especially journals from different disciplines.The journal literature varies in its importance as a means of disseminating information in different fields.Wide citation may be necessary practice in one field, but a redundancy in another discipline because of other means of dissemination.Citation practices differ from one field to another.The difference may be complicated by a difference in the cited half-life of journal literature in different fields, as well as the size of the extant citable literature.Rapid obsolescence may characterize one field but not another.Thus, for example, it would be foolish to conclude merely on the basis of citation counts that the journal of the American Chemical Society is a "better" journal than Annals of Mathematics, or to hypothesize, without a great deal of study, which serves its own field "better".To enable the JCR user to analyze more carefully these data within subject groupings, a breakdown of journals by subject categories with "impact factors" and cited half-life indicators is provided in the "Subject Category Listing" (6).
To overcome these limitations, an adjustment is proposed in order to harmonize discrepancies.The adjustment is not an ideal solution, but may prevent distortions, errors or injustices in judgments in which scientific production -evaluated by the "impact factor" -constitutes the principal element in the decision process.
To illustrate, characteristics of the distribution of the "impact factor" of scientific journals from three areas, Parasitology, Microbiology and Immunology, are presented in Table 1.These areas are considered jointly (Biology 3) during the process of evaluation of Graduate Programs conducted by the Coordenação do Aperfeiçoamento de Pessoal de Nível Superior -CAPES (Brazilian Ministry of Education).The analysis of these distributions (Shapiro-Francia H test for normality) has revealed that, with the exception of Parasitology journals, they are not normal.In these situations, if the data are not transformed, the median should be the measure of central tendency (or location) to be used to characterize or compare distributions.The Kruskal-Wallis test for equality of populations showed a highly significant difference (P = 0.0006) between their medians.In other words, these distributions are not comparable.By transforming the value of the median to 1.000, adjusted "impact factors" can be generated by a simple calculation.For example, in the area of Parasitol-ogy/Tropical Medicine, the adjusted value for the Parasitology "impact factor" would be: where aif = adjusted "impact factor"; m = median, and if = "impact factor" (for values, see Table 2).
The original (from JCR) and adjusted "impact factor" values can be seen in Tables 2 to 4. Comparison of the distribution of the adjusted values showed that 1) they are not normal (Shapiro-Francia H test for normality) and 2) their medians do not differ significantly (Kruskal-Wallis test, P = 0.772).
Since review journals do not publish original data, we suggest that these journals be Continued on next page journals maintain higher "impact factors" compared to other fields.This would compensate for the apparent handicap that the adjustment might have caused to the median level values.Exceptions to this procedure would be journals included in more than one area with different "impact factors"; in these cases, the higher value would be the choice, acknowledging the journal's capacity of being included in more than one area.

Final considerations
The judgment of scientific production quality is a difficult process and controversies regarding its limitations and applications will always be present.In addition to the "impact factor", other indexes, such as the half-life of scientific papers, can also be used to evaluate publications (3).
Presently, an initiative, under the leadership of Drs.Varmus, Brown and Eisen, aiming at creating a "new system based on-line peer review pre-print publishing" is being implemented.It is named Public Library of Science (PLOS); additional information can be obtained through the Internet (www.publiclibraryofsciencemag.org).The Science Now, a subsidiary journal of Science, published an interview with Dr. Eisen, who announced the publication of "PLOS Biology", discussing the policy of "free-ofcharge access for readers".This system is intended to be faster compared to the traditional printed journals.However, the financial dependence of scientific societies on their journals led their Editorial Boards to oppose the idea.Scientists, even those who endorsed this initiative, are still publishing their papers in the traditional scientific journals, probably afraid of losing prestige with the absence of publications in these journals.An additional problem, especially for developing countries, is the publication costs (ca.US$ 1,500/article) (7).
It is expected that this article can contribute to the discussion of the "impact factor", raising questions and motivating different viewpoints to be expressed with the intention of rationalizing its use in the complex process of evaluating scientific production quality.

Table 1 .
Characteristics of the distribution of original and adjusted "impact fator" (from JCR, SCI, ISI).

Table 2 .
"Impact factors" in decreasing order of periodics in the areas of Parasitology and Tropical Medicine and respective adjusted values.

Table 4 .
"Impact factors" in decreasing order of periodics in the area of Immunology and respective adjusted values.