The tyranny of the impact factor : why do we still want to be subjugated ?

A collective obsession with the Impact Factor (IF) has led to major changes in political science in many countries, and in recent years, these changes have been especially visible in Brazil. Despite critiques of both this measure’s ability to evaluate the scientific publications and its spread in the evaluation of researchers and institutions, Brazil has taken the IF as an important element in evaluating Brazilian graduate programs. In this article, we briefly try to demonstrate that the IF can be subject to many biases that seem to be completely ignored or unknown.


Introduction
We have certainly witnessed a great change in the way that we regard scientists and, indeed, practice science itself.Though these questions are perhaps as old as scientific debate, the recurrence with which some topics arise throughout history is truly impressive.Without a doubt, one of these important recurring topics is the evaluation of scientists by the Impact Factor of the journals in which they publish their work (see Colquhoun 2003).
In recent years, especially in Brazil, the Impact Factor of journals has received considerable attention, because it is one of the criteria used to evaluate the quality of graduate courses in Brazil, and therefore, to evaluate scientists.However, it is necessary to emphasize that the IF does not evaluate the quality or output of scientific work, but only its visibility.While there is certainly no consensus on this subject, the IF is supported by a great majority of scientists; unfortunately, it is not clear to young students and new researchers whether an evaluative policy for staff members will be adopted based on the Impact Factor.Surprisingly, despite all of its shortcomings, many of which have been previously elaborated upon (e.g.Kokko & Sutherland 1999), the IF is increasingly used to evaluate institutions and to determine how resources will be allocated (see Ruiz et al. 2009).
According to Kokko & Sutherland (1999: 382) "the impact factor is the average number of times that articles published in a specific journal in the two previous years (e.g. 1997-1998) were cited in a particular year (e.g.1999)".By accepting the impact factor as a measure of quality, a prevalent view among many groups and institutions2 (see the criticism by Kokko & Sutherland 1999), a message is sent that articles published in high impact journals are superior to those published in low impact journals or a journal with no impact at all (see the discussion in Leimu & Koricheva 2005).However, we must clarify that this is just one of the understandings on the IF.Other researchers assume that the IF is a measure of visibility of a publication, and quality and visibility certainly not always go together.While not going together, implications of terms in practice are quite different.In this article, I assume that IF is a measure of visibility, which many researchers and institutions take as a measure of quality (see notes below).Thus, every time we refer to the IF, we will be giving the visibility connotation.Box 1 presents a discussion of questions the IF can help answer.
If we then assume that citations constitute an important and robust variable in the evaluation of science, we should also assume that these citations are not influenced by other factors or variables (see Fig. 1).If other factors do have an influence over the number of times that a work is cited, we must necessarily acknowledge that the Impact Factor is not a good measure.Thus, this article aims to reinvigorate the discussion of how citations are used in the calculation of the Impact Factor (IF) and to show how the number of citations of a work seems to be a not good variable (in the way it is used today) to stratify scientific production and scientists.While it is understood that citations strongly influence calculations of the IF, it is not assumed here that the two are equivalent.It should be noted that this topic is very complex and controversial; my arguments will be developed using specific case studies drawn from the biological sciences (especially from ecology).Moreover, the IF is susceptible to several factors that may not be related to citations.

Are citations influenced by the number of authors of an article or by gender?
We define publication bias as the situation in which the merit of a manuscript is evaluated using factors tangent to its intrinsic characteristics as a scientific work (Fig. 1).It has been observed that male authors attract more citations than female authors (Cassey & Blackburn 2003, 2004).This relationship was shown to be insignificant in the works of Leimu & Koricheva (2005) and Borsuk et al. (2009) in an analysis of publications in the field of ecology.The author's gender also does not seem to affect the rejection rates of articles (Aarssen et al. 2008).Nonetheless, Tregenza (2002) reached a conclusion that is quite interesting: there are differences related to gender in the acceptance rate of works that would be difficult to explain without accepting the idea of bias.
A significant relationship has been found between the number of authors of an article and the citation rate (see Tregenza 2002).This suggests, for example, that when there are more authors on an article the possibilities of self-citation also increase (Borsuk et al. 2009).However, there are measures that may be used to avoid bias caused by 3 When I finished writing this article, a strong debate about the high Impact Factor that a Brazilian journal achieved in 2009 (3.46) started to circulate in the internet.There were some discussions that this value was only reached because of the high number of self-citations incentivized by the editors.This episode only shows the great fragility of the IF, how it can be easily manipulated and this was not a unique episode: self-citations are current practices in many international journals.So, why do we still want to be subjugated by this failure system? the self-citation effect 3 .Borsuk et al. (2009) suggest that the number of authors could reflect: a) the external perception that multi-authored works have better quality, as they contain the collaboration and vision of different people, or b) that the scientific community "perceives" greater merit in the work; after all, "how many people could be deceived?"Figure 1 -A conceptual illustration of the publication and dissemination process in ecology with potential publication related biases listed to the right.Attributes at each level of the process are listed as within study (i.e.attributes of the study itself and subsequent interpretations by the author), publication level (properties of the published study), and journal level (associated attributes of the Journal where in a given study is published).Citation frequency refers to the number of times a particular publication is Referenced in other publications; however, the impact factor attribute of a journal is also calculated using mean citation frequency of all papers published there in within the last two years and also best conceptualized as an emergent property.The biases listed are generally recognized within ecology and sometimes quantitatively analyzed and published.Nonetheless, few analyses have explored either the potential relationships between attributes of the publication process, as denoted by the dotted arrows, or between the attributes of the process and the various biases which may inûuence relative success or performance of a study/publication, as denoted by the double-headed arrows.Source: Lortie et al. (2007Lortie et al. ( : 1250)).Reproduced with the permission of the publisher: John Wiley and Sons (License number: 2482660790074).
Regardless of the possible explanations, it is evident that collaboration in scientific works brings many benefits, such as those cited above.Leimu & Koricheva (2005) add that this relation increases when the number of authors exceeds four.Curiously, these authors also found that interdisciplinary collaboration enhances the citation rates of ecological studies, though this is not true if the collaboration involves different departments within the same institution.Also, in studies on the field of ecology, an interesting article by Leimu & Koricheva (2005) states that a positive relation exists between self-citation and external citation.
In these times of economical crisis, there is an increasing pressure on scientists to increase their publication output, and misconduct, such as including authors who did not directly participate in the production of an article, has become the norm.For more on the consequences of this pressure to increase one's number of publications, see Martinson et al. (2005) and Albuquerque (2009).

Are citations influenced by the name of the author or his language?
According to Tregenza (1997), "Darwin may be a better name than Wallace!" Tregenza suggests that a significant relationship exists between citations and the initial of the last name of the author in a scientific publication.However, Leimu & Koricheva (2005) were not able to reproduce these findings in relation to the alphabetical order of the last names of authors in ecology.These contradictory results suggest that there is a need for new evaluations in other areas of research.
Authors whose native language is English attract more citations than authors whose native language is not English, even when publishing in English (Leimu & Koricheva 2005).This assertion has been reinforced by the findings of Borsuk et al. (2009).In addition, this factor appears to affect not only the citation rates but also the probability of an article being accepted in ecology journals (Tregenza 2002).
Tregenza expresses surprise at this pattern when he states that "Patterns of acceptance relative to the country of author affiliation are intriguing and might indicate that non-English speaking nations are at a disadvantage either as a result of discrimination or perhaps because of the additional burden of having to write in a foreign language".(p.350).
One could argue that this might reflect a higher requirement on the part of journals to accept works written in higher quality English.Considering that most journals that have been classified by CAPES as possessing impact are not national, could this possible bias not be an impeditive factor that hampers the publication of Brazilian researchers?Meneghini et al. (2008) obtained a disturbing finding: there is a tendency for Latin American authors to be less cited in journals with high impact factors, except when there is international collaboration.Although the authors argue that it is currently difficult to define the reasons for this phenomenon, they do suggest two possible reasons: psychosocial source bias (discrimination) or real differences in the scientific visibility of these articles.An anonymous reviewer, who was commenting on this paragraph, highlighted important issues that also deserve our consideration in the statement: "Has this to do with bias, or has to do with the fact of science in Latin America not having historically concern about its international visibility?In other words, is it guilty of a Machiavellian system or is it simply our historic failure?" (p.4).

Final considerations
The aforementioned examples, based on systematic studies, were highlighted to engender debate without passing premature judgment.However, the writer's inclination is to be severely critical of applications of the IF.It has become evident that evaluations based on the IF may suffer from a strong bias, causing research to be judged not solely on its scientific merit but also on attributes of the authors, nationality, the existence of cooperation, etc. (see Lortie et al. 2007).
With this assumption, we should necessarily push for debate and discussion of the validity of the IF.Moreover, I believe that a culture of reflection and self-criticism should be encouraged.It is clear that scientists publish their results to be read, cited, and remembered (Abt 1998); however, the discussion herein shows that accomplishing these goals, as measured by the Impact Factor, depends upon more than the quality or visibility of the work.Abt (1998) states what many people already know for certain: the number of citations is influenced by the field of knowledge (velocity and growth) and even the length of the research itself!This debate is obviously very rich as well as conducive to the enrichment of scientists knowledge.To further illustrate the complexity of these issues, consider the statement by Simkin & Roychowdhury (2003) that many authors categorically do not read the papers they cite.They estimated that only 20% of authors have read the work they cite.They are several motives (psychological, sociological, political, historical, etc.) influencing an author's decision to cite a study, and this is an object of ongoing interest by scholars in various fields (see Brambilla et al. 2006).The latter authors reached this interesting conclusion: "It was found through content analysis, that all works cited an article for the same reason: a description of theoretical and practical methods to solve problems.However, subjective reasons permeated the act of quoting, and only a more accurate analysis showed such reasons.It was observed that the article was also cited because their authors share a common paradigm and build expertise in network and collectively.This could be verified because the citing articles were written in co-authored by the group, in collaboration with researchers from other Brazilian and foreign institutions, and were published in national and international journals with importance in the area)."(p.206).
Thus, it becomes more evident the complexity of the issue and the need to treat it with care and seriousness.Finally, to conclude my discussion on the IF, Colquhoun (2003) categorically states that: "Eugene Garfield, who invented the wrecheted impact factor, himself said that it is not appropriate for ranking individuals (...).Astonishingly, these facts are not known (or are ignored) by some selection committees 4 ."(p.479).Colquhoun (2003) has a curious example of having published a work in Nature (with an impact factor at the time of 27.9) and having only been cited 57 times, while another work published in a journal of less impact (3.1) attained more than 400 citations!This indicates that the publication of a work in a high impact journal is not a guarantee that the work will achieve the much-desired dream of the author: to be read, cited, and remembered.The data of the ISI (The Institute of Scientific Information), for Nature in 2001, reinforce this.In 1999, of the 858 works published, the data show that the 80 works most cited comprised half of all citations!(Colquhoun 2003).This finding is not new, rather it is typical.As a result, many began to adopt the h-index 5 as a measure of authors' visibility.According to Scarano (2008):"While evoking the need of better ways of assessing scientific literature, the editors admitted they would be lying if they said they were not interested in their journal's impact factor.This exemplifies that despite one's awareness of the limitations and flaws of impact factor judgement, it is simultaneously the most practical way of producing any type of objective analysis of scientific value." (p. 191).
I have many doubts about the statement above.Everyone can agree that objective and fair evaluation criteria are required.The limits of any measure should also be widely appreciated to avoid any potential inconsistencies.While I would sincerely like to end this text with a proposal that mitigates the shortcomings discussed, I will conclude by repeating the topical question, "What will be then the best existing alternative?" 6and by suggesting that this issue should be discussed more openly within the scientific community.While this fact may be known by many development agencies in the world, the criticism is valid because many people remain unaware. 5The h-index was proposed by Hirsch (2005: 16569).Thus the author stands on its proposal: "This large amount of information will be evaluated with different criteria by different people.Here, I would like to propose a single number, the "h index," as a particularly simple and useful way to characterize the scientific output of a researcher.A scientist has index h if h of his or her Np papers have at least h citations each and the other (Nph) papers have <h citations each".Some criticisms were made use of this measure to assess "the cumulative impact and relevance of an individual's scientific research output" (Hirsch, 2005: 16569).See Zhang (2009) for criticisms and suggestions for improvement of the h-index.

6
The well-known failings and limitations of the IF have led to many proposals for improved measures (examples in Dong et al. (2005)). 4