The Impact Factor: its popularity and impacts, and the need to preserve the scientific knowledge generation process

The evaluation process of scientific production based on objective criteria, at the national level, despite its relatively recent deployment in Brazil (1998), and due to its magnitude, was enough to widely disseminate the use of the Impact Factor, making its use very popular, unfortunately. Now, its broadly mentioned limitations should be made popular as well¹.

This indicator, initially proposed in the 1950s², is a result of prior efforts of other researchers³, whose intention, when presenting the bibliographical references of an issue of The Journal of the American Chemical Society, was to identify the most relevant chemistry journals for the library of a small college. This initiative of journal selection had worldwide implications considering that, in the 1970s, the Science Citation Index (SCI)^* started to publish its Journal Citation Reports, widely known by its acronym JCR. This report offered the possibility to reproduce the study conducted in 1927³, as it ranked the journals cited by (or that cite) any of the journals indexed by the SCI. Besides this possibility, the Impact Factor of each journal was presented, as well as the Immediacy Index and Cited/Citing Half-Life, which are currently maintained. Other indicators have been presented in the report, but without the same popularity.

Using the Impact Factor in evaluations means using an analysis that is limited to citations of journals indexed in the Web of Science (a web interface that accesses the SCI and its complementary databases in social sciences, arts and humanities). In addition, it restricts citations of recent articles, as it only considers the first and second years after a study is published. That is, if a journal has articles from a certain year cited one or two years later, the impact of this journal is not taken into account. Is that the way we handle the literature of our field, determining such a short validity period? Then, if that is the case, should the expensive subscriptions of journals be paid for two years only?

Another aspect of concern: in terms of Brazilian science, citations of journals not indexed in the Web of Science will not be considered - when observing the percentage of Brazilian journals cited in articles of Brazilian authors, it is possible to estimate how much information is lost. In addition, an even greater restriction occurs when citations of other types of document are ignored - such as books, conference articles, and dissertations. Should we stop requiring the proposal and defense of a dissertation to obtain the PhD degree? And how about doctorate program students, do they no longer need to build the state of the art regarding their theme of study?

Besides the limitations related to the use of the Impact Factor to the evaluation process, the effects of such limitations should be analyzed in the scientific communication process as well. References are essential for the generation of scientific knowledge. The establishment of the Impact Factor sets off chain deformations, because it favors recent literature to the detriment of older literature - as if the speed of idea maturation process in various fields was fast and constant -, besides prioritizing the literature published in journals and discouraging the citation of books, for instance - which may inhibit the citation of fundamental studies conducted in the field. These effects not only impact authors, but also editors, making publication guidelines influenced by criteria that promote the citations considered by the indicator formulae. What should be considered the most appropriate criterion: scientific habit or the indicator?

Besides being an essential element, the bibliography per se is a source of information, whose value is still greater as it expresses the result of a selection, presenting the most relevant documents for the development of a scientific argument. Thus, if such selection is influenced by artificial criteria, there is interference on the scientific knowledge generation process. The developer of the SCI² himself indicates his intention to facilitate access to the bibliography cited in articles.

The authors of a study published in 2012⁴, when referring to altmetric indicators - indicators based on social media, also called "alternative metrics" -, differentiate the use that can be done of these new indicators: as a filter, being an analogy with the Impact Factor, due to the fact it is used for journal selection and classification; and as a self-evaluation tool, as authors or editors can track the repercussion of their articles inside the science scope beyond, through these new means.

Another study⁵, conducted in 2016, defines this heterogeneous group of "academic metrics" (as it names them) as indicators based on registered events of acts (viewing, reading, saving, diffusing, mentioning, citing, reusing, modifying) related to academic documents (papers, books, blog posts, datasets, codes) or academic agents (researchers, universities, funders, journals).

This new horizon enabled by the digital era allows the measurement of other dimensions of the scholarly communication and, consequently, the scholarly impact. On the other hand, it is very important to be careful so that the adoption of a new indicator will not affect critical acts of scientific knowledge production, such as the undiscriminating use of the Impact Factor. In fact, the community does not need a new indicator, it has to perform the work of documentalists from the beginning of the last century, and handling references would be a good start, so that new indicators could (possibly) be proposed.

References

• ¹
  Vanclay JK. Impact factor: outdated artefact or stepping-stone to journal certification? Scientometrics. 2011;92(2):211-38.
• ²
  Garfield E. Citation indexes for science: a new dimension in documentation through association of ideas. Science.1955;122(3159):108-11.
• ³
  Gross PL, Gross EM. College libraries and chemical education. Science. 1927;66(1713):385-9.
• ⁴
  Wouters P, Costas R. Users, narcissism and control: tracking the impact of scholarly publications in the 21st century. Utrecht: SURF Foundation; 2012.
• ⁵
  Haustein S. Grand challenges in altmetrics: heterogeneity, data quality and dependencies. Scientometrics. 2016;108(1):413-23.

• *
  The Science Citation Index, according to Garfield⁽²⁾, was inspired by Shepard's Citations, a citation index available for American lawyers since 1873, which allowed, for each legal case, to access the publications citing it or legal decisions influenced by it. This initiative for science aimed to help retrieve the origins of an idea and allows the repercussion of an idea in subsequent studies. Garfield describes in detail the methodological challenges while materializing his idea and the need to limit the bibliographical references to be included in the index.

Publication Dates

• Publication in this collection
  Sep-Oct 2016