Brazilian impact factor of physics journals-the third side of the coin

The lack of correlation between the Journal Impact Factors and the most cited Brazilian papers in physics is statistically demonstrated. The existence of an effective "Brazilian Impact Factor" is observed, being its values, in general, lower than the international Impact Factors. In some cases, discrepancies from the international values are huge, leading to doubts on whether it is appropriate to use this indicator to judge Brazilian scientists.


INTRODUCTION
The principle behind the definition of the journal Impact Factor (IF) can be traced back to a classical paper by Garfield (Garfield 1972).Its use worldwide has been boosted by the publication of the Journal Impact Factor index by the Institute of Scientific Information (ISI) since 1972.The original intention in creating the IF was to help librarians in choosing what journals they should order.A collection of papers concerning different aspects of the use of IF was recently published (Scientometrics 2012).
The subsequent history developed in a much unexpected way.As a simple and quantitative criterion for judging scientists, otherwise a quite hard job, IF has seduced many people, particularly those from the science funding agencies around the world.Presently, however, we witness a kind of insurrection against the IF, more intense in the field of biomedicine as well as in developed countries (Marder et al. 2010, Alberts 2013), which has culminated in the recent DORA declaration (DORA 2012).Criticisms have focused mainly on the bad influence the IF fever has on the formation of young scientists, who may feel constrained to avoid risks of innovative researches and fields.This debate promises to extend itself and to become more intense.In the last section the discussion is particularized to Brazilian issues.
The coin has a third side, however.Those who advocate the use of the IF may argue that, due to its transparency, competition among IF-ranked scientists occurs on clear and fair conditions.But while the IF is an international index, based on the total of publications in international indexed journals, there are reasons to believe that it does not apply homogeneously around the world.This suspicion becomes more relevant as one realizes that funding agencies are local!In Brazil, as possibly in other scientifically peripheral countries, the main governmental agencies for scientific development use the IF as a determinant factor to decide who is going to get money for research and who is not.The principle is simple and apparently correct: the most impacting scientists should have the money.A hasty corollary is the source of possible distortions: the most impacting scientists are those who publish in the journals with the largest IF.
In fact, scientists in these countries, Brazil in particular, usually have more difficulties in getting published in high-IF journals than their colleagues in the international scientific hard core.The reasons are multiple and it is not our objective to discuss them here.Instead, we want to quantitatively show that, as a result, in these peripheral countries the IF may not only be the presently criticized metrics but in fact, it may actually be a fake.In other words, not even the fairness and transparency of the competition based on the IF are assumptions that can be taken for granted.This proposition is developed on quantitative grounds in the next sections.

PHYSICS CORRELATE WITH IF?
Take the N p papers published in a journal in two subsequent years, say 2010-2011, and the N c citations of these papers in the following year, 2012, for example.The two-year 2012-IF of that journal is defined as Hence, in view of the proportionality between IF and N c , the answer to the subtitle question should be an obvious yes.But, once we consider a representative sub-system of the whole scientific community it seems that this implication cannot be assured.
To check this hypothesis we collected data of the most cited papers on the Brazilian physics community published in 2012.The field of physics was chosen in view of its major international insertion in Brazil, being therefore reasonable to consider that for other fields our conclusions could hold or even step up.
The systematics of the search in ISI Web of Science (WoS), made in April 26, 2014, is as follows: Address: Brazil or Brazil; Time span: 2012; Databases: Science Citation Index and Conference Proceedings Citation Index -Science; Research Area: Physics; Document types: articles and proceeding papers; group-authors: exclude all.
The exclusion of group-authors papers aimed at restricting our search to the physics papers having substantial contribution from Brazilian physicists.Since the "exclude group-authors" facility of WoS does not eliminate all group papers, a one-by-one check had to be performed and papers with more than 15 non-Brazilian authors were arbitrarily eliminated.Note that review papers were also ruled out at this stage in view of them not being specific to our present goals.For the sake of comparison we performed the same search for papers originated in the UK (just changing, in the WoS search, the address to England or UK).UK was chosen randomly among the forefront countries of scientific production.The data are plotted against the IF in Figure 1.
A simple linear analysis exposes relevant comparative aspects (note that if all papers had been included, the centroids of N cit for each journal would lie on a straight line).The linear analysis of the UK results, in green (fit line in black), are as expected, with a correlation factor r 2 =0.095 and slope=0.13,which are quite reasonable values in view of the IF definition.On the other hand, in the Brazilian case, in red (fit line in blue), the lack of correlation meets the eye and is confirmed by the quite vanishing correlation factor r 2 =0.000013 and slope=0.001.Furthermore, these results also mean that in 2012, for the same number of citations, on average Brazilian physicists published in journals of smaller impact factor than the UK physicists.Though not shown, if we go back in time, this feature becomes clearer.
Even though this linear correlation analysis is not enough to lead to definitive conclusions, it clearly points out that something is wrong and justifies the next procedure.

THE EFFECTIVE BRAZILIAN IF
The previous results, if extended to all years, can only be explained once we devise a particular Brazilian IF for the physics journals.To check this proposition in quantitative grounds we first calibrate our procedure, trying to replicate the published 2012-IF (symbolized simply as IF in what follows) of the most important physics journals, our IF calc .The Create Citation Report tool on WoS allows us to do it by the IF definition, eq.(1).Following the guidance at the Journal of Citation Reports (JCR) website we restrict N c to the total of citations to articles, proceeding articles and reviews, the socalled scholar papers.Even though, the fact that N c and N p come from different sources, JCR and WoS respectively, prevents us from obtaining the exact published IFs.Our values are systematically smaller, but most of them are close enough to our purposes (see Tables below).
Following the same systematics we then define two Brazilian IFs: one including reviews and group-authors papers, IF Braz1 , and another excluding them, IF Braz2 .Dozens of physics journals have been searched, advancing a general rule that effective Brazilian IFs are almost systematically lower than both IF calc and IF.Again it is pointed out that Brazilian papers are less cited than the international average.On the other hand, it is not our intention here to rank all physics journals from our particular systematics so that, in the following tables, we sample just a few singular and illustrative cases.For some important journals the number of Brazilian papers published in 2010-2011 is not enough to grant reliable statistics.For them we thus focus on the similarly defined 5-year (2007-2011) IF, fixing an arbitrary minimum of 15 papers to enrol a journal in the search.In Table I we show some journals, distributed in sub-areas of physics, presenting large variations of the IFs.For the sake of consistency, the percentual variation of IF Braz2 in the last column is calculated with reference not to IF but to IF calc , except for JACS.In this single case, the value of N p exceeds 10,000, thus being unavailable, therefore we refer to IF for the percentual variation.Unexpected variations above 40%, are detected, reaching values as large as -59%.The exclusion of one review paper for Nano Letters could be seen as contributing to overthrow its IF Braz2 to an artificial value.To check this hypothesis one would need to know the weight of this type of article in the construction of the international IF.Anyway, the fall of both IF Braz1 and IF Braz2 in the other cases is visible (the elimination of group-authors papers is better analyzed in the following table).Against the main trend, a remarkable surprise is the performance of Icarus, with a variation of +73%.
In Table II   Table II also shows the mostly more important IF-based Brazilian government ranking of physics journals, namely the Qualis factor (Qualis), and how this ranking would change (follow the arrow) in case IF Braz2 were to be considered.The Qualis ranges for this ranking are: A1 (FI ≥ 6.0), A2 (6.0 > FI ≥ 3.5), B1 (3.5 > FI ≥ 2.0), B2 (2.0 > FI ≥ 1.5).

DISCUSSION AND CONCLUSIONS
In recent years some works appeared under the scope of the evaluation of scientific work in Brazil, which deserves to be emphasized here.Meneghini et al. (2008) showed that papers originated from Latin America with no collaboration from authors in the first world are systematically less cited.Rocha e Silva (2010, 2011) have harshly criticized the construction of the Qualis ranking, proposing further evaluation paradigms, which are more focused on the quality of the articles and not on the journal FI.He also considered possible new metrics excluding review articles and a reassessment of local journals.Finally, Meneghini (2011) used data consisting of Brazilian publications in physics to approach the evaluation of institutions instead of individual researchers.
This literature differs essentially from the present work in the sense that here we did not necessarily put into question the appropriateness of the use of the IF but, instead, we question its calculation method.Nevertheless, some common points meet the eye.There is a clear connection between the observation of Meneghini et al. (2008) that the IF of a journal is affected detrimentally by publications from Latin America and the existence of an effective Brazilian IF.Furthermore, the exclusion of group authors exemplified here, (not assessed by previous criticism) in the construction of IF Braz2 does increase the incidence of purely local papers.Thus, in some aspects IF Braz2 also appears as a possible new metric designed to more positively evaluate local scientific production, as recommended by Rocha e Silva (2010, 2011).On the other hand, IF Braz2 is unable to meet the prospect of a particular evaluation of local journals or to assess the particularities of different areas of knowledge, advocated in Rocha e Silva (2010,2011).Finally, a particular point, the exclusion of review papers (from any evaluation index), appears as an overall agreement.
In conclusion, ranking scientists by IF implies a reversal of values by which the scientific journals turn to lend prestige to their authors, not the opposite as would be normally expected.We showed here that this process is even more harmful for science in Brazil, perhaps in other peripheral countries as well.
In fact, Figure 1 clearly shows that either Brazilian papers are i) less cited or ii) are published in journals of lower IF; presumably both.The tables show that different areas of physics have different international insertion of Brazilian scientists.Furthermore, the participation of group-authors papers and, in a lesser extent, review papers, is unbalanced.
These factors, together, lead to effective Brazilian IFs of physics journals, which in general are smaller than the published ones, but with quite different percentual shifts.We thus conclude that the international published IF does not properly depict the quality of the physics research in Brazil, a conclusion that could be extended to other fields and to other developing countries.These features show that the IF-based ranking of physicists and its consequence, the distribution of scientific resources in Brazil can be being strongly distorted.

Figure 1 -
Figure 1 -Journal impact factor (IF) versus number of citations of papers published in 2012 from Brazil (red triangles, blue linear fit) and UK (green circles, black linear fit).For clarity data are limited to IF>20 (UK) and IF>10 (Brazil).Data collected in April 26, 2014.
, which shows the 2-year IFs for some other journals, we see that group-authors raises the IF Braz1 of Physical Review Letters, Physics Letters B, New Journal of Physics and Journal of High Energy Physics to values even

Table I ,
Table II is dominated by journals grouped in a sub-area of physics, roughly Nuclear and Particle Physics.The journals in this sub-area present the highest decreases of IF Braz2 .