Scientometrics: Nature Index and Brazilian science

ABSTRACT

A recent published newspaper article commented on the (lack of) quality of Brazilian science and its (in) efficiency. The newspaper article was based on a special issue of Nature and on a new resource for scientometrics called Nature Index. I show here arguments and sources of bias that, under the light of the principle in dubio pro reo, it is questionable to dispute the quality and efficiency of the Brazilian science on these grounds, as it was commented on the referred article. A brief overview of Brazilian science is provided for readers to make their own judgment.

Key words:
bibliometrics; Brazil; internationality; Journal Impact Factor; research; science

A recent published article by Dr. Rogério Cezar de Cerqueira Leite in the Folha de S. Paulo (Leite 2015) commented on the (lack of) quality of Brazilian science and its (in) efficiency. The newspaper article was based on a supplement of Nature [2014; 515 (7526): S49-S108; see http://www.nature.com/nature/supplements/nature-index-2014-global/index.html] and on a new database for scientometrics called Nature Index (NI; Nature Index 2016), which encompass three metrics. In one of the alarming highlights of the article, Brazil's scientific efficiency was compared with Chile's (15 times more efficient than Brazil): in 2013 Brazil spent US$ 30 billion in Science and published 670 articles "in the most rigorous journals regarding to its scientific quality," while in the same period Chile published 717 articles in those same journals while investing only US$ 2 billion.

The Nature Index (2016) encompass three metrics:

1. Article count (AC): A count of one is assigned to an institution or country if one or more authors of the research article are from that institution or country, regardless of how many co-authors there are from outside that institution or country;

2. Fractional count (FC): A fractional count that takes into account the percentage of authors from that institution (or country) and the number of affiliated institutions per article. For calculation of the FC, all authors are considered to have contributed equally to the article. The maximum combined FC for any article is 1.0;

3. Weighted fractional count (WFC): A modified version of FC in which fractional counts for articles from specialist astronomy and astrophysics journals have been down weighted. These journals encompass a much larger proportion of the total publication output of these fields than any other field covered by the Nature Index. The WFC allows ordering of institutions and countries so as not to give undue emphasis to these fields. The weighting is achieved by multiplying the fractional count from these astronomy and astrophysics journals by a factor of 0.2. This down weighting is in proportion to an approximation of the level to which astronomy and astrophysics articles are overrepresented compared to the total publication output of other fields covered by the Nature Index.

These three metrics use the number of publications that come from institutions and countries in selected journals. The selection of these 68 journals was carried out by polling 68 scientists for their 10 favorite journals in which they would publish their best scientific works. There was an attempt to confirm the 68 journals by polling via email 100,000 life, physical, and medical scientists, allegedly from a broad geographical mix. Only 2,800 (2.8%) out of 100,000 scientists answered the questionnaire. The 68 journals, which are part of the NI, constitute less than 1% of journals in the Journal Citation Reports (2013 JCR, ISI Web of Knowledge, Thomson Reuters), but are responsible for about 30% of the citations in natural sciences. Interestingly enough, journals that are recognized as the top 10 of the 100 most influential journals of the 20th century in biology and medicine are not part of the journals list of the NI (e.g., The New England Journal of Medicine, JAMA, BMJ, The American Journal of Botany, The Journal of Zoology, The American Journal of Physical Anthropology and The Journal of Paleontology; see http://dbiosla.org/publications/resources/topten.html).

Similar to other metrics of science (e.g., Impact Factor, h-index), NI has limitations:

1. The main criticism for NI is that there is no apparent reason for the selection of journals. Moreover, it is not entirely clear how the different disciplines were considered in the selection of journals. The NI may represent a metric only for Chemistry, Earth & Environmental Sciences, Life Sciences and Physical Sciences; other fields like Humanities or Social Science, NI does not cover.

2. In an attempt to verify the set of 68 journals by email survey, 97.2% of the 100,000 scientists did not reply. The low response rate may indicate that the NI does not seem to be important for several scientist respondents (assuming the scientists knew the reason for the research). In addition, it is unclear which sampling techniques were used for the selection of the 100,000 scientists, e.g., sample size calculation, sample stratification, randomization, sample loss, statistical techniques applied, etc.

3. The indicators depend more on the absolute number of publications of an institution or country in the selected journals than on the methodological quality or impact of these publications. Furthermore, it would be more logical to measure performance based on individual activities rather than journals.

Given the limitations discussed here and based on principle of in dubio pro reo, it is questionable to dispute the quality of Brazilian science and its efficiency based on the assertions given by Dr. Leite. Moreover, there is a high risk of ecological fallacy in the comparison regarding efficiency in the use of resources applied in research. The US$ 30 billion invested should yield results in the middle and long term, so probably not implying in the 670 published articles. Considering only the NI, there is limited evidence regarding the quality and efficiency of Brazilian science; although the 2012-2015 change in WFC, the main metric of NI, showed an increase of 18.9% for Brazil, ranked 24th globally, 8 places ahead of Chile (Nature Index 2016).

To be fair, Chile is a noteworthy example of science internationalization. Chilean scientists tend to publish high-impact papers in collaboration with international teams in large expensive projects coordinated and funded from abroad. Based on the ratio of AC to FC (Nature Index 2016), there is an indication of the internationalization degree: Chile: 1069/212.70 = 5.0; Brazil: 960/304.33 = 3.2. The superiority of Chile's science internationalization is similar when analyzed based in data from SCImago (2016) (see ^{Figure S6}). On the other hand, it may also indicate that Chile's dependency on external resources to do science is about 1.6 times higher than Brazil.

A brief overview of Brazilian science production is provided for readers to make their own judgment (see ^{Supplementary Material}). The data are based on 29,713 journals and 239 countries/settings contained in the Scopus(r) database (Elsevier B.V.) and retrieved by the scientometric portal SCImago Journal & Country Rank (SCImago 2016). Figures S1 to S3 are exclusively about Brazilian science production and show analyses of the citable papers (^{Figure S1}), number of citations received (^{Figure S2}) and average of cites per paper (^{Figure S3}). ^{Figures S4} to ^S6 allow comparing Brazilian science with world and Latin America (^{Figure S4} and ^{Figure S5}), as well as with Chile (^{Figures S5} and ^{Figure S6}).

Finally, I agree with Dr. Leite that much can be done to help Brazilian science grow not only in quantity, but also in quality (e.g., less bureaucracy, more internationalization, more meritocratic and well-managed financial incentives). Another important issue to highlight and solve is that, formally, the profession of scientist in Brazil is not a recognized position by the Ministry of Labor and Employment.

ACKNOWLEDGMENTS

I thank Carolina Gomes Freitas (Universidade Federal de São Paulo, São Paulo-SP, Brazil) and Francisco Silva Neto (Universidade de São Paulo, São Paulo-SP, Brazil) for their comments and suggestions. I also thank to the reviewers for their helpful and valuable comments and suggestions.

REFERENCES

• LEITE RCC. 2015. Produção científica e lixo acadêmico no Brasil. Folha de S. Paulo, São Paulo, p. A3, 06 jan. 2015. Available at Available at http://www1.folha.uol.com.br/fsp/opiniao/202892-producao-cientifica-e-lixo-academico-no-brasil.shtml (accessed January 14, 2015).
  » http://www1.folha.uol.com.br/fsp/opiniao/202892-producao-cientifica-e-lixo-academico-no-brasil.shtml
• NATURE INDEX. 2016. Available at Available at http://www.natureindex.com/ (accessed on July 19, 2016).
  » http://www.natureindex.com/
• SCImago. 2016. SJR - SCImago Journal & Country Rank; 2007. Available Available at http://www.scimagojr.com (accessed on July 17, 2016).
  » at http://www.scimagojr.com

SUPPLEMENTARY MATERIAL

Figure S1 Number of citable papers* published by Brazilian scientist. * Citable papers include exclusively articles, reviews and conference papers. Figure S2 Number of citations received by papers published by Brazilian scientist. Figure S3 Average citations received by papers published by Brazilian scientist. Figure S4 World ranking of h-index*. NL – Netherlands; CH – Switzerland; AU – Australia; BR – Brazil; MX – Mexico; AR – Argentina; CL – Chile; CO – Colombia; PR – Puerto Rico; VE – Venezuela; PE – Peru; PA – Panama; CR – Costa Rica. * The h index is a country’s number of articles (h) that have received at least h citations, e.g., Brazil have 412 papers that have received at least 412 citations. It quantifies country scientific productivity and scientific impact. Figure S5 Contribution* of Brazilian and Chilean scientists to the proportion of published papers in the Latin America and world. * Country relative contribution to world and Latin America publication output. Figure S6 Comparison of internationalization* between Brazil and Chile. * Papers ratio whose affiliation includes two or more countries address.

Publication Dates

• Publication in this collection
  Sept 2016

History

• Received
  27 Jan 2015
• Accepted
  21 July 2015