Contributions to the debate on the assessment of scientific research production

In the reviewing of the procedures of the assessment of graduate studies programs, the Assessment Division of the Brazilian Graduate Studies Coordinating Board (Capes) has established new criteria for classifying scientific journals in the “Qualis Journals” system (Memorandum n. 6/2019-CGAP/DAV/CAPES). The methodology proposes a unique reference classification for journals (“Qualis Único”), based on the combined use of the bibliometric indicators CiteScore (Scopus), Impact Factor (Web of Science), and h5 (Google Scholar). In the case of journals not indexed in Scopus and Web of Science, a regression model was fitted to establish the relationship between the h5 index and CiteScore. The classification is based on the percentile in which the journal is situated in each database, according to the thematic category. This percentile is divided into 8 groups (for each 12.5% of the total). It is possible to modify the classification of up to 30% oh the journals in each area.

The premise for the adoption of a unique assessment criterion for scientific journals is questionable, considering the profound differences among academic fields in the production and communication of knowledge, which may caused disastrous results for consolidated journals in their specific fields. The process also lacked transparency and participation by the respective academic communities. It is also necessary to situate the context in which this proposal is formulated.

Research assessment, which is certainly essential, is permeated by different interests ^1,2: large corporate groups, small publishing houses and research centers, and regulatory and funding agencies. The science publishing market is among the most lucrative in the world. The Elsevier publishing company, for example, reaped a 36% profit in 2010, outstripping Apple, Google, and Amazon ³. Publishing fees and journal subscriptions represents 45% of the revenue, with the public sector covering another 31% ⁴.

In Brazil, Capes plays a fundamental role in the assessment of journals, which in turn serves the assessment of graduate studies programs, impacting their financing and the availability of student scholarships. This function has been based on the use of bibliometric indicators, developed for purposes unrelated to assessment of the quality of science production.

The incorporation of these indicators favors the prioritization of themes that interest the science policy of Northern countries (mainly the United States and United Kingdom), and the adjustment of the content published by journals from nonhegemonic countries ⁵ to what generates citations in international literature bases. Publishing in English, necessary for this objective, rules out the science produced in Brazil for unspecialized readers. It also hinders the journals’ important role in making current scientific knowledge available to support training at various graduate studies levels, including executive Master’s programs, whose social impact, indispensable in key fields, is not amenable to recognition and assessment by these metrics.

The assessment model impelled by these indicators is based on a fetishization of excellence backed by a narrative of scarcity ⁶, not supported in the face of online publishing resources such as preprints (https://blog.scielo.org/blog/2018/09/21/pkp-e-scielo-anunciam-desenvolvimento-de-um-sistema-de-codigo-aberto-de-servidor-de-preprints/), and which inflates the prices of so-called high-impact journals (such as Lancet Infectious Diseases, where publishing an article cost USD 5,000 in August 2019; http://www.thelancet.com/pb/assets/raw/Lancet/authors/tlid-info-for-authors.pdf). The model thus stimulates predatory competition between scientists, between graduate studies programs, and between journals.

Equally important are scientists’ collective criticisms of the bibliometric indicators, contained in the international guidelines for the assessment of research output, such as DORA (San Francisco Declaration on Research Assessment) ⁷, and the Leiden Manifesto, crystallized at the 19^th International Conference on Science and Technology Indicators (http://sti2014.cwts.nl) ⁸. DORA highlights two major limitations: impact can be manipulated, and even within the same journal, the articles receive an extremely varied number of citations. Meanwhile, the Leiden Manifesto warns of the harms to the science system resulting from the pervasive and inadequate use of these indicators, while proposing ten principles to guide science assessment.

We comment on the Capes proposal in light of the 10 principles in the Manifesto:

1. Quantitative evaluation should support qualitative, expert assessment - the Qualis classification should be defined after establishing a qualitative assessment model;

2. To measure performance against the research missions of the institution, group, or researcher - the policy of incentives for graduate studies should condition the assessment; standardization of a unique indicator deepens the inequality between regions and between fields of knowledge, becoming a real obstacle to the incentives policy for graduate studies;

3. To protect excellence in locally relevant research - preserving pluralism and relevance for society, not subordinating the assessment to publication in journals indexed in the selected bases;

4. To keep data collection and analytical processes open, transparent, and simple - the databases used for the initial classification and the journals’ classification by fields should be openly available;

5. To allow those assessed to verify data and analyses - based on the available crude data, it should be possible to perform comparative analyses and study the proposal’s impact on different fields of knowledge;

6. To take into account the differences in publication and citation practices by field - “Qualis Único” prevents the selection of a broader set of indicators and does not allow the different fields to define the indicators that are most adequate for them. Besides, classification based only on bibliometric indicators impair new journals and articles published in Portuguese;

7. To support the assessment of individual researchers on a qualitative judgment of their portfolio, a principle that can be extended to research institutions, programs, and groups and that should consider the research output over time and not only in the two or three years included in the indicators;

8. To avoid misplaced concreteness and false precision - where it is appropriate to question the precision of 12.5% cutoff points;

9. To recognize the systemic effect of assessment and indicators - such aspects condition not only financing but also the graduate studies programs’ capacity to meet this criterion, whether by forcing students to have articles accepted before receiving their degrees, by decreasing the number of articles published in the journals in order to inflate the impact indicators, or by adjusting programs to the proposed criterion, including or excluding students, or even encouraging publishing in better classified journals, even if they are outside the respective knowledge core field;

10. To scrutinize indicators regularly and update them - a guideline met by Capes, although not debated collectively.

Some problems in the immediate application of the Capes proposal have already been identified. The definition of the core field - the graduate studies field with the majority of articles published in the journals - is heavily influenced in the case of interdisciplinary fields of knowledge and by the field with the most graduate studies programs and students. In the field of collective health, 11 journals were immediately reassigned, and several more are in the same process, indicating the weakness of the so-called “objective” criterion that was adopted. In other fields of knowledge, markedly interdisciplinary, the journals have been classified in inappropriate core fields as the result of automatic accounting that fails to consider their identity and target public.

The allocation of journals not indexed in Scopus and Web of Science based on the use of h5 (Google Scholar), known to be more inclusive, is also problematic. Despite the limitations addressed above, basing the allocation of journals not indexed in the other databases on a correlation of only 0.5 means more ad hoc decisions.

Highlighting that the function of citation measures is only justified in the long term ⁹: how is it possible to compare the impact of an article that was published in the two or three previous years and cited in the reference years with an article that received citations over the course of 20 years? Why limit the assessment to indicators whose intention is to encourage ever-growing competition between so-called high-impact journals and within a limited timeframe? Other scientometric indicators can contribute to assessing journals. The assessment of journals’ inclusion in the SciELO base ¹⁰ is a quality criterion and assigns privilege to open-access publications, an important condition for national research output. The total number of citations a journal received in the year, also available in Google Scholar, independently of the year of publication, reflects the journal’s accumulated impact in science communication. Various altimetric measures are available, for example the number of article downloads in SciELO.

It is possible to anticipate the proposal’s consequences: drainage of articles to fields with journals in classes A1 and A2 (and whose publication rates are reasonable); a decrease in submission of articles from well-assessed graduate studies programs, with more resources to pay for publication in journals from large international publishing houses; restriction of financing for journals, in a process of “more for those with more”. The model encourages competition rather than solidarity between peers, as well as undermining newer journals.

This process in Brazil is associated with a context of diminishing resources and the production of discredit to science ¹¹, in which a single individual’s observations are considered equivalent to scientific evidence. Considering the critiques and the debates on the topic, it is necessary to ask how long we will adopt classifications that contribute little to assessing the quality of knowledge production.

• ¹
  de Camargo Jr. KR. Big publishing and the economics of competition. Am J Public Health 2014; 104:8-10.
• ²
  Guédon J-C. Future of scholarly publishing and scholarly communication: report of the expert group to the European Commission. https://doi.org/10.2777/836532 (acessado em 03/Ago/2019).
  » https://doi.org/10.2777/836532
• ³
  Buranyi S. Is the staggeringly profitable business of scientific publishing bad for science? The Guardian 2017; 27 jun. https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science
  » https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science
• ⁴
  Lawson S, Gray J, Mauri M. Opening the black box of scholarly communication funding: a public data infrastructure for financial flows in academic publishing. Open Library of Humanities 2016; 2:e10.
• ⁵
  Losego P, Arvanitis R. La science dans les pays non hégémoniques. Revue d'Anthropologie des Connaissances 2008; 2:334-42.
• ⁶
  Moore S, Neylon C, Paul Eve M, Paul O'Donnell D, Pattinson D. "Excellence R Us": university research and the fetishisation of excellence. Palgrave Commun 2017; 3:16105.
• ⁷
  San Francisco Declaration on Research Assessment. https://sfdora.org/read/ (acessado em 03/Ago/2019).
  » https://sfdora.org/read/
• ⁸
  Hicks D, Wouters P, Waltman L, de Rijcke S, Rafols I. Bibliometrics: The Leiden Manifesto for research metrics. Nature News 2015; 520:429.
• ⁹
  Rabóczkay T. Significado real do número de citações de um artigo científico. Jornal da USP 2019; 26 jul. https://jornal.usp.br/artigos/significado-real-do-numero-de-citacoes-de-um-artigo-cientifico/
  » https://jornal.usp.br/artigos/significado-real-do-numero-de-citacoes-de-um-artigo-cientifico/
• ¹⁰
  SciELO Brasil. Seleção de Periódicos da Coleção SciELO Brasil. http://www.scielo.br/avaliacao/avaliacao_pt.htm (acessado em 09/Ago/2019).
  » http://www.scielo.br/avaliacao/avaliacao_pt.htm
• ¹¹
  Proctor R, Schiebinger LL, editors. Agnotology: the making and unmaking of ignorance. Stanford: Stanford University Press; 2008.

Publication Dates

• Publication in this collection
  07 Oct 2019
• Date of issue
  2019