The Qualis system : a perspective from a multidisciplinary journal

There is quite an amount of literature out there regarding the Qualis system and one wonders what someone can write about it that has not been discussed already. In this short essay, I will show the perspective of a multidisciplinary journal that has to live (or survive) in the very complex environment created by Qualis and the generated deleterious side effects for periodicals with broad scopes like that of the Annals of the Brazilian Academy of Sciences (AABC).The Qualis system was introduced in 1998 by CAPES (Coordenacao de Aperfeicoamento de Pessoal de Nivel Superior), an agency of the Brazilian government whose purpose is to evaluate and foster the Brazilian graduate programs (e.g., Andrade and Galembeck 2009). Right from the beginning, it was clear that one of the major aspects of a graduate program are publications by both, researchers and students. Since the amount of produced papers is high, a system had to be developed to discriminate this production (e.g., Barata 2016). This is how Qualis got into the picture. Essentially, this system classifies publications in strata that are further used to evaluate a particular graduate program. This system has been changed along the years, and presently there are eight levels: A1 and A2, the highest; B1 to B5; and C, the lowest that does not add anything for the evaluation process. So far so good, if it were not for a small detail: the percentage of A1 + A2 cannot be more than 25% of all periodicals evaluated. Moreover, the percentage of A1 + A2 + B1 cannot be over half the periodicals that are taken into consideration in a given year (e.g., Rocha-e-Silva 2009). I will come back to this later.As known, the AABC is the sole periodical produced in Brazil that can be regarded as truly multidisciplinary, publishing articles from physics and mathematics, to paleontology and scientometrics, passing through the description of new species (biodiversity), research of new chemical compounds, studies that could lead to the cure of diseases, and new methodologies applied to engineering, just to cite a few. No wonder our journal appears in several different areas evaluated by CAPES, that are now slightly less than 50 (Barata 2016). Since 2010, the AABC is listed in 28 (2011) up to 39 (2016) areas with quite some variation regarding the strata. For example, in the last Qualis list (2016), the classification of the AABC fluctuates from A2 (e.g., Anthropology/ Archaeology; Agrarian Sciences) to B5 (Biological Sciences I and II). Just a reminder: B5 is close to nothing in the present evaluation system.

There is quite an amount of literature out there regarding the Qualis system and one wonders what someone can write about it that has not been discussed already.In this short essay, I will show the perspective of a multidisciplinary journal that has to live (or survive) in the very complex environment created by Qualis and the generated deleterious side effects for periodicals with broad scopes like that of the Annals of the Brazilian Academy of Sciences (AABC).
The Qualis system was introduced in 1998 by CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), an agency of the Brazilian government whose purpose is to evaluate and foster the Brazilian graduate programs (e.g., Andrade and Galembeck 2009).Right from the beginning, it was clear that one of the major aspects of a graduate program are publications by both, researchers and students.Since the amount of produced papers is high, a system had to be developed to discriminate this production (e.g., Barata 2016).This is how Qualis got into the picture.Essentially, this system classifies publications in strata that are further used to evaluate a particular graduate program.This system has been changed along the years, and presently there are eight levels: A1 and A2, the highest; B1 to B5; and C, the lowest that does not add anything for the evaluation process.So far so good, if it were not for a small detail: the percentage of A1 + A2 cannot be more than 25% of all periodicals evaluated.Moreover, the percentage of A1 + A2 + B1 cannot be over half the periodicals that are taken into consideration in a given year (e.g., Rocha-e-Silva 2009).I will come back to this later.
As known, the AABC is the sole periodical produced in Brazil that can be regarded as truly multidisciplinary, publishing articles from physics and mathematics, to paleontology and scientometrics, passing through the description of new species (biodiversity), research of new chemical compounds, studies that could lead to the cure of diseases, and new methodologies applied to engineering, just to cite a few.No wonder our journal appears in several different areas evaluated by CAPES, that are now slightly less than 50 (Barata 2016).Since 2010, the AABC is listed in 28 (2011) up to 39 (2016) areas with quite some variation regarding the strata.For example, in the last Qualis list (2016), the classification of the AABC fluctuates from A2 (e.g., Anthropology/ Archaeology; Agrarian Sciences) to B5 (Biological Sciences I and II).Just a reminder: B5 is close to nothing in the present evaluation system.When the Qualis classification started to focus on the Impact Factor (IF) to rank journals in the respective strata, there was an immediate concern of Brazilian periodicals that had little chances to compete with others abroad.The fear was that the contributions with a higher impact potential produced by Brazilian scientists would fade away from the editorial surface of journals produced in the country.Aware of that, CAPES allowed each area to choose a small selected number of national periodicals considered relevant for the respective area to be classified in higher strata (Barata 2016).For some time, this was also followed by a substantial increase in funding, helping several periodicals to rise (e.g., Santos 2012).This inductive method was very welcome and attenuated the problems created by the Qualis system.But not for the AABC.
Along the years and with different editors-in-chief, the AABC has made great efforts to improve the journal.Since 2008, the main goal was to create the necessary conditions to increase the amount of peer-reviewed published manuscripts of good quality, that changed from 61 in 2008 to 200 (not including editorials) last year (2016; Fig. 1a).This number is still far too low when compared with the more than 3000 published by PNAS (Proceedings of the National Academy of Sciences of the United States of America) and we are presently aiming to reach 500 in the near future.Despite difficult economic times, we managed to introduce several improvements, such as the possibility to publish color pictures in the printed version for low cost (free for the online version), color cover, publishing papers ahead of print, newsletters, special volumes and, for all, a new online system.The latter was made available by SciELO in November of 2012, resulting in a dramatic increase of submissions (Fig. 1a).
The submission rate varies from field to field.By far, the lowest number of articles received by this journal comes from Physical Sciences.In the last eight years, the AABC has received from one to a maximum of five manuscripts per year (Fig. 1b).In 2011 and 2013 we did not publish a single paper of this basic scientific field.Screening through the Qualis system, the best classification of the AABC in Physics (actually Astronomy and Physics) was B3, but most of the times this journal was not listed, even when manuscripts were published.Honestly, I keep wondering when this important area of science is going to become extinct in our journal.
It is well known that the Qualis system affects submission rates (e.g., Cabral Filho 2010), but not many empiric data was made available by the journals.The AABC has also felt this effect, what can be exemplified in two areas: Biological and Agrarian Sciences.Most of the manuscripts we receive in Biological Sciences fall in what CAPES regards as Biodiversity, which unites areas like ecology, zoology and botany.This area was always the one with the highest submission rate.Not anymore.In 2013 the AABC was downgraded from A2 to B2.The effect on the submission rate was not immediate, particularly since we had a special volume in production.But in 2016, the number of submitted manuscripts went down from 355 (received the year before) to 274 (Fig. 1c).There is indication that another drop should be expected for this year.
On the other hand, Agrarian Sciences was always well classified in the Qualis system.It is in the higher strata (A2) and therefore submissions are increasing to a point that last year it surpassed Biological Sciences (Fig. 1d).Considering the submission rate history of the AABC, this is a highly unexpected result and there seems little doubt that the Qualis system is behind it.
Lets discuss the downgrade the AABC suffered in Biodiversity in 2013, from which it never recovered.What was the reason for that?The main criteria for classifying periodicals in a certain strata was the IF (despite the known problems of this number, see discussion in Casadevall et al. 2016).With the customary ups-and-downs of this value, the AABC would have fallen from the strata A2 to B1 -which is still quite attractive to authors in Brazil!But, since A1 + A2 + B1 cannot be over 50% of the total periodicals evaluated in that year, the evaluation committee of the Biodiversity area had to make some "tough choices".Therefore, in order to produce less "harm", periodicals where less researchers of the area have published papers, were artificially downgraded to comply with the 50% rule of CAPES.In other words, although the AABC had the numbers for a better classification, this periodical has published less papers in Biodiversity that year compared to others, and was "sacrificed" for the "overall good" of the area.Of course, the AABC, being a multidisciplinary journal, can never publish the same amount of papers of journals with a more specific focus!In another situation I have witnessed something similar happening when I was on the CAPES evaluation committee -needless to say the bad feeling that this has caused.
The 25% and 50% rules of CAPES have been seriously questioned for some time (e.g., Andrade andGalembeck 2009, Rocha-e-Silva 2009), but still apply.The attempts to mitigate the unwanted problems caused by the Qualis system have never reached the AABC.Understandable, each time an area has the possibility to select periodicals for any benefit, it will always choose the ones that publish more articles in that specific area.No way the AABC can compete.
In the review published by Barata (2016), some criticisms of the misuse of the Qualis system were presented.However, it must be regarded as naive not to have foreseen that this would happen in several circumstances, including promotions (e.g., Ferreira et al. 2013).For example, Barata (2016) criticizes editors for using the Qualis ranking to get more funding for their journals.However, the last grant announcement for funding possibilities for journals of both Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) (Chamada MCTIC/CNPq Nº 25/2016) and CAPES (Edital N° 13/2016) clearly state that, to be able to apply, the periodical must be at least classified at the B2 level of the Qualis list.Such contradictions show the complexity of the situation.
Although there is a general recognition of the difficulties imposed by the Qualis system to multidisciplinary journals, nothing was done so far.Due to its nature and size, the AABC does not offer competition to any journal published in Brazil, since we would never be able to publish all the good work that one specific area produces.Therefore, an affirmative action should be taken in the case of the AABC regarding the Qualis system.This is the only way by which the country, that suffers from Qualisitis (= obsession in using the Qualis system for several evaluations, inducing the decision for authors where to submit their manuscripts) might aspire to publish a stronger multidisciplinary periodical.If such an action is not done to foster the journal of the Brazilian Academy of Sciences, then for which other?

1340Figure 1 -
Figure 1 -Graphs showing the annual balance between 2008 and 2016 for papers received, rejected and published in the AABC.a -total values for all AABC sections; b -total