Qualis in four quarters: history and suggestions for the Administration, Accounting and Tourism area

1 INTRODUCTION

While the so-called "hard areas" always favored citation metrics to define their Qualis - specially the Journal Citation Reports' impact factor (currently published every year by Thomson-Reuters), several of the humanities and of the social applied areas (as, among others, psychology, administration, education and anthropology) began the systematic approach to the task via an evaluation form. It was only after the restructuration of the system (Capes, 2008), at the stage we call "the third quarter", that metrics got a more decisive role also in these latter areas. The first and second quarters of this paper tell the story of one of these - the Administration, Accounting and Tourism area. The story, however, is largely similar to those of all the others, especially when, also in the third qua rter, we discuss and present arguments that show the non-suitability of the use of the impact factor (or citation metrics in general) as a preferred criterion to build the Qualis hierarchy. The fourth quarter then presents a proposal that does not involve citation metrics and gives journals that are not indexed by leading private companies of the northern hemisphere a real chance to climb the upper strata of the classification.

Considering this framework, there were two main reasons behind elaborating this editorial. One was to review the area's almost 15 year history, from the 19 journals in the first formal evaluation in 2002 up to the impressive SPELL (Scientific Periodicals Electronic Library - www.spell.org.br). The other involves the perception of the importance of Qualis as an instigating instrument for the academic area. Sure enough, this history only exists thanks to the orientation and evaluation of scientific publications in the context of Capes (Coordination for the Improvement of Higher Education Personnel). This role is not always properly understood, and consequently, the aims may also not be achieved. Thus, this text was put together in order to (i) show the evolution of the process over time, (ii) analyze the progress achieved, and (iii) reflect upon elements for improvement.

The changes verified were philosophical, structural, and both qualitative and quantitative.

2 THE FIRST QUARTER: FORM-AIDED EVALUATION PROCESSES (2002-2007)

2.1 The First Form

It all began at the beginning of 2002: on the initiative of the area coordinator at the time, professor Sonia Calado, we discussed evaluating scientific journals during a Capes Committee session, "expressing dissatisfaction with the poorly structured model for evaluating journals" (F. Frezatti, August, 2016). After participating for various years on the Editing Advisory Group (GAE, discontinued in 2005) of the National Council for Scientific and Technological Development (CNPq) and also having spent some time on the Editorial Committee, I had come to focus on the quality of scientific journals and ideas regarding its four dimensions (technical-normative, product or content, productive process, and market) had been taking form. These ideas, which subsequently resulted in an article (Trzesniak, 2006), were laid out in this first meeting, in a more or less formal presentation (www.infocien.org/QualAval), which included discussing the Capes/Anpepp Evaluation Form (Yamamoto et al., 1999, 2002), which was, at the time, the "state of the art" reference for constructing the Qualis for areas that (i) viewed the participation of Brazilian journals as relevant in the respective flow of produced articles and (ii) were interested in strengthening and consolidating these journals.

The members of the Area Committee showed motivation and enthusiasm for the proposal of a form, particularly because they identified not only a way of systematically and objectively organizing Qualis using it, but also because of the potential to significantly contribute to improving journals. It was decided that a second meeting would be held with the aim of creating an initial evaluation form taking into account (i) the thematic characteristics of the area, (ii) the average quality of journals at the time, and (iii) the elements that were considered most relevant for evaluating a set of journals between a first and second evaluation (a set of signposts or guide for orientation in the future). The teamwork was intense and dedicated, with the participation of Alberto L. Albertin, Antonio Freitas, César Tibúrcio, Fábio Frezatti, Jaime Fensterseifer, Neusa M. B. F. Santos, Piotr Trzesniak, Rachel C. M. Silva, Reynaldo C. Marcondes, Sergio Bulgacov, Silvia H. Koller, Sônia M. R. Calado, and Tomás A. Guimarães. The relevance of each requirement was discussed and for those that were accepted, the levels for meeting them were determined.

Despite the journals already being classified into strata A, B, and C at the time, none of these were subdivided into levels (A1, A2, B1 etc.), as is done in 2016. However, there was a second, independent criterion, known as ambit, which could be International, National, or Local. Thus, a journal could be International A, B, or C, National A, B, or C, or Local A, B, or C.

Under these circumstances, the 2002 Evaluation Form (www.infocien.org/QualAval/Hist/Capes/AdmConT) was concluded, by which a journal was scored in five dimensions, as shown in the second and third columns of Table 1. Out of the total of 34 evaluation items that the form presented, 27 involved only quality, four exclusively involved context, and three (items 20, 31, and 31) involved both characteristics. Thus, a journal could achieve up to 73 points on quality and up to 34 points on ambit.

Table 1
The journal evaluation dimensions of the Administration, Accounting Sciences, and Tourism area (five from the form and one ad hoc) and the evolution of the respective maximum scores over time

Dimension	2002			2005		2007
	Quality	Ambit	Quality	Ambit	Quality	Ambit
Standardization	16.0	0.0	16.0	0.0	15,0	0,0
Publication	12.0	0.0	12.0	0.0	12,0	0,0
Circulation and visibility	14.0	11.0	14.0	11.0	14,0	9,0
Authorship and content	15.0	16.0	15.0	16.0	13,0	12,0
Editorial management	16.0	7.0	16.0	7.0	15,0	7,0
Impact (ad hoc)	0.0	0.0	10.0	0.0	15,0	0,0
Total	73.0	34.0	83.0	34.0	84,0	28,0
Form total*	95.0			105.0		100.0

• * Lower than the sum of the quality and ambit totals due to there being three items (numbers 30, 31, and 32) computed for both criteria.
• Source: Elaborated by the author.

2.2 The Process Principles

In the message sent with the form to the editors, besides the explicit incentive for all to adhere and get involved, some fundamental principles were set out for the journals to grow and increasingly consolidate:

• transparency: clear criteria, objectively described, with "help text" offered in the form itself, accessible via pressing the F1 key;

• participation: each editor carried out the initial evaluation of her/his journal and sent the result to the Commission for verification and standardization;

• editor self-improvement: the combination of (i) well elaborated items supported by explanatory texts, (ii) exercising self-evaluation of each one's own journal, and (iii) the inclusion of items containing signposting for future advancement created a context of learning for novice editors and new journals in order for them to mature more quickly.

It was in this context that, in mid-2002 and with professor Antonio Freitas, director of Ibmec/Rio de Janeiro, acting as host, the Commission met at this institution for the first Qualis evaluation in accordance with the new system. 19 journals took part, with the result which, compared to that from 2007, is presented in Table 2. More important than the classification itself, however, was being able to confirm practically the viability and effectiveness of the process. At no time was it questioned whether this would in fact be the best path to take, adhesion was complete and immediate, and yes, it would go ahead.

And that was what happened.

2.3 The 2002-2004 Evaluation

The lessons from the first round led to some improvements to the form, with the introduction, in the 2002-2004 version (www.infocien.org/QualAval/Hist/Capes/AdmConT), of the impact element, worth up to 10 points. It was understood that the initial version mainly considered technical-normative aspects, and for various elements, scores were attributed on a merely declaratory basis. What was lacking, to some extent, was the valuation of content. Impact was a concept that was related to this, but how could it be measured? It was then decided, at first, to trust in the judging ability of the Commission itself: it would be up to it, provisionally, to define the score for each journal with regards to this element, however without giving up the search for alternatives that would allow it to be established in a more transparent and objective way.

It is funny to record that, in this version, the inclusion in the circulation and visibility dimensions of two "signposting elements", which both bore an expectation of continuity of printed journals: (i) effective distribution of printed editions, with the aim of avoiding shelves overloaded with "deadlocked" editions and (ii) distribution to members of scientific societies. If the merit of the first was merely economic, that of the second contained the hope of promoting possible future sharing of editorial responsibility between these societies and universities or research institutes (or their subunits), which constituted (and still do) the institutional rearguard of most of the journals in the area. Normally, institutions are superior concerning physical-financial resources, while societies, without doubt, have the advantage of a much larger body of specialist researchers (Döbereiner, 2001). The hypothesis behind this was: if the members of a society came to receive a journal, they would feel more committed to the latter, each one would see it as "her/his journal", and they would be more motivated to cooperate as reviewers, members of the editorial board, or even associate editors. Strictly speaking, there are no conclusive experiments that refute this hypothesis, even in the current context of solely electronic journals. There are also no confirmations, but, apparently, there is no way that a serious, transparent, and well negotiated partnership between an institution (or various) and a scientific society (or various) cannot result in significant advantages for all parties. Editors and leaders of scientific societies should pay more attention to this possibility.

In operational terms, supplementary tables were added to the form, facilitating the compilation of data and the calculations necessary for determining the scores regarding the international, national, and inter-institutional authorship items. The 2002-2004 form was used to evaluate 39 journals in 2005.

Table 2
Absolute numbers of journals in each classification category from the first (2002) to the last (2007) of the three carried out evaluations and the respective percentage variations

		National				Local				Total
	2002 (abs)	2007 (abs)	Variation (perc)	2002 (abs)	2007 (abs)		Variation (perc)			2002 (abs)	2007 (abs)		Variation (perc)
A	1	6	+500 %	0	0		+0 %			1	6		+500 %
B	1	13	+1200 %	4	3		-25 %			5	16		+220 %
C	5	8	+60 %	8	12		+50 %			13	20		+54 %
Total	7	27	+286 %	12	15		+25 %			19	42		+121 %
	37 %	64 %	-	63 %	36 %		-			100 %	100 %		-

• Source: Elaborated by the author.

2.4 The Third Evaluation and the Benefits Accomplished

The use of a form to classify journals from the area last occurred in 2007, five years after the first evaluation. The number of electronic journals had become significant, signalling that the elements specific to printed versions would quickly cease to make sense. This new context warranted highlighting in the texts sent to editors together with the 2007 Evaluation Form (www.infocien.org/QualAval/Hist/Capes/AdmConT):

Editors of exclusively electronic journals should also complete the form, but they will also undergo an examination of their special characteristics, for example: requirements for and ease of access, viewing, information available, search mechanisms, availability of complete texts, access statistics, and especially, preservation mechanisms. This additional examination will be carried out by the Committee (p. 1).

Moreover, the scores for some of the elements which previous evaluations revealed to be too highly valued were reduced, as well as those that were difficult to verify and those with a poor ability to differentiate (in a classifying procedure, maintaining an element met by all participants does not make sense). The score distribution is given in the last two columns of Table 1.

Table 2 compares the evaluations from 2002 and 2007. With regards to context, a significant inversion in the percentage of national journals is observed, growing from 37% to 64% in relation to local ones, which decreased from 63% to 36%. However, much more dramatic are the numbers corresponding to quality levels. It can be observed that, while the quantity of evaluated journals increased by 121%, from 19 to 42,

• that of national A journals increased by 500%;

• that of national B journals increased by 1200%;

• that of A journals increased by 500%;

• that of B journals increased by 220%,

that is, in all of the highest quality segments, the growth observed was greater than that for the number of journals. This leads us to conclude that the form, as a tool for executing the policies that the area considered important for Qualis (and, therefore, for the flow of its scientific production), in conjunction with a transparent and cooperative partnership between the Capes Committee and the editors, led to an expressive evolution for Administration, Accounting Sciences, and Tourism journals as a group in the period covered up to here.

3 THE SECOND QUARTER: GUIDELINES FOR THE 2007-2009 EVALUATION

3.1 A Renewed Evaluation

Having concluded the 2007 evaluation, the Commission understood that it had reached the end of the cycle regarding the 2007 form and versions of it, and went on to discuss more thorough modifications to the process. It was clear that all involved - editors and commission - had done their "homework" well, and that the time had come to work towards new goals.

On one hand, all the journals had incorporated the technical-normative aspects well and definitively, with there being no contributions to the evaluation that they would continue to be scored. On the other hand, it was more and more important for the journals to increase their visibility, which meant valuing indexations, which appeared on the form in a generic way in one single item. The question of content preservation of the exclusively electronic journals was another concern (we will come back to this). And it could not be ignored the possibility of electronic management of the editorial process via the Electronic System for Editing Journals (SEER), the Portuguese language version of the Canadian Open Journal Systems (OJS), translated by and freely available from the Brazilian Institute for Information on Science and Technology (Ibict).

These aspects and others, after discussions with the editors and the approval of the Area Commission, were published on December 31, 2007, by the area coordinator, professor Tomás de Aquino Guimarães, in the document Evaluation Guidelines for Journals for 2007-2009 (www.infocien.org/QualAval/Hist/Capes/AdmConT), or 2007-2009 Guidelines, for short, for which we highlight the main conceptual aspects:

1. (...)

2. The focus of evaluation becomes electronic journals, and subsidiarily, printed journals, unlike what occurred until 2004-2006. This means that the Area Commission will value the cost-free, online availability (free or open access) of the whole text of articles published in journals, whether they are printed or electronic.

3. Valuing indexation criterion. Indexers are "journals of journals" (currently, they are electronic databases) that publish summaries, abstracts, or whole texts. They can be general (such as SciELO, RedAlyc, and Current Contents) or thematic and area-specific. The Qualis Committee verified that many editors consider journal listing directories (for example, ISSN, Latinex, and Ulrich's) to be indexers. This is an important aspect to be reviewed by editors, who should invest in actual indexers for the benefit and greater visibility of scientific production in the area.

4. Adoption of automated management systems in the editorial process, for example, the SEER system, distributed cost-free by the Brazilian Institute for Information on Science and Technology, and recognized by Capes.

5. Qualifications, specialization area, and diversity of editors, scientific or editorial committees, authors, evaluators, and other players involved in the journal production process. The diversity criterion includes penalizing, more heavily than occurred up to 2006, journals that publish articles from authors linked to the higher education institutes or programs responsible for editing them.

6. Carrying out a survey of lecturers-researchers from the area to evaluate the perceived quality of journals. This survey will aim to identify how much a journal is recognized and therefore perceptions of its impact in the scientific community, according to the point of view of a representative sample of people responsible for intellectual production in the area.

7. Preservation information and the presence of metadata. Preserving information aims to guarantee the integrity of what has been published over long periods of time and metadata consists of important mechanisms in the system for recovering and viewing scientific information (pp.1-2).

Considering that these guidelines were defined around nine years ago, the advanced nature of their vision is notable. For example, what is set out in item 7 is not yet adequately fulfilled by many journals (this claim is with regards to journals in general, not to the journals from the area).

3.2 Preservation

With regards to preservation, this comes from demands for the perpetuity of referential information: since new research is built on previous research, descriptions of the latter need to be available indefinitely. No material published in journals that lie on a single server and are subject to destruction by fire, floods, earthquakes, tsunamis, plane crashes, hackers etc., although produced with the upmost rigor and dedication, presents referential quality: those journals receive grade zero for the perpetuity requirement, which together with topicality, accessibility, and reliability, form "the four 'ities' that ensure the fifth", full quality of a scientific reference (Trzesniak, 2014).

It is noted that replicating a server (or resorting to offline backup) only eliminates the grade zero for preservation if the server (or the permanent storage media used) is located (or maintained) some hundreds of kilometers from the main server.

The good news is that those included in an indexer that appropriates content (such as SciELO, RedALyC, and SPELL) have their material replicated safely, since these portals do not neglect digital preservation. Another good practice, for those that use the SEER/OJS platform, is to form part of LOCKSS (https://www.lockss.org/), the aim of which is precisely to preserve content; however this involves a resource that needs to be activated on installation. We recommend that all editors who use SEER do this.

This whole discussion only took files themselves into account. And as for the question of access to these files? Will doc, docx, and pdf formats still exist a hundred or two hundred years from now? Will there be hardware that reads hard drives (HDs), compact discs (CDs), or digital video disks (DVDs)?

And humanity? Will it take responsibility for the immense effort required to convert all existing information onto new formats, together with storing all that is created? If today we convert the entire collection of human knowledge onto pdf files, and maintain this standard for 50 years, however in 2066 the decision is made to change to "xkf", all of the current collection will need to be converted, plus what is created in these next 50 years! And 50 years later, all there is in this new conversion, plus 50 more years of information... this appears to be a worrying snowball...

Well, the 2007 document contemplated this element, requiring a position with regards to accessibility in the final part of the following item:

c. Explicit description of the mechanisms that you propose and adhere to with the aim of preserving scientific content over time, contemplating both the physical integrity of information (with respect to catastrophes and accidents) and guaranteeing that it remains accessible in the face of technological evolution in terms of software (file types and their coding) and hardware (media, devices). (p. 3)

3.3 Metadata

The second part of item 7 - metadata - is another point that is often neglected. The basic proposal behind this concept is to avoid information being replicated indiscriminately and overloading the planet's electronic storage systems. Metadata functions based on the existence of data providers - which maintain information - and service providers - which discover and offer it in accordance with criteria that attach value to it so that it can be recovered. For example, someone may find it interesting to create a service involving... well, say the topic of metadata itself... then they organize a website concerning everything related to it - however, how can it be exhaustive and not let anything or very little escape? Spending all day on a search engine, trying words, always getting 7,854,567 answers - and having to filter these results manually? Unfeasible, of course. Here, metadata comes in in order to solve three problems: those regarding precision and exhausting the information recovered, as well as that involving the need for human intervention.

It works (or would work) like this: the data provider stores the main information and, at the same time, displays the key set of data concerning the information stored - the metadata - which describes it essentially and completely. Service providers create harvesters that scan the web and each time they find some metadata corresponding to what they are interested in, they capture only the links to the main information, organize them, and offer them in accordance with the added values they have chosen.

Generalized use of a set of standardized metadata - such as the 15 that form the nucleus of dublin core - displayed via an also standardized internet protocol (such as OAI-PMH, open archives initiative - protocol for metadata harvesting) standardizes the architecture for organizing information on the web, and the harvester spots all of this as if it were a database located in a single machine, since it finds the same panorama in them all.

Objectively, offering good metadata is more important than having the impact factor. The latter is a post mortem calculation: when it is carried out, nothing more can be done to promote any improvement in its value. Metadata influences the discovery and dissemination of information and only what is discovered can have an impact.

Just as with regards to preservation via LOCKSS, SEER/OJS may or may not be configured to require metadata at the time of submission, and unfortunately, the default is for it not to. However, metadata completed by authors are often flawed, incomplete, imperfect, and no database can be better than the content that feeds it. Thus, the advanced proposal for the area, in December 2007 (Comissão de Administração, Ciências Contábeis e Turismo, 2007), valued the presence of someone specifically responsible for metadata quality:

II.3.11. Special services. The following special services will be valued and scored: 1. The presence of metadata in all of the official languages and in English. 2. The existence of a specialized editor for preparing/revising metadata (p. 6)

3.4 The Survey

Another interesting proposal from the new evaluation was to introduce the survey, item 6 of the passage highlighted at the beginning of this section. The inspiration came from France, where it was introduced, under the coordination of Phillipe Jeannin (2002), because "the indicators related to France constructed using these databases [of the Institute for Scientific Information (ISI), the organization at the time responsible for the Journal Citation Reports (JCR)], do not reflect the real research activity [in Social Sciences] in this country" (p.4). Researchers from the areas were asked, with regards to each journal included in a previously elaborated list, to answer the questions "Is this an A journal?" and "Is this a B journal". The answers accepted were "yes", "no", and "I do not wish to answer". There was a brief introduction, prepared by the area itself, with a subjective statement of what should be understood as an "A journal" and a "B journal".

Both the survey and the impact factor belong to the market quality dimension (Trzesniak, 2006), however unlike the latter, the former aims to capture, explain, and formalize the subjective perceptions, latent in the community, that researchers have of the journals. These perceptions can be acquired by the respondents via the most varied means: hearing it said, colleague influence, journal marketing efforts, credibility of the publishing institution, scientific notoriety of the editor or of the members of the scientific body, reading a memorable article, and even occasionally (but only occasionally) via critical examination of some complete edition, that is, by means of an objective product evaluation.

Besides its intrinsic merit as an evaluating tool, the survey is comprehensive in terms of the realm of journals evaluated and is not limited to the set of journals that are part of a limited database and need to be monitored for many years, as is the case for calculating impact. And, also for this reason, it can be much more simply operationalized.

Concerning the introduction of the survey, Trzesniak (2006), our basic reference for all of this section, states:

The Capes Committees for Psychology and for Administration, Accounting, and Tourism have been concerned with finding alternatives for market evaluation (impact), with the use of a survey being discussed, inspired by the French experience, but adapted to the Brazilian characteristics and executed electronically. (p. 354)

3.5 Others

As other interesting aspects, the 2007-2009 Guidelines contained minimum criteria for a journal to deserve classification in the international context and specifically valued: (i) active electronic management (added emphasis) of the editorial process, (ii) adoption of the OAI-PMH standard, and (iii) the publication and permanent maintenance on the website of an Annual Management Report, with a nominata of ad hoc evaluators, average time interval between submission and publication, number of articles submitted, rejected, removed, accepted, and in process, and with regards to the printed journal, information on circulation (libraries, subscriptions, exchanges, courtesies etc.).

3.6 An Unfortunate Epilogue

The area Qualis was ready and organized to face another three years when, in a meeting that took place on April 16^th and 17^th 2008, the Capes Scientific Technical Council for Higher Education (CTC) (Capes, 2008) defined a general conceptual restructuring of the system for classifying journals, discontinuing the ambit criteria and dividing the A quality stratus into two sub-strata (A1, the best, and A2) and the B stratus into five (from B1 to B5). Up until here, nothing would be unfortunate, and the criteria could continue to be applied. However, in item 7, the Capes document signals:

7. Each area should lay out in its Qualis Journal Criteria document the criteria by which each stratus is defined (impact factor, h-index, or other ways of measuring the quality). Above all, the two highest strata need to be defined in a particularly thorough and rigorous way. (p. 2)

The use of citation metrics as the main classification criterion was always the preferred practice of the areas with representative production indexed in Thompson-Reuters which, especially in 2007, exerted a kind of monopoly of the impact factor in its Journal Citation Reports. The explicit mention of this factor and the semi-disqualification of any alternative criteria as "other ways of measuring quality", spread a culture of subservience to a private company, created and principally dedicated to science in the northern hemisphere, a culture which, especially for the areas of human and applied social sciences, is very difficult to understand and accept. More arguments supporting this statement are presented in the second subsection of section 5.2, the title of which begins with Is it always reasonable to hierarchize.

The reformulation and the stress given to the use of metrics created a redirection for the dimensions and items to be considered in evaluations, and the 2007-2009 Guidelines ended up on the margins of the Qualis evaluation. But, then, why dedicate so much space to them in this text? There are at least two important reasons: one of them is that, even though not used in the evaluations, they draw attention to aspects that, even almost a decade later, can still orientate editors, especially novices, to better perform their role. Another is that they arose from broad discussions in meetings that involved editors and evaluators from the area, which means they are endorsed and validated by all of the players involved.

And there may also be a third one. It is quite understandable that aspects such as preservation and metadata were not the concern of the CTC: although vital - the first for perpetuity and the second for dissemination - they do not form part of the day-to-day of a researcher who is not from the area of information science. Thus, it does not seem inappropriate to address them here.

4 THE THIRD QUARTER: QUALIS TODAY

The existing criteria for the construction of the Administration, Accounting Sciences, and Tourism Qualis are described in a specific document (Capes, 2015).

For any journal to be classified, it needs to have an ISSN and publish at least two editions per year. Moreover, (i) the overdue index (i_over ) is calculated, given by the ratio of number of overdue editions to editions promised per year; (ii) the impact factors are collected, namely I _JCR, from the Thomson-Reuters Journal Citation Reports and I _Sci, from the SciELO database; and (iii) the H _Sco indicator is collected from the Scopus database.

Thus the following are applied to define inclusion in the tiers:

• B5: i _over ≤ 1;

• B4: i _over ≤ 0.5 and two or more years in existence;

• B3: i _over ≤ 0.5, three or more years in existence and confirmed as present in at least two of the indexers Ebsco, Doaj, Gale, Clase, Hapi, ICAP, and IBSS;

• B2: either I _Sci ≤ 0.01, or it is indexed in RedALyC or edited by Sage, Elsevier, Emerald, Springer, Interscience, Pergamon, Wiley, Routledge, or Taylor & Francis;

• B1: either I _Sci ≤ 0.01, or 9 ≥ H _Sco > 0, or 0.7 ≥ I _JCR > 0;

• A2: either 24 ≥ H _Sco > 9, or 1.4 ≥ I _JCR > 0.7;

• A1: either H _Sco > 24, or I _JCR > 1.4.

Regarding the result of this initial inclusion, the Report informs of two adjustments being made:

• Journals that have not been listed as being from the area, according to the impact factor calculation bases, have been reallocated to the starus immediately below.

• Some journals edited in Brazil, considered the most relevant for the area, and responsible for the publication of 10% of the total number of articles from the area in 2013 and 2014, have been reallocated to the strtus immediately above.

In a general analysis, the study of the report shows the participatory spirit, dedication, care, and intense effort, which since the form era continue to guide Qualis preparation in the area. A highly positive point is the second adjustment, involving raising the journals that publish the most articles from the area to the immediately above stratus. This is an interesting practice and the seed for one of the proposals for improvement discussed in the next section.

On the other hand, one point stressed which is of great concern is that the only path towards appearing in strata A1 and A2 is for the journal to be indexed in Scopus or in Thomson-Reuters. Must this be the case? Does it make sense in 2016, when there are so many other serious and relevant indexers that are much friendlier to research from the southern hemisphere and to applied human and social sciences? There are strong arguments, also discussed in the next section, that lead to doubts regarding the convenience of adopting this type of criterion for the highest strata, especially in these just mentioned areas. In fact, the arguments suggest that this practice is a form of self-punishment.

5 THE FOURTH QUARTER: SUGGESTIONS TO DISCUSS TO IMPROVE QUALIS

5.1 What is Really the Key Question?

In an objective analysis, Qualis only serves to score production from post-graduation programs. This obviously leads researchers to aim to include their articles in journals belonging to the highest possible stratus. However, the desired purpose, explained in item 9 of Restructuring Qualis (Capes, 2008), is related to the quality of scientific production:

9. The highest two strata - A1 and A2 - could be left empty, by decision of the area, in terms of articles effectively published, with it being recommended that only the highest quality journals be included in them, in which the area judges that it should make some indication in order to advance the quality of scientific production in its community. Qualis will thus play a clearly defined inductor role, not being limited to analyzing where the area publishes, but also indicating where it should publish (emphasis added). (p. 2)

In this same item, immediately before the passage highlighted, journal quality is chosen as a proxy for quality of production. This involves an aspect that should always be taken into consideration by those who prepare a Qualis classification: the real focus of the question is quality of production, not that of the journal.

But when do quality of content and that of a journal overlap? It is in the peer review, in the guiding and execution of the editorial process. All articles that undergo a serious and well managed editorial process are of quality. The flaws that occasionally arise and draw attention result from errors that are human, wrongful (involuntary, malpractice or negligence), or fraudulent, in the case of predatory journals.

The necessary and sufficient condition for a quality editorial process is an involved, dedicated, and active editor who knows how to choose associate editors and members of the scientific board with this same profile. Peer review should not be seen (as is unfortunately found) as a just a stage to be completed during the scientific publishing ritual, but as "a cordial complicity of authors, referees, and editors, with everyone making the maximum effort and working hard towards content excellence".

5.2. Indexation, Without a Doubt, But... Impact Factor?

5.2.1 What is the relationship between journal impact factor and quality, visibility, and article citations?

Unfortunately, it is not possible to construct a Qualis by auditing the journal editorial process or evaluating editor involvement. The areas have sought alternatives, placing emphasis on indexations, and among these, defining a hierarchy based on some citation metric, such as impact factor, H-, or Q-, or J*-index (among others).

There are no doubts with regards to indexations in selective databases, that is, those that carry out evaluations for entry and permanence of journals in the collection. Such evaluations require technical-normative requirements to be met, peer review, and non-endogamy of authors, editors, and scientific board. This is one point in which databases that opt to be exhaustive fall short. However, both display their journals and give visibility to their articles, thus providing the condition necessary for an article to be cited: to be found. Indexation is, thus, a type of citation catalyst - however the "active agent" is the quality of research. It is worth mentioning here that quality metadata is another important catalyst, since it likewise significantly contributes to an article being found.

In contrast, employing the impact factor or any metric indicator for journals as a decisive element for defining the quality of an article is a highly questionable practice. Otherwise, we will see:

• The impact factor considers, in the numerator, the quantity of citations attributed to a journal, in the year of reference, in relation to the number of articles published in it in the two (or three, or five, or ten) immediately preceding years. It can reflect the success of only some articles or of only one in particular, but it allows absolutely nothing to be concluded regarding the quality of a particular article; production, therefore, ends up being valued more by way of "free-riding" than by means of intrinsic merit.

• The following argument reinforces the previous point: the impact factor transfers the occasional quality of some articles to the journal, which is already a questionable feature. By using it to extend this journal quality to all articles linked to it, a second transfer is carried out, surely much less defendable than the previous one. Difficult to accept indeed.

• Despite being robust, the impact factor generally changes every year, which introduces a luck/random aspect into evaluations: yesterday my article was great, but today it is only good because the journal's impact factor has decreased. But is it not the same article? Has its quality decreased overnight?

• Finally, it is debatable whether a high impact factor promotes visibility and increases the number of citations attributed to the articles included in a journal. This may have eventually occurred in the Gutenburg era, when the researcher went to the library and consulted physical editions: it may be possible that she/he would start with those journals with the highest impact, but it is more likely that she/he would first examine those that were regularly publishing the largest number of articles in her/his area of interest. With the current (2016) ease of electronic searches, it is difficult to imagine someone beginning their bibliographic search limited to just one high impact factor journal or filtering the results created by the database, requiring only those published in journals with an impact factor higher than y to be displayed. It must also not be common - if it does actually happen - for someone to cite an article only because it was published by a journal with a high impact factor, or fail to cite another otherwise.

These arguments allow for a conclusion to be reached in favor of indexation, yes. However, establishing hierarchy by journal impact factor or h-index, or any other metric that does not directly reflect the quality of an article or promote future citations of it, is difficult to defend and support. Answering the question featured in the title of this section, "the relationship between a journal's impact factor and quality, visibility, and citations of articles published in it, is inexistent or, at most, occasional."

If the already presented objections, all technical in nature, were not enough, one still has that the impact factor is subject to be inflated by ethically questionable procedures, with notorious cases of coercive citation (Coercive citation, 2016), self-citation (Van Norden, 2012), and formation of editorial cartels (Davis, 2012; Van Norden, 2013).

5.2.2 Is it always reasonable to hierarchize Qualis favoring metrics established by private companies from the northern hemisphere?

Letícia Strehl, specialist and habitual speaker with regards to production metrics, defines as "preconditions for calculating impact indicators (i) the existence of databases that index a representative quantity of documents and (ii) the references cited in the publications" and continues: "currently, the main databases for fulfilling this goal are Web of Science and Scopus" (Strehl, 2013, p. 10). Below this (p.15), she makes the first limitation clear: "the number of Brazilian journals in the databases is paltry". These numbers, in 2015, were 133 in Web of Science (129 in 2009) and 424 in Scopus (266 in 2009) (Strehl, 2015, p. 12), corresponding to, approximately and respectively, 1% and 2% of the total number of journals monitored in each one of the databases. Despite the growth in the second database, it is impossible to disagree with Strehl.

These numbers make it clear that these databases were not made to receive journals, articles, and researchers from the southern hemisphere. In order to compete in their territory, you need to adapt to their language and the problems that are of interest to them. This is actually possible for some areas, but certainly cannot be extended to all. Added to this context is a problem mentioned a number of times in meetings with editors: Brazilians tend not to cite Brazilians, preferring foreign authors. Thus, Brazilian journals, besides competing unequally with practically 100% of those held on databases from the northern hemisphere, are also not mentioned by authors from the country themselves - and they will certainly be barely mentioned or fail to be mentioned by foreigners. How can they, in this context, achieve a significant impact factor or at least one that is not nil? In short, requiring a minimum level for some Scopus or Thomson-Reuters metric as an exclusive criterion for reaching the A1 and A2 strata practically eliminates Brazilian journals from them and constitutes the subservience of Brazilian science to services created for the northern hemisphere and maintained by private companies from there, a condition that is very difficult to accept.

And there are even more objections. Updating the work of Trzesniak (2012) on CNPq data up to 2014, it can be verified that, of all the articles classified as international which Brazil produces, only 6.8% come from Humanities (Linguistics, Literature, and Arts; Applied Social Sciences; Human Sciences), while the other 93.2% originate from "hard" sciences (Engineering and Computer Sciences; Exact and Earth Sciences; Agricultural Sciences; Health Sciences; Biological Sciences). Despite the small evolution, this panorama does not differ significantly from that of 2010, 5.1% and 94.9%, respectively. Transforming these vertical percentages into horizontal ones, considering that production includes books, chapters in books, national articles, and international articles, it can be observed that the latter correspond to 57.6% of the total for "hard" sciences, but only 9.7% for Humanities (in 2010, 53.5% and 7.3%, respectively)!

It is concluded that the adoption of foreign metrics as the only criterion capable of accrediting a journal to strata A1 and A2 may even be acceptable for "hard" sciences, in which international articles correspond to more than half of all that is produced. However, in humanities, it corresponds to self-punishment, "since it banishes more than 90% of all that the area produces from these strata!"

If, in the previous section, arguments were presented against the use of any citation metric created for journals to hierarchize Qualis, directed towards evaluating articles, in this one the aim is to highlight the perversity of, for this purpose, prioritizing metrics developed for the context of the northern hemisphere by private companies based in that part of the planet.

What alternatives are there?

5.3 A Simpler, Fairer, and More Objective Qualis

By reviewing the Qualis criteria in the documents from the areas of Capes, many interesting and creative solutions can be found for the problem of hierarchizing, besides the direct use of metrics: standardization using the median, equivalence between impact factor and H-index, J* and Q indicators... Let there be no doubt regarding the recognition of the merit of these efforts. In particular, it is worth mentioning the interdisciplinary area, which needs to find a solution that considers journals from practically all areas of knowledge in a uniform, fair, and balanced way. The respective area document (Capes, 2012) deserves a good read. Moreover, there is an intense discussion regarding aspects to improve and perfect in the Qualis system (Pascutti, 2016).

However, this whole context of elaborating and refining the construction of the Qualis hierarchy based on citation metrics creates the impression that more attention and effort is being given to a particular solution - the use of metrics - than to the problem itself - broadening the reach of and valuing the research developed by the programs. And, as was already discussed, metrics leave a lot to be desired both as criteria for production value and as media for providing visibility and citations.

And so the question that is left is: since citations is a variable that considers market quality (Trzesniak, 2006), if we want to substitute it for another from the same dimension, which one could it be?

The answer comes from a combination of the survey proposal from Jeannin (2002) and a practice from the Gutenberg era. Jeannin aims to capture the subjective value of journals attributed by researchers. The Gutenberg era practice teaches that the first journals that researchers consult are those that have the greatest thematic affinity with their investigations and those in which, precisely because of this, they publish most and are going to publish most. The quantity of articles that the area publishes in a particular journal thus becomes an interesting proxy for market quality, since it indicates researchers' preference for that journal. And so: why not use this publication indicator in composing a general criterion for elaborating the Qualis hierarchy?

Another key merit in using volume of articles published instead of foreign metrics lies in at least opening up the possibility of access to strata A1 and A2 for journals that are not in the Scopus and Thomson-Reuters databases, valuing these journals' own efforts, and not obliging them to make achievements that do not depend solely on themselves and need to be made in frankly hostile territory. Then, a Brazilian journal could rise in the general criterion via author preference, surpassing foreign ones that publish a tiny part of what the area in fact produces and only maintain their privileged position because they are included in Scopus and Thomson-Reuters.

The adoption of a procedure along the here proposed path will include the areas of humanities and applied social sciences in a specially fairer way. Mugnaini, Digiampetri & Mena-Chalco (2014) analyzed the distribution of the production of articles from all areas, published in journals indexed in SciELO and in Thomson-Reuters, covering the period of five triennial Capes assessments (1998 to 2012). The journals were organized in descending order of the number of published articles and distributed into three "Bradford zones," which included, respectively, those responsible for the first, the second and the third third of all published articles. The conclusions speak for themselves:

Finally, analysis of the percentage change of the scientific output of the areas in international journals reveals three differentiated groups of areas. In the area of social sciences and humanities, zones 1 and 2 are composed exclusively by national journals, while zone 3 is showing decreasing percentage due to the effort of publishing in foreign journals. In Physics, Space Science and Immunology, production takes place in international journals, regardless of zone. The remaining areas (exact and earth sciences, health sciences and biological sciences) have shown increase in the percentage of publication in national journals in zones 1 and 2, zone 3 remaining exclusively international (p. 251).

Thus, a simpler, more inclusive and fairer Qualis with regards to Brazilian journals, particularly for areas that mainly publish in these, could:

1) Disregard the impact factor (and other metrics) due to it being too indirect and, as already stated, portraying past performance, calculated post mortem, when nothing more can be done to improve it, and in which, to paraphrase investment fund advertising, "occasional high value does not guarantee future results" (that is, citations).

2) Score presence in indexers, because the greater the number of journal indexations, the greater the visibility of the articles published in it, increasing the likelihood that they will be found and therefore cited.

It will be up to each area to determine the number of points to be attributed to each indexation, with the sum of all being a journal's visibility score. The distinct difference in production profile between humanities and "hard" sciences makes it clear that it is not appropriate to simply transfer criteria from one of these hyper-areas to the other.

Aspects to consider when defining indexer scores are:

• those that grant free access to the indexed content provide a better service than those that charge for this;

• selective ones (that evaluate journals for inclusion) should be more highly valued that exhaustive ones;

• those that store copies of indexed content contribute decisively to preserving knowledge, thus being much better than those that merely indicate content that exists on other websites;

• a specific indexer from the area should not be allocated to a lower level than the "big" ones only because it is smaller;

• particularly for areas that mainly produce national articles, prioritizing indexers that only consider Brazilian journals peripherally does not make sense;

• certain target public characteristics or the thematic scope of indexers could be considered and a second indexer that overlaps another, already scored one could be scored lower. For example, SciELO only or RedALyC only, 12 points, but SciELO and RedALyC, 16 points;

• for journals without any indexation, a visibility score equal to 1 could be assigned.

3) Tying the visibility score for each journal to the quantity of articles that the area has published in it, for example via multiplication, creating the respective Qualis score, ordering from the largest to the smallest and establishing the cut-off points for the different strata, respecting the general guidelines for the respective population.

Taking into account the total number of articles that a journal publishes awards editorial effort, recognizes a journal's contribution to divulging the knowledge produced by the area, and translates researcher perceptions with regards to the preference for a vehicle in which their work has better chances of being found and cited. How does the policy of constructing Qualis based on the impact factor, leading authors to prefer journals that have a high one, contribute to each one of these points?

Multiplication is a simple procedure and makes sense. It is reasonable to assume that publishing an article in a journal with two or three indexers, respectively doubles or triples the chances of it being found, compared to publishing it in a journal with only one indexation. However, the simple product may not be appropriate in all cases. For example, in areas that cover more than one specialty, such as Administration, Accounting Sciences, and Tourism, production and number of researchers and journals can differ substantially, and the specialty with the highest volume of articles would end up dominating. Similarly, generalist journals in whichever area end up publishing more items that those with specialized subjects. This may require correcting, which could be done by dividing the merely multiplicative score by the number of doctors or programs dedicated to the journal theme of subarea. However, creativity already shown by the areas, concerning the development of indicators derived from citation metrics, allows one to admit that this issue, if relevant, will shortly be adequately solved.

Acknowledgements

I wish to give many, many thanks to the entire area of Administration, Accounting Sciences, and Tourism for the personal and professional welcome, and for allowing me to participate in the construction of a strong and consolidated system of quality scientific journals in the area, which today includes SPELL, a top-notch indexer developed by the area itself.

REFERENCES

• Capes - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (2008). Reestruturação do Qualis. Retrieved from http://www.capes.gov.br/images/stories/download/avaliacao/Restruturacao_Qualis.pdf
  » http://www.capes.gov.br/images/stories/download/avaliacao/Restruturacao_Qualis.pdf
• Capes - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (2012). Comunicado n^o 003/2012 - Área Interdisciplinar: Atualização do Webqualis da área - Ano base 2010. Retrieved from https://www.capes.gov.br/images/stories/download/avaliacao/Qualis_Interdisciplinar2010.pdf
  » https://www.capes.gov.br/images/stories/download/avaliacao/Qualis_Interdisciplinar2010.pdf
• Capes - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (2015). Relatório do processo de classificação de periódicos da Área de Administração, Ciências Contábeis e Turismo, Quadriênio 2013-2016. Retrieved from http://www.capes.gov.br/images/documentos/RelatorioQualisAdministracao-2015final.pdf
  » http://www.capes.gov.br/images/documentos/RelatorioQualisAdministracao-2015final.pdf
• Coercive citation. (2016, September 16). In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/w/index.php?title=Coercive_citation&oldid=739759440
  » https://en.wikipedia.org/w/index.php?title=Coercive_citation&oldid=739759440
• Comissão para avaliação de periódicos científicos em Administração, Turismo e Contábeis (2002): Processo de Avaliação 2002 (p. 3). Documento de circulação restrita à área, distribuído aos integrantes do Comitê Capes, aos editores científicos e aos coordenadores de programa de pós-graduação. Retrieved from http://www.infocien.org/QualAval/Hist/Capes/AdmConT
  » http://www.infocien.org/QualAval/Hist/Capes/AdmConT
• Comissão para avaliação de periódicos científicos em Administração, Turismo e Contábeis (2004): Processo de Avaliação 2003-2004 (p. 5) Documento de circulação restrita à área, distribuído aos integrantes do Comitê Capes, aos editores científicos e aos coordenadores de programa de pós-graduação. Retrieved from http://www.infocien.org/QualAval/Hist/Capes/AdmConT
  » http://www.infocien.org/QualAval/Hist/Capes/AdmConT
• Comissão para avaliação de periódicos científicos em Administração, Contabilidade e Turismo (2005): Processo de Avaliação 2004-2006 (p. 5) Documento de circulação restrita à área, distribuído aos integrantes do Comitê Capes, aos editores científicos e aos coordenadores de programa de pós-graduação. Retrieved from http://www.infocien.org/QualAval/Hist/Capes/AdmConT
  » http://www.infocien.org/QualAval/Hist/Capes/AdmConT
• Comissão de Administração, Ciências Contábeis e Turismo (2007): Diretrizes de Avaliação de Periódicos 2007-2009 (p. 7) Documento de circulação restrita à área, distribuído aos integrantes do Comitê Capes, aos editores científicos e aos coordenadores de programa de pós-graduação. Retrieved from http://www.infocien.org/QualAval/Hist/Capes/AdmConT
  » http://www.infocien.org/QualAval/Hist/Capes/AdmConT
• Davis, P. (2012, april 10). The emergence of a citation cartel [Blog post]. Retrieved from https://scholarlykitchen.sspnet.org/2012/04/10/emergence-of-a-citation-cartel/
  » https://scholarlykitchen.sspnet.org/2012/04/10/emergence-of-a-citation-cartel/
• Döbereiner, J (2001). A publicação científica vista por um pesquisador. In P. Trzesniak (Org.), X Curso de Editoração Científica: notas de aula compiladas (pp. 7-11). Petrópolis, RJ: Associação Brasileira de Editores Científicos.
• Jeannin, P. (2002). Pour une "revuemétrie" de la recherche en Sciences sociales. Un point de vue français (communication au Conceil scientifique du CNRS). Retrieved from http://pertimm.dsi.cnrs.fr/data/intranet/sc/moyenscommuns/www.cnrs.fr/comitenational/cs/exposes/Jeanin.pdf
  » http://pertimm.dsi.cnrs.fr/data/intranet/sc/moyenscommuns/www.cnrs.fr/comitenational/cs/exposes/Jeanin.pdf
• Mugnaini, Rogério, Digiampetri, Luciano Antonio, & Mena-Chalco, Jesús Pascual. (2014). Comunicação científica no Brasil (1998-2012): indexação, crescimento, fluxo e dispersão. Transinformação, 26(3), 239-252. https://dx.doi.org/10.1590/0103-3786201400030002
  » https://dx.doi.org/10.1590/0103-3786201400030002
• Pascutti, P. G. (2016). Entendendo o Qualis. Retrieved from http://www.abecbrasil.org.br/eventos/xxivcurso/palestras/quarta/pedrogpascutti.pdf
  » http://www.abecbrasil.org.br/eventos/xxivcurso/palestras/quarta/pedrogpascutti.pdf
• Strehl, L. (2013). Indicadores de produção e impacto das publicações científicas. Retrieved from http://pt.slideshare.net/leticia.strehl/indicadores-hcpa2-22448626?qid=71f623cb-6492-4bbb-8680-a92618647d9a&v=qf1&b=&from_search=5
  » http://pt.slideshare.net/leticia.strehl/indicadores-hcpa2-22448626?qid=71f623cb-6492-4bbb-8680-a92618647d9a&v=qf1&b=&from_search=5
• Strehl, L. (2015). As bases de dados e a cienciometria. Retrieved from http://pt.slideshare.net/leticia.strehl/as-bases-de-dados-e-a-cienciometria
  » http://pt.slideshare.net/leticia.strehl/as-bases-de-dados-e-a-cienciometria
• Trzesniak, P. (2006). As dimensões da qualidade dos periódicos científicos e sua presença em um instrumento da área da educação. Revista Brasileira de Educação 11(32), 346-377. doi: //dx.doi.org/10.1590/S1413-24782006000200013.
  » https://doi.org/10.1590/S1413-24782006000200013
• Trzesniak, P. (2012). A questão do livre acesso aos artigos publicados em periódicos científicos. Em Aberto, 25(87), 77-112. Recuperado de http://emaberto.inep.gov.br/index.php/emaberto/article/view/2364/2326
  » http://emaberto.inep.gov.br/index.php/emaberto/article/view/2364/2326
• Trzesniak, P. (2014). Hoje vou escrever um artigo científico: a construção e a transmissão do conhecimento. In S. H. Koller, M. C. P. de P. Couto, J Von Hohendorff (Org.), Manual de produção científica (pp. 15-38). Porto Alegre, RS: Penso.
• Van Noorden, R. (2012, 29 de junho). Record number of journals banned for boosting impact factor with self-citations [Blog post]. Retrieved from http://blogs.nature.com/news/2012/06/record-number-of-journals-banned-for-boosting-impact-factor-with-self-citations.html
  » http://blogs.nature.com/news/2012/06/record-number-of-journals-banned-for-boosting-impact-factor-with-self-citations.html
• Van Noorden, R. (2013). Brazilian citation scheme outed: Thomson Reuters suspends journals from its rankings for 'citation stacking'. Nature, 500(7464), 510-511. doi:10.1038/500510a.
  » https://doi.org/10.1038/500510a
• Yamamoto, O. H., Koller, S. H., Guedes, M. C., Lobianco, A. C., Sá, C. P., Hutz, C. S. & Menandro, P. R. M. (1999). Periódicos científicos em Psicologia: uma proposta de avaliação. Infocapes, 7(3), 5-11.
• Yamamoto, O. H., Menandro, P. R. M., Koller, S. H., Lobianco, A. C., Hutz, C. S., Bueno, J. L. O., & Guedes, M. C. (2002). Avaliação dos periódicos científicos brasileiros da área de psicologia. Ciência da Informação, 31(2), 163-177. doi: http://dx.doi.org/10.1590/S0100-19652002000200017.
  » https://doi.org/10.1590/S0100-19652002000200017

Publication Dates

• Publication in this collection
  Dec 2016