EDITORIAL
Changes in the evaluation system for graduate programs
The Coordination Office for University-level Personnel Advancement (from portuguese Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, Capes), which manages the evaluation system for Brazilian postgraduate programs, seeks to imprint a dynamic nature on the process. For each cycle, which currently corresponds to a three-year period, the fields of knowledge and respective programs are evaluated according to the performance presented, especially with regard to intellectual production. The performance presented over one period serves as the basis for the evaluation criteria of the subsequent three-year period in progress). This strategy constitutes a stimulus or inducement towards developing different fields of knowledge, particularly for researchers who present distinct stages of consolidation within the scientific community. It is in this context that the expression "raising the bar" is usually used, as an analogy for the performance level required in the high jump in an athletics competition. Each time the athlete succeeds in jumping without knocking down the bar, it is put back again a little higher.
Within this spirit and over the course of the past decade, postgraduate programs have performed an important role in both quantitatively and qualitatively improving Brazilian intellectual production internationally. Moreover, they have stimulated researchers, teachers and students to face new challenges every three years, in the attempt to have the knowledge that has been produced published in periodicals with international circulation, which are increasing competitive and impactful.
Over the last few years, nine strata for classifying periodicals have been adopted for evaluating the scientific production from Brazilian postgraduate programs. These strata consist of three category levels (A, B and C) and three circulation levels (International, National and Local). Periodicals are classified in the different strata, going from the highest (International A) to the lowest (Local C). This system has received the name Qualis index, with a view towards qualifying and differentiating periodicals, thereby assisting in ranking the postgraduate programs. Despite the important role played by this system in the evolution of postgraduate programs and in the scientific development of each field of knowledge, Capes has detected a need for improvement in the present assessment criteria, through recent studies based on the production from the three year period 2004-2006. This is principally due to the deterioration in the power of this system to discriminate between postgraduate programs that has occurred over the years.
It can be seen that the discriminatory power of this system has become weakened through the observation that none of the 47 fields of knowledge has been using all nine of the strata. Few fields of knowledge have been publishing articles in the Local circulation strata, which makes these strata meaningless. A significant number of fields of knowledge have been making use of only four strata, while many have been publishing a high percentage of their papers only in the highest stratum (International A). This makes it difficult to discriminate between programs within the same field. In addition to the large imbalance in the distribution of the levels or the classifications of the periodicals between different fields, major fields and branches of knowledge, striking and growing heterogeneity in the classification given to the same periodical by different fields of knowledge has also been noted.
The results from the study carried out by Capes have stimulated many discussions and reflections on the stratification used until now, and have justified the need for improvement in the system. Among the desired changes, a single periodical can no longer be assessed differently between the areas that form parts of the same major field. Likewise, new indicators and indices have been incorporated, such as the h and SCImago indices of the Scopus database, thus enabling greater comprehensiveness in the evaluation, when combined with the well-known Journal Citation Reports® (JCR®), which is produced by ISI.
One point to highlight from the Capes study is that papers from many fields of knowledge are published in periodicals that are considered to belong to other fields or to correlated fields. This situation may occur for large numbers of reasons, from the progressive growth in multidisciplinarity itself to the small number of specialized periodicals in some fields that are indexed in regional ands international databases. Nevertheless, this is of concern because it may cause serious distortions in characterizing and consolidating each field of knowledge.
In an attempt to better identify where the knowledge from different fields is being produced, the Pareto principle was applied. This addresses the concentration of articles within a set of periodicals. Through this, it has been established that the periodicals in which a given field of knowledge publishes 79% of its papers are classified as those of greatest concentration (P1), followed by those of lower concentration (P2 and P3). The field that is responsible for the greatest number of articles published in a periodical classified as P1 is named the mother field, which will define the stratification of this periodical. This should be followed by other sister fields (areas within the same major field). Proximity or familiarity of the periodical with other major fields of the same branch of knowledge will be named cousin fields, resulting in going down just one stratum. Dropping down two strata will occur when the periodical belongs to an In-law Field, id est, a field within another branch of knowledge.
The Capes Technical-Scientific Council for University-level Education (CTC-ES) has approved eight strata for the new classification system for periodicals. Normative Instruction nº. 5 from the Assessment Board explains that these new strata will be constituted thus: A-1 (highest level), A-2, B-1, B-2, B-3, B-4 and B-5 (lowest level), followed by a final one, level C, which will accommodate the periodicals that are considered inappropriate for publishing postgraduate production. Each major field of knowledge should reclassify the periodicals in which it published during the previous three-year period, into these new strata, with the aim of restoring homogeneity of the levels of these periodicals for all the areas within the field.
In the light of these changes, what will be their impact on area 21? Independent of any pessimistic forecasts regarding "raising the bar", it has to be borne in mind that the likely impact will depend on the reactions among those involved, along with the present stage of development of this field. There is no doubt that there is an urgent need for internal reorganization of the production of knowledge and, above all, a need for special attention towards improving the indexation of the periodicals corresponding to the mother field of area 21. It is believed that, although these changes represent an enormous amount of reorganization work over the short term, with a direct impact on the next three-year evaluation, they may have a very positive inducement effect on the qualitative development of the field over the medium and long term.
Today, one of the greatest challenges faced by area 21 is to expand the small number of national periodicals indexed in regional and international databases, especially SciELO, Scopus and ISI (JCR®). It also has to be borne in mind that the evaluation of postgraduate programs should be characterized as an instrument not only for measuring and classifying, but also as a strong agent for stimulating, encouraging and inducing the scientific development of each field of knowledge.
Dirceu Costa
Coordinator of area 21 in Capes
Juarez Vieira do Nascimento
Adjunct Coordinator of area 21 in Capes
1 CNPq/AET, Tabelas 1.4.1, 1.4.2, 1.4.4, 2008.
2 Dados provenientes de acompanhamento pessoal do sistema nos últimos anos.
Publication Dates
-
Publication in this collection
13 Oct 2008 -
Date of issue
Aug 2008
