He who ignores the past is doomed to repeat it: distortions of postgraduate evaluation in Brazil based on Capes’ history

Abstract

This paper aims to highlight some distortions in the Capes evaluation system, particularly in the area of Administration, Accounting and Tourism, starting from the history of Capes in Brazil. Through this analysis, it was possible to conclude that the change in the assignment of grades from 1 to 5 to 1 to 7 in the 1990s was not followed by a change in the Capes evaluation system, which still uses scores from 1 to 5 to evaluate the programs in each of their requirements. Such distortion becomes incompatible with an evaluation system that is summative and classificatory. As a consequence, productivity is confused with productivism, methodological rigor for the evaluation of programs is jeopardized, and the educational character of a formative evaluation is abandoned.

Keywords:
Postgraduate; Teaching; Research; Capes; Evaluation.

Resumo

Este artigo tem por objetivo apontar algumas distorções no sistema de avaliação da Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Capes) no Brasil, em particular na área de Administração, Ciências Contábeis e Turismo, a partir da história da própria Capes. A análise dessa história possibilitou concluir que a mudança da atribuição de notas de 1 a 5 para 1 a 7, adotada nos anos 1990, não se fez acompanhar de uma mudança do sistema de avaliação da Capes, que ainda se vale de notas de 1 a 5 para avaliar os programas em cada um de seus quesitos. Tal distorção se mostra incompatível com um sistema de avaliação somativo, classificatório. Como consequência, confunde-se produtividade com produtivismo, compromete-se o rigor metodológico para a avaliação dos programas e abandona-se o caráter educativo de uma avaliação formativa.

Palavras-chave:
Pós-graduação; Ensino; Pesquisa; Capes; Avaliação

Resumen

Este artículo tiene como objetivo señalar algunas distorsiones en el sistema de evaluación de la Capes, específicamente en el área de Administración, Ciencias Contables y Turismo, desde la historia de la propia Capes. El análisis de esta historia lleva a la conclusión de que el cambio de atribución de notas de 1 a 5 a de 1 a 7, realizado en los años 1990, no fue acompañado por un cambio en el sistema de evaluación de la Capes, que todavía se vale de notas de 1 a 5 para evaluar los programas en cada una de sus cuestiones. Tal distorsión es incompatible con un sistema de evaluación sumativa, clasificatoria. En consecuencia, se confunde productividad con productivismo, se compromete el rigor metodológico para la evaluación de los programas y se abandona el carácter educativo de una evaluación formativa.

Palabras clave:
Posgrado; Enseñanza; Investigación; Capes; Evaluación

INTRODUCTION

The National Campaign for the Improvement of Higher Education Personnel (Capes) evaluation system was implemented in 1976 with the goal of establishing the quality standard expected from Master’s and Doctoral courses in Brazil. It originally aimed to train professors and researchers for Brazilian universities. In 1988, there was a shift in priority from training to evaluation with a clear focus on research and scientific publications. Activities related to extension, teaching, and cooperation, for instance, were neglected in the evaluation process of professors and stricto sensu Post-Graduation Programs (PGP) (HORTA and MORAES, 2005; KUENZER and MORAES, 2005).

Productivistic culture has transformed the process of knowledge into a commodity, limiting it to products, performance levels, and quality standards (MOREIRA, 2009). It is important to point out Capes does not evaluate the professors’ individual performance, but rather the PGP’s. However, in reality Capes’ criteria pressure the professors to better their individual performance (NASCIMENTO, 2010). Therefore, it is important to study academically the contributions of this system.

Evaluation of Brazilian post-grad programs, despite being thoroughly debated (MATTOS, 2008, 2012; BIANCHETTI and MACHADO, 2009; MOREIRA, 2009; NASCIMENTO, 2010; TREIN and RODRIGUES, 2010; MASCARENHAS, ZAMBALDI and MORAES, 2011; ALCADIPANI, 2011), has not generated solutions to the contradictions pointed out by professors and researchers. This study aims to unearth the history of Capes and how the PGP’s evaluation in Brazil has changed throughout the years into what it is today. From a practical point of view, it is justified because it studies the influence of Capes’ evaluation system on Master’s and Doctoral programs. From a theoretical point of view, it allows for a deeper debate, bringing historical elements that permeated the current process of institutional evaluation. After all, those who do not know their history are doomed to repeat it.

This essay is divided in four parts, the first being this introduction. The second is the description of the state of the art Capes, divided in three moments (beginning, maturity, and turning point). The third describes the Post-Graduation evaluation system in Brazil and how Capes’ evaluation influences institutions. At the end, you will find the final considerations.

CAPES AND ITS HISTORY

Capes was created in 11 July 1951 with the goal of training enough qualified professionals to answer the needs of public and private enterprises in order to aid in the development of the country (MACCARI, RODRIGUES, ALESSIO et al., 2008; VIANA, MANTOVANI and VIEIRA, 2008; CAPES, 2012). Among its main activities, Capes is responsible for the evaluation of stricto sensu Post-Grad programs, scientific production access and dissemination, investment in the creation of high level resources in the country and abroad, the promotion of international scientific cooperation, and the induction and fomentation of professors’ initial and continued formation (VIANA, MANTOVANI and VIEIRA, 2008).

Institutional pressures for publication created by regulations are seen as the driving force behind productivistic culture, causing changes in routine and norms, complaints (MATTOS, 2008), objections from the interested parties, and alterations in the professors’ activities. Those concerned start worrying about the logic of the system, questioning productivistic culture (LUIZ, 2006), or developing activities to better their evaluation and ensure their reaccreditation as a PGP professor. To understand the causes and offer resources to consider the Post-Grad evaluation system, Capes’ history was divided in three moments: (a) beginning (1951 to 1981): quality teaching as a priority; (b) maturity (1982 to 1989): consolidation of the Post-Graduation evaluation model; and (c) turning point (1990 to present day): focus on the evaluation of the professor-researcher, and not only the professor, as shown in Figure 1.

Figure 1
Timeline - Capes

Beginning: 1951 to 1981

Before even Capes, it was created the national system of Post-Graduation, in 1950, guided by two main reasons: the need for highly qualified personnel in the market and the need for highly qualified faculty members (SGUISSARDI, 2006). In this manner, an initial group of specialists allowed for the creation of a system of Post-Graduation that consolidated itself mainly through the State’s investment in professors being formed in North American and European universities, as well as having professors from these countries in Brazil (MOREIRA, 2009). With that, the National System of Post-Graduation had the mission of training professors and researchers (PNPG, 2010).

In 1952, Brazil starts seeing a strategic importance in scientific and technological development, as well as industrial development. It is in this moment that the Brazilian Development Bank (BNDE) was created and started acting as a financing agency (PNPG, 2010). In 1953 was created the University Program for Higher Education Institutions (HEI). At that same time, the then head of the program Anísio Teixeira hired foreign visiting professors, fostering exchange programs and international cooperation between institutions, aside from granting 79 scholarships (in the country and abroad) and supporting scientific events (CAPES, 2012). In 1954, this concession was expanded to 155 scholarships.

Post-Graduation programs were created in 1960 as a way of allowing the country to have a competitive edge. In 1961 Capes became subordinate to the Presidency. But in 1964 it falls once again under the jurisdiction of the Ministry of Education (MEC). On that same year, Capes and other public organs were given the mission of consolidation the Post-Graduation policy (CURY, 2005; BIANCHETTI and SGUISSARDI, 2009; NASCIMENTO, 2010; DANTAS, 2012).

The first classification of 38 Post-Graduation courses - 27 Masters programs and 11 Doctoral programs - in Brazil happened with the arrival of the new director Susana Gonçalves in 1965 (CAPES, 2012). In that same year, Report 977, inspired by the American university structural model and approved by the Federal Education Council (CFE), regulated Post-Graduation in Brazil considering its definition, objective, and the levels of Master and Doctor (DANTAS, 2012). In 1966, the government presented the Institutional Development Plan (PDI), which gave Capes new duties and budgetary options to multiply its actions and intervene in the qualification of faculty members of Brazilian universities. Capes became a central figure in the development of the new policy for Post-Graduation (CAPES, 2012). Licensing standardization for stricto sensu Post-Graduation courses was established by another CFE report in 1969 (CURY, 2005; MACEDO and SOUZA, 2010; NASCIMENTO, 2010; DANTAS, 2012).

With the creation of the Evaluation Program for Post-Graduation Courses in 1976 by Capes, some goals were defined to justify the conception and adoption of these programs: (a) facilitating the distribution of scholarships for Masters’ and Doctoral students; (b) guiding the investment of federal agencies in the creation of high-level human resources; (c) subsidizing educational policy related to Post-Graduation and to universities as well as their relationship to basic education; (d) creating a permanent system of information on Brazilian Post-Graduation (MACCARI, LIMA and RICCIO, 2009); (e) regulating the expansion of Post-Graduation programs, recommending support to new programs and progressively incorporating them to the evaluation and monitoring system; and (f) certifying these courses, validating at a national level the diplomas they grant (MACCARI, LIMA and RICCIO, 2009).

The results of this process since the creation of Capes are the basis for the discussion of which programs would have their certification renewed for the next three years (CAPES, 2006; VIANA, MANTOVANI and VIEIRA, 2008). However, it was from the 1980s onward that the system of Post-Graduation evaluation in Brazil took root with the introduction of forms for obtaining the programs’ data and with continuous informatization, such as the addition of specialists in each area of knowledge for in loco visits to the Programs (KUENZER and MORAES, 2005).

Maturity: 1982 to 1989

Between 1982 and 1989 Capes experienced a period of stability, since the transition to the New Republic in 1985 did not cause major changes in it or in society in general. In 1982 Decree 86,816 (BRASIL, 1982) altered Capes’ functions, defining its goals as: (a) elaborating the National Post-Graduation Plan (PNPG) and coordinating its execution; (b) following up on and evaluating Post-Graduation programs as well as the interaction between teaching and research; and (c) maintaining exchanges and contact with other public administration or private organs, national or foreign, aiming to celebrate agreements, concords, contracts, and adjustments related to Post-Graduation and the improvement of higher-education personnel (CURY, 2005; KUENZER and MORAIS, 2005; MACEDO and SOUZA, 2010; NASCIMENTO, 2010; DANTAS, 2012). The II PNPG (1982) was expected to be put in practice between 1982 and 1985 and proposed the institutionalization of evaluation as an instrument for controlling the quality of teaching (CAPES, 2004) and for creating human resources qualified for teaching and research activities (PNPG, 1982). In this context, institutionalization and improvement of this evaluation become paramount (KUENZER and MORAES, 2005).

Although this was a period of stability, there was a moment of agitation in the Brazilian academic community due to the publication of the list of 56 Post-Graduation programs that had been given the worst possible grade in Capes’ evaluation. This information was at the time printed at the Estado de São Paulo newspaper and led to a debate that got considerable attention from the press (CASTRO and SOARES, 1983). That led universities to display their programs’ grade. Later, Capes made public the list of all the HEI’ grades. Capes’ evaluation in 1985 took into consideration only the number of publications; evaluations from 1989 onwards took into consideration both the number and the quality of publications (STRATHERN, 1997; FONSECA, 2001).

The III PNPG (1985), executed between 1986 and 1989, had as its guideline the institutionalization of scientific research at universities (CAPES, 2004) as well as the consolidation and improvement of the Post-Graduation programs performance and their integration to the system of science and technology, including the productive sector (PNPG, 1985). Aiming to fulfil this ambition, the III PNPG prioritized research at the universities, emphasizing their role in national development and the integration of Post-Graduation to the system of science and technology (PNPG, 1985). This was not enough, however, to overcome the tradition of Post-Graduation being intensively centered on teaching.

Up until the 1990s, Capes (2004) implemented and consolidated Brazilian Post-Graduation focusing on scientific advancement at the fringes of knowledge, though it received effective collaboration from the National Council for Scientific and Technological Development (CNPq) and from the Funding Authority for Studies and Projects (FinEP).

Turning point: 1990 to present days

The model introduced by Capes to evaluate Brazilian Post-Graduation in 1996 and 1997 consolidated itself in the following triennium, further effecting change in the conception and direction of politics for this level of education in the country (HORTA and MORAES, 2005). In 1990, during the Collor administration, Provisional Measure 150 extinguished Capes, causing intense mobilization (BRASIL, 1990). Research and Post-Graduation Dean’s Office in universities mobilized academic and scientific opinions, who, aided by the Ministry of Education, managed to revert the measure in April 12th of that same year through Law 8028 (CAPES, 2012; BRASIL, 1990). During that time, Capes aimed to keep focusing on its previous plan and add quality to Post-Graduation activities, using the evaluation process that was already there in a developing state since 1976 and which was perfected and institutionalized. There was also a better division of the areas of knowledge as well as the adoption of stricter criteria for assigning grades to the programs (MACCARI, LIMA and RICCIO, 2009). In the early 1990s, it became clear both to the agency and to the academic community that the current evaluation paradigms had been exhausted and that a new evaluation model was urgently needed (KUENZER and MORAES, 2005).

In 1992, Law 8405 authorized public power to institute Capes as a Foundation, making it a main subsidiary of the Ministry of Education for creating politics for Post-Graduation and giving the Institution a new beginning (BRASIL, 1992). Since then, it is in Capes’ authority to develop, evaluate, and observe quality standards for stricto sensu Post-Graduation in the country (CAPES, 2004; MACCARI, RODRIGUES, ALESSIO et al., 2008). In 1995, Capes was restructured, gaining strength as the institution responsible for monitoring and evaluating these programs (CAPES, 2012). During that same period, the Post-Graduation system had over 1000 Masters programs and 600 Doctoral programs. Since then, Capes has been consolidating its position as a promotion agency, even granting scholarships and grants and developing other mechanisms to enable to formation of highly qualified scientists and human resources to answer the demands of research and higher education institutions (MACCARI, RODRIGUES, ALESSIO et al., 2008; CAPES 2012).

Capes’ first evaluation model focused on the visitation of consulting commissions to the programs to assess their workings and offer assistance in perfecting them. Capes’ current evaluation model can be seen as linked to the regulation and control process (SGUISSARDI, 2006). The institution understood it could not still use the same criteria to evaluate Post-Graduation programs as it did in the 1970s. The intense growth of programs rendered the consulting commission visitation evaluation model unfeasible. This radical change aimed to allow greater discrimination between the programs, leading to a new evaluation paradigm introduced in 1996-1997.

From 1998 onwards, certain changes were made to the evaluation process. Capes decided that, among other changes, the evaluation system would: (a) focus on the PGP, not on the performance of each individual program they offered (Masters and Doctoral); (b) express a grade in a scale of 1 to 7 without fractions (5 being the equivalent of the old A grade and programs with that grade being able to match excellence criteria to be graded as 6 or 7 exceptionally (FONSECA, 2001); (c) have to reflect the concern with a high level of quality in accordance with international standards; (d) attempt to stimulate flexibilization of the Post-Graduation models and the offer of quality education with adequate average duration; (e) verify the relationship between the program’s project and the effective impact of their actions on the institution and on society, promoting permanent self-evaluation in each program; and (f) try to reconcile the use of quantitative indicators and standardized criteria with the appraisal of aspects related to the dynamics and specificity of each program (CAPES, 1998; MOREIRA, 2009).

The fourth PNPG (2004), to be executed between 2005 and 2010, reaffirmed that evaluation should be based on quality and excellence of the results, in the specificity of the areas of knowledge, and in the impact these results have in the academic and corporate communities as well as in society (CAPES, 2004), and that research should be institutionalized in universities, to ensure Post-Graduation programs function properly (PNPG, 2004). As for the evaluation model, the fourth PNPG registers that Capes and CNPq have created along their trajectories two important systems for national academic evaluation: (a) quality evaluation in the programs of human resources (post-graduates) generation; and (b) individual evaluation of the researchers and research groups’ leaders (CAPES, 2004; DANTAS, 2012).

In 2007 the National Congress approved Law 11502, thus creating the new Capes, which not only coordinated the high standard for the Brazilian National Post-Graduation System, but also started to induce and promote initial and continued formation of teachers for basic education (BRASIL, 2007). This was consolidated by 2009’s Decree 6755, which instituted the National Policy for the Formation of Teaching Professionals for Basic Education (CAPES, 2012; BRASIL, 2009). It is important to note that up until this point Post-Graduation has consolidated itself little by little with the privileged area of research, which joins teaching and guidance activities, as well as guidelines for publications by professors and students (MOREIRA, 2009).

The fifth PNPG (2010), to be executed between 2011 and 2020, is responsible for continuing previous plans, especially those of the fourth PNPG - which was not executed properly due to problems of several kinds - as well as introducing new and important inflections, lasting longer than the previous PNPG (PNPG, 2010).

The new evaluation model, considered by some a turning point (SOUZA, 2008; PNPG, 2010; DANTAS, 2012), changes the focus from teaching to research. With this new evaluation instrument, some indicators became evident: (a) the introduction of the idea of whole programs being evaluated, instead of separating between Masters and Doctoral programs; and (b) special attention being given to research lines and their organicity with subjects, projects, and research products, theses, and dissertations. Instead of the professors’ preferences, research lines now define: (a) curricular pathways, organized based on research and not on subjects; (b) seminars for research and dissertation; (c) the definition of tutors at the beginning of the courses; and (d) the objects of investigation as determiners of curricular pathways, which is now more flexible. In this model of Post-Graduation focused on research, intellectual, bibliographic production is now more important and justified as an evaluation indicator (FONSECA, 2001; CURY, 2005; BIANCHETTI and SGUISSARDI, 2009; MACEDO and SOUZA, 2010; NASCIMENTO, 2010; ALCADIPANI, 2011; MACHADO and BIANCHETTI, 2011; DANTAS, 2012). With this, the radical changed Capes aimed for had the goal of correcting the pyramid, which would allow discrimination between programs (FONSECA, 2001) given that many courses had been achieving the maximum grade of 5 at that moment. Social insertion, meaning serving regions of less favored public, and internationalization started being important factors for the distinction between programs graded 5, 6, and 7.

In 2011 the Ciência sem Fronteiras (Science Without Borders) program was launched. It was partially interrupted in 2016. Still in 2016 Portal EduCapes was to be launched aiming to promote knowledge through free access to basic and higher education students, teachers, and professors.

The PGPs data is currently collected annually through an online platform called Sucupira. Teaching institutions are responsible for filling in the data and the Sucupira is integrated with the Lattes Platform. From the period of 2013-2016 evaluation started occurring every four years instead of every three years. Visitations by evaluation committees to HEIs only happened in special occasions or when the Area’s Committee required diligence. Programs with grades 1 and 2 are not recognized by MEC and programs with grades 6 and 7 are considered excellence programs (LEITE, VIANA and PEREIRA, 2006).

The Capes evaluation model is legitimized by the Institution and many other Brazilian official regulation agencies. This model, however, is profoundly ambiguous. At the same time the HEIs recognized it as legitimate, they also fear it because of its nature of regulation and control in the name of the Government (SGUISSARDI, 2006). Although severely questioned due to its criteria, it is seen as a decisive factor in maintaining a high level in post-graduation programs in Brazil and in increasing research (SOUZA and BIANCHETTI, 2007; MOREIRA, 2009).

As postulated in the final considerations of this article, this model’s controversy is partially created by the fact that evaluation is made based on an old paradigm, with grades ranging from 1 to 5, even after evaluation criteria were changed (in 1996-1997) in an attempt to distinguish programs with the highest grade, introducing grades 6 and 7. The Sucupira Platform system, used during evaluation week, calculates the programs inside this interval. At the area of Administration, programs which have achieved a grade of 5 are once again graded from 5 to 7 after evaluations are uploaded to Sucupira. We believe that this is an important contributor to contradiction in the evaluation of Post-Graduation and to the polemic present in the academic community. For this reason, this historical record is justified and important when considering this theme.

POST-GRADUATION PROGRAMS EVALUATION SYSTEM AS CONDUCTED BY CAPES

The evaluation system coordinated by Capes is used as a quality standard in public and private stricto sensu Post-Graduation teaching, encompassing all areas of knowledge in every Brazilian state. In this section, the presentation of criteria and data relative to the Programs is limited to Masters and Doctoral Programs in Administration. Professional Masters Programs were not considered in this approach.

To perform this evaluation, Capes counts on the support of appointed consultants who base their decisions based on the reports produced by the Sucupira Platform from data sent by the programs under the responsibility of their Coordinator. The evaluation of 2013-2016, according to Capes’ Area Document (CAPES, 2017), is based on five criteria: program’s proposition (weight 0), teaching staff (weight 20), student body, theses and dissertations (weight 35), intellectual production (weight 35) and social insertion (weight 10). Each of the items is composed by subitems with their respective weights as shown on Box 1 .

Box 1
Evaluation Criteria - Capes (1 to 5)Source: Administration Area Document (CAPES, 2017)

Given that the evaluation model is graded from 1 to 5, at first the programs are included in these five levels and those with Doctoral courses graded 5 are evaluated using new criteria based on international and excellence standards indicating those capable of being raised to levels 6 or 7. Programs graded under 3 are de-accredited from the system (HORTA and MORAES, 2005).

In this second phase of the evaluation, programs with Doctoral courses given the highest grade (5) are nominated and carefully evaluated in order to verify quality levels associated to international insertion indicators for their areas and other differential indicators as applicants for one of these two levels (6 or 7). According to the 2017 Four Year regulation, only PGPs graded 5 and VG (Very Good) concepts in all aspects of the evaluation form will be considered for the grades 6 and 7 (CAPES, 2017). It is important to point out that there are five concepts, each representing a grade. In the evaluation of items and aspects, the VG concept corresponds to grade 5, and thus respectively: G (Good) = 4; R (Regular) = 3; W (Weak) = 2; and D (Deficient) = 1.

For a grade 6, a PGP needs to have predominantly VG concepts in all items of an aspect of the evaluation form, even with the eventual G concept in some items, and for grade 7, a VG concept in every item of all aspects of the evaluation form. The same rules demand that the PGPs perform as well as international reference centers forming doctors in their areas, have a differentiated performance level in relation to other PGPs in their areas, and characterize their solidarity and nucleation (CAPES, 2017).

According to the most recent document in the area (CAPES, 2017), the PGPs which reach a grade of 5 must present the following qualifiers to be eligible for grades 6 or 7: (a) a stable Permanent Faculty Nucleus (PFN); (b) scientific publications in journals with impact factor or books classified by the area as L4; (c) consolidated history in the formation of doctors; (d) regular recent flow in the formation of doctors; and (e) adequate distribution between professors of cooperation with groups acting outside of Brazil, scientific production, formation of students, and academic leadership, showing that the PGP is not dependent on the contribution of one or a few professors, thus ensuring its high level performance for a period of time.

The PGPs leadership in the national environment is analyzed based on: (a) the percentage of doctoral alumni acting in the faculty staff in other Post-Graduation institutions or HEIs in the country; (b) the PGP’s impact in the management of public or private organizations; (c) post-doctoral students who were not doctoral students at the PGP; (d) the leadership of national entities related to their area of knowledge; (e) how many students from different regions of the country it attracts; (f) the number of students from other national PGPs enrolled in its subjects or in partial stages of their research; (g) leadership in research projects financed by public or private agencies in cooperation with professors of other national PGPs; (h) the participation of permanent professors in committees of research regulation and support agencies; (i) awards given to permanent professors or students and alumni who received awards from national entities; and (j) participation and leadership in programs in regions with little to offer in the formation of qualified personnel in their area (CAPES, 2017) such as, for instance, Minter and Dinter.

Aside from checking the PGP’s national leadership, their degree of international insertion is also verified as a way of evaluation their degree of equivalence to centers located abroad, regarding the quality of their teaching. In the area of Administration, this criterium has as its main indicators (CAPES, 2017):

1. PGP’s alumni acting abroad;

2. professors who were directors or presidents of the international scientific society;

3. professors who participate in editorial committees of journals edited abroad and part of Scopus or Web of Science;

4. participation in agreement or research project with international financing;

5. international scientific meetings organized by the PGP;

6. participation in Post-graduation monitoring panels/committees abroad;

7. foreign doctoral students who partially develop their projects at the PGP;

8. Brazilian doctoral students who partially develop their projects at the a foreign HEI;

9. articles written with researchers from foreign institutions;

10. books or book chapters published by renowned international publishing houses;

11. participation in international networks in their areas;

12. courses ministered at the PGP by foreign researchers;

13. workshops or seminars ministered at the PGP by foreign researchers;

14. tutoring or co-tutoring of a student from the PGP by foreign professors;

15. tutoring or co-tutoring of a foreign student by a permanent professor of the PGP;

16. post-doctoral internship of foreign doctors at the PGP;

17. professors who have participated in scientific events abroad;

18. professors and alumni who received awards from international entities for work conducted at the PGP;

19. students who have taken courses abroad;

20. foreign students enrolled on subjects at the PGP;

21. permanent professors who have attended post-doctoral programs abroad during those four years;

22. recruitment of foreign researchers for the PGP’s faculty body;

23. cooperation agreements between the PGP and foreign partners;

24. double titulation agreements with international institutions;

25. subjects in other languages;

26. offices to receive researchers;

27. international accreditations;

28. room at the PGP to house foreign professors and students.

Four conclusions can be derived from the Capes’ Post-graduation Programs Evaluation System. Firstly, there are two sequential evaluations: the first one with grades up to 5; and the second one for programs graded 5 in order to select candidates for grades 6 or 7. Knowing the history behind Capes and the way Post-Graduation evaluation was structured help understand why. With the change that came in 1990, the highest grade went from 5 to 7. However, the evaluation system grading up to 5 still exists while new criteria were created for an eventual upgrade in the evaluation of PGP’s graded 5. This creates a distortion on the normal distribution curve of the PGPS in Administration, leading to a higher concentration of courses with lower grades, as can be seen on Figure 2.

Figure 2
Distribution of Post-Grad programs in Administration/Grade

Secondly, there is a need to distinguish academic productivity from academic productivism. The Post-Graduation evaluation criteria in their five aspects (Box 1) show that intellectual production is just one of the evaluation criteria and represents 35% of the concept. Shigaki and Patrus (2012) have objectively shown that intellectual production is not enough to reach grade 5 on the evaluation of academic PGPS on the area of Administration. Focusing only on publication does not guarantee the highest grade. The number of approved dissertations and theses, the time of conclusion for Masters and Doctoral programs, and the quality and participation of the PFN are important variables in the final grade, not to mention the importance that evaluation places on the Program’s proposition, its coherence, and the articulation between researches, publications, and conclusion papers with research lines.

Thirdly, it is important to note the evaluation’s role and purpose. Scriven (1967) classifies evaluation as either summative or formative. The first aims to provide information as to the use of the evaluated model for an educational system. Formative evaluation, however, aims to continuously perfect the curriculum. A few years after Scriven, authors Bloom, Hasting, and Madaus (1971) extended these learning evaluation concepts: the first, summative, was used to either move the student forward or to hold them back, such as the SATs, and the second, formative, was used during the process of teaching-learning as diagnosis and prognosis for perfecting. The Brazilian Post-Graduation academic community must answer this question: what form of evaluation do we want, summative or formative?

Since publication has also become a condition for obtaining financing for research and resources, such as productivity grants, better grades in Post-Graduation ranking, prestige among peers, participation in national and international academic events, among others (TREIN and RODRIGUES, 2011; RODRIGUES, 2007), it seems Post-Graduation’s evaluation paradigm is summative. If it were formative, criteria would be defined a priori to guide the perfecting of the PGP and not a posteriori to classify it competitively among others. In this way, science, which brings results through collaboration, cooperation, and research, is replaced by competition (FREITAS, 2011).

By relating the first and the third conclusions, we arrive at a fourth conclusion that deserves attention: if the PGP’s evaluation conducted by Capes is summative, the double evaluation presented in the first conclusion points to a methodological flaw in the way the grades are attributed to programs, compromised the evaluation’s scientific rigor. This flaw is derived precisely from the double evaluation for programs graded 5 and not a classification among all programs. As seen before, the second evaluation for programs graded 5 is not quantitative, present on the indicators through metric and averages, but rather on criteria and their subjective evaluation. Not only that: it is done during the evaluation week in Brasília, only after the programs are evaluated and classified. The first evaluation previously conducted by consultants, who write a report and give a grade that will be confronted, discussed, endorsed, ratified, or rectified initially by two consultants and later in plenary.

It is, therefore, important to reconsider Capes’ evaluation and its distortions. Many of them arise from its history, here described. Capes back in the 1990s changed its evaluation system, removing importance from teaching and prioritizing research. In loco visitations by consultants gave way to a complex evaluation system. The need for greater distinction between programs graded 5 led Capes to introduce criteria such as internationalization, leadership, and solidarity, leading to the higher grades, 6 and 7. The evaluation model kept its grade paradigm from 1 to 5, leading to each three-year (now for-year) period having a double evaluation for programs graded 5 to find possible candidates for grades 6 and 7. Therein lies, in our opinion, one of the origins of the distortion of the current evaluation model. Finally, evaluation takes on a summative, qualifying, aimed at ranking programs that compete for public research grant, abandoning the evaluation’s formative character, perverting the system. This includes the definition of goals and the classification of journals a posteriori with a retroactive evaluation. Even accepting summative evaluation as a criterium for scarce resources, it has been compromised by the fact that it is not based on a normal curve distributed between grades 1 to 7.

FINAL CONSIDERATIONS

Even if we regret the lack of attention given to teaching, extension, cooperation with the business and government sectors (SPAGNOLO and SOUZA, 2004), we must admit research’s priority over teaching in the evaluation process of Brazilian Post-Graduation programs. This paradigm shift (a “turning point”, according to specialists) however brings contradictions in a system that demands at the same time integration of stricto sensu Post-Graduation and Graduation programs. We can only focus on contributions from this historical journey to propose two things.

The first is that we must think of a Post-Graduation system that respects diversity. We are not talking about regional diversity present in a country with such levels of inequality, but to the diversity of vocations in each PGP in Administration. There are programs that have no vocation for internationalization. And this goal may legitimately not be in their strategic planning for the foreseeable future. To put it in another way, there are programs that do not aim for a score of 7 because they do not have the resources to fight for it. However, items and aspects metric grade up only to a score of 5. To achieve this score of 5, at least in the last triennial evaluation (2011-2013), a program needed a global VG (Very Good) evaluation and at least a G (Good) concept in Faculty Members and Student Body, with no R (regular) concept in the items of these aspects. It also needed a VG concept in Intellectual Production and in all its items.

These requirements explain the distortion of the normal curve seen in the previous section, which is caused not only by the quality of the courses, but also by the historical fact that the scores were altered without the evaluation being altered. This is an example of what Pena (2003) calls subjacent continuity of the paradigm shift. There was a change in the scoring of concepts from 5 to 7, but the evaluation model remained for scores up to 5 and another evaluation process was created for programs with a score of 5. According to Pena (2013), even if the Copernican revolution remains the best example of a paradigm shift, the Copernican model preserved Platonic presuppositions for astronomical investigation including the ideas that all celestial movements are circular and that the speed of celestial bodies does not change. We can then recognize continuity even in a process considered revolutionary. It took Kepler’s Laws for us to overcome the idea that the circle, and not variation, was the perfect expression of cosmic order.

In the Post-Graduation world, we have already experienced a Copernican revolution by changing our priority from teaching to research. However, we must contribute as Kepler did, in a way that allows us to structure a new model of evaluation that incorporates criteria such as international insertion, leadership, and solidarity to the aspects of Faculty Members, Professors’ Production, Students’ Production, and Social Insertion. The criteria presented here for a program to be given a score of 6 or 7 can be incorporated in these aspects.

The second proposition this article contributes is a reflection on the meaning of PFN. It is a body of professors considered stable. As a body, it gives identity to a set of systems coordinated by a thinking head. A body of professors must not have only people with similar profiles, just as the human body must not have members or organs with the same functions. What gives a body organicity is the ability to integrate different organs and members. In a body, each cell matters. Disproportional growth of similar cells is like a cancer that prevents the exercise of diversity.

Considering this metaphor, it is important to conceive in the evaluation process of a PGP the importance of respecting the diversity of professors. Some professors are better suited to teaching, other to research internationalization, and others for managerial activities. What is best for a PGP: to have a senior professor that does not teach Grad students, but has high intellectual productivity and great international insertion, or not having them if the first demand is universal? Is it best for the PGP to have or not to have a professor with teaching skills, the ability to write textbooks, and expertise to manage the program, but who does not publish their articles in well-known international journals?

This reflection highlights the need for considering the distribution of items evaluated among the professors of the PFN. The program must be considered from its group of professors as a body, as a team, not as a group of individuals with a specific profile as researchers. We add our voices to that of Nascimento (2010), who points out the risk of the system leading researchers to participate only in activities that add points to their Capes’ evaluation, directing the programs and courses towards what is being called academic productivism.

REFERÊNCIAS

• ALCADIPANI, R. Resistir ao produtivismo: uma ode à perturbação acadêmica. Cadernos Ebape.BR, v. 9, n. 4, p. 1174-1178, 2011.
• BIANCHETTI, L.; MACHADO, A. Trabalho professor no stricto sensu: publicar ou perecer? In: FIDALGO, F.; OLIVEIRA, M. A.; FIDALGO, N. (Orgs.). A intensificação do trabalho professor: tecnologias e produtividade. Campinas: Papirus, 2009. p. 49-89.
• BIANCHETTI, L.; SGUISSARDI, V. Desafios e dilemas da gestão e da avaliação da pós-graduação. Campinas: Autores Associados, 2009.
• BLOOM, B.; HASTINGS, J.; MADAUS, G. Handbook of formative and summative evaluation of student learning. New York: McGraw-Hill, 1971.
• BRASIL. Decreto n. 86.816, de 5 de janeiro de 1982. Dispõe sobre a Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Capes), e dá outras providências. 1982. Disponível em: <Disponível em: http://www2.camara.leg.br/legin/fed/decret/1980-1987/decreto-86816-5-janeiro-1982-436737-publicacaooriginal-1-pe.html >. Acesso em: 30 abr. 2018.
  » http://www2.camara.leg.br/legin/fed/decret/1980-1987/decreto-86816-5-janeiro-1982-436737-publicacaooriginal-1-pe.html
• BRASIL. Medida Provisória n. 150, de 15 de março de 1990 convertida na Lei n. 8.028 de 1990. Dispõe sobre a organização da Presidência da República e dos Ministérios, e dá outras providências. 1990a. Disponível em:<Disponível em:http://www.planalto.gov.br/ccivil_03/mpv/1990-1995/150.htm >. Acesso em: 30 abr. 2018.
  » http://www.planalto.gov.br/ccivil_03/mpv/1990-1995/150.htm
• BRASIL. Lei n. 8.028, de 12 de abril de 1990. Dispõe sobre a organização da Presidência da República e dos Ministérios, e dá outras providências. 1990b. Disponível em: <Disponível em: http://www.planalto.gov.br/ccivil_03/leis/l8028.htm >. Acesso em: 30 abr. 2018.
  » http://www.planalto.gov.br/ccivil_03/leis/l8028.htm
• BRASIL. Lei n. 8.405, de 9 de janeiro de 1992. Autoriza o Poder Executivo a instituir como fundação pública a Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Capes) e dá outras providências. 1992. Disponível em:<Disponível em:http://www.planalto.gov.br/ccivil_03/leis/l8405.htm >. Acesso em: 30 abr. 2018.
  » http://www.planalto.gov.br/ccivil_03/leis/l8405.htm
• BRASIL. Lei n. 11.502, de 11 de julho de 2007. Modifica as competências e a estrutura organizacional da fundação Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES, de que trata a Lei n. 8.405, de 9 de janeiro de 1992; e altera as Leis n. 8.405, de 9 de janeiro de 1992, e 11.273, de 6 de fevereiro de 2006, que autoriza a concessão de bolsas de estudo e de pesquisa a participantes de programas de formação inicial e continuada de professores para a educação básica. 2007. Disponível em: <Disponível em: http://www.planalto.gov.br/ccivil_03/_ato2007-2010/2007/lei/l11502.htm> Acesso em: 30 abr. 2018.
  » http://www.planalto.gov.br/ccivil_03/_ato2007-2010/2007/lei/l11502.htm>
• BRASIL. Decreto n. 6.755, de 29 de janeiro de 2009. Institui a Política Nacional de Formação de Profissionais do Magistério da Educação Básica, disciplina a atuação da Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES no fomento a programas de formação inicial e continuada, e dá outras providências. 2009. Disponível em:<Disponível em:https://www.capes.gov.br/images/stories/download/legislacao/Decreto-6755-2009.pdf >. Acesso em: 30 abr. 2018.
  » https://www.capes.gov.br/images/stories/download/legislacao/Decreto-6755-2009.pdf
• COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR (CAPES). Plano Nacional de Pós-Graduação, 1982. 1982. Disponível em:<Disponível em:http://www.capes.gov.br/images/stories/download/editais/II_PNPG.pdf >. Acesso em:15 fev. 2016.
  » http://www.capes.gov.br/images/stories/download/editais/II_PNPG.pdf
• COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR (CAPES). Plano Nacional de Pós-Graduação, 1985. 1985. Disponível em: <Disponível em: http://www.capes.gov.br/images/stories/download/editais/III_PNPG.pdf >. Acesso em: 15 fev. 2016.
  » http://www.capes.gov.br/images/stories/download/editais/III_PNPG.pdf
• COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR (CAPES).Reformulação do sistema de avaliação da pós-graduação: o modelo a ser implantado na avaliação de 1998. Brasília, DF: Capes, 1998.
• COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR (CAPES). Plano Nacional de Pós-Graduação, 2004. 2004. Disponível em:<Disponível em:http://www.Capes.gov.br/avaliacao/avaliacao-da-pos-graduacao >. Acesso em: 15 fev. 2016.
  » http://www.Capes.gov.br/avaliacao/avaliacao-da-pos-graduacao
• COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR (CAPES). Relatório de avaliação 2007-2009. 2006. Disponível em:<Disponível em:http://trienal.Capes.gov.br/?page_id=100 >. Acesso em: 15 fev. 2016.
  » http://trienal.Capes.gov.br/?page_id=100
• COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR (CAPES). Plano Nacional de Pós-Graduação (PNPG): 2011-2020. Brasília, DF: Capes, 2010. 1 v. Disponível em:<Disponível em:http://www.capes.gov.br/images/stories/download/Livros-PNPG-Volume-I-Mont.pdf >. Acesso em: 15 fev. 2016.
  » http://www.capes.gov.br/images/stories/download/Livros-PNPG-Volume-I-Mont.pdf
• COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR (CAPES). História e missão. 2012. Disponível em: <Disponível em: http://www.Capes.gov.br/sobre-a-Capes/historia-e-missao >. Acesso em: 15 fev. 2016.
  » http://www.Capes.gov.br/sobre-a-Capes/historia-e-missao
• COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR (CAPES). Documento de Área 2016. 2017. Disponível em:<Disponível em:http://capes.gov.br/images/documentos/Documentos_de_area_2017/27_ADMI_doc_area_2016_final_20jan2017.pdf >. Acesso em: 15 fev. 2016.
  » http://capes.gov.br/images/documentos/Documentos_de_area_2017/27_ADMI_doc_area_2016_final_20jan2017.pdf
• CASTRO, C. M.; SOARES, G. A.D. Avaliando as avaliações da Capes. Revista de Administração de Empresas, v. 23, n. 3, p. 63-73, 1983.
• CURY, C. R. J. Quadragésimo ano do parecer CFE n. 977⁄65. Revista Brasileira de Educação, v. 30, p. 7-30, 2005.
• DANTAS, D. C. Dimensões da pós-graduação e o modelo Capes de avaliação: uma discussão sobre critérios e sua mensuração. In: ENCONTRO DA ASSOCIAÇÃO NACIONAL DE PÓS-GRADUAÇÃO E PESQUISA EM ADMINISTRAÇÃO, 36., 2012, Rio de Janeiro. Anais...Rio de Janeiro: Anpad, 2012.
• FONSECA, C. Avaliação dos programas de pós-graduação: do ponto de vista de um nativo. Horizontes Antropológicos, v. 7, n. 16, p. 261-275, 2001.
• FREITAS, M. Avaliação da produção científica: considerações sobre alguns critérios. Psicologia Escolar e Educacional, v. 2, n. 3, p. 211-228, 1998.
• HORTA, J. S. B.; MORAES, M. C. M. O sistema Capes de avaliação da pós-graduação: da área de educação à grande área de ciências humanas. Revista Brasileira de Educação, v. 30, p. 95-181, 2005.
• KUENZER, A. Z.; MORAES, M. C. M. Temas e tramas na pós-graduação em educação. Educação e Sociedade, v. 93, n. 26, p. 1341-1362, 2005.
• LEITE, M. F. B.; VIANA, A. B. N.; PEREIRA, G. G. Métodos quantitativos na avaliação da CAPES: uma pesquisa bibliográfica. Facef Pesquisa, v. 9, n. 2, p. 166-174, 2006.
• LUIZ, R. R. Avaliação de produtividade acadêmica: uma proposta de quantificação. Revista Brasileira de Pós-Graduação, v. 3, n. 6, p. 300-312, 2006.
• MACCARI, E. A.; LIMA, M. C.; RICCIO, E. L. Uso do sistema de avaliação da Capes por programas de pós-graduação em administração no Brasil. Revista de Ciências da Administração, v. 11, n. 25, p. 68-96, 2009.
• MACCARI, E. A. et al. Sistema de avaliação da pós-graduação da Capes: pesquisa-ação em um programa de pós-graduação em Administração. Revista Brasileira de Pós-Graduação, v. 9, n. 5, p. 171-205, 2008.
• MACEDO, E.; SOUZA, C. P. A pesquisa em educação no Brasil. Revista Brasileira de Educação, v. 43, n. 15, p. 166-176, 2010.
• MACHADO, A. M. N.; BIANCHETTI, L. (Des)fetichização do produtivismo acadêmico: desafios para o trabalhador-pesquisador. Revista de Administração de Empresas, v. 51, n. 3, p. 244-254, 2011.
• MASCARENHAS, A.; ZAMBALDI, F.; MORAES, E. Rigor, relevância e desafios da academia em administração: tensões entre pesquisa e formação profissional. Revista de Administração de Empresas, v. 51, n. 3, p. 265-279, 2011.
• MATTOS, P. Nós e os índices: a propósito da pressão institucional por publicação. Revista de Administração de Empresas, v. 48, p. 144-149, 2008.
• MATTOS, P. Pés de barro do texto “produtivista” na academia. Revista de Administração de Empr esas, v. 52, n. 5, p. 566-573, 2012.
• MOREIRA, A. F. A A cultura da performatividade e a avaliação da pós-graduação em educação no Brasil. Educação em Revista, v. 25, n. 3, p. 23-42, 2009.
• NASCIMENTO, L. F. Modelo Capes de avaliação: quais as consequências para o triênio 2010-2012?. Administração: Ensino e Pesquisa, v. 4, n. 11, p. 579-600, 2010.
• PENA, R. P. M. A ciência como jogo. Belo Horizonte: Com Arte, 2003.
• RODRIGUES, L. O. C. Publicar mais, ou melhor? O Tamanduá Olímpico. Revista Brasileira de Ciência e Esporte, v. 29, n. 1, p. 35-48, 2007.
• SCRIVEN, M. The metodology of evaluation. In: TYLER, R.; GAGNÉ, R. M. Y; SCRIVEN, M. (Ed.). Perspectives of curriculum evaluation. Chicago: Rand Mac Nally, 1967. p. 39-83.
• SGUISSARDI, V. A avaliação defensiva no “modelo Capes de avaliação”: é possível conciliar avaliação educativa com processos de regulação e controle do Estado?. Perspectiva, v. 24, n. 1, p. 49-88, 2006.
• SHIGAKI, H. B.; PATRUS, R. Avaliação de Artigos Científicos em Administração: Critérios e Modelos de Avaliadores Experientes. Teoria e Prática em Administração, v. 6, p. 107-135, 2016.
• SOUSA, E. F. O discurso da Capes para a avaliação dos programas de pós-graduação: da (des)fragmentação à comunicação em rede. 2008. 306 f. Tese (Doutorado em Linguística) - Universidade de Brasília, Brasília.
• SOUZA, S. Z.; BIANCHETTI, L. Pós-graduação e pesquisa em educação no Brasil: o protagonismo da ANPEd. Revista Brasileira de Educação, v. 36, n. 12, p. 389-409, 2007.
• SPAGNOLO, F.; SOUZA, V. C. Modelo Capes de avaliação. Revista Brasileira de Pós-Graduação, v. 2, p. 8-33, 2004.
• STRATHERN, M. “Improving ratings”: audit in the British University System. European Review, v. 3, n. 5, p. 305-321, 1997.
• TREIN, E.; RODRIGUES, J. O mal-estar na academia: produtivismo científico, o fetichismo do conhecimento mercadoria. Revista Brasileira de Educação, v. 16, n. 48, p. 769-819, 2010.
• VIANA, A. B. N.; MANTOVANI, D. M. N.; VIEIRA, A. R. Análise dos programas de pós-graduação avaliados pela Capes: relação entre conceitos dos programas e índice de publicação. In: ENCONTRO DA ASSOCIAÇÃO NACIONAL DE PÓS-GRADUAÇÃO E PESQUISA EM ADMINISTRAÇÃO, 32., 2008, Rio de Janeiro. Anais... Rio de Janeiro: Anpad , 2008.

• 8
  The authors would like to thank Capes and Fapemig for the financial support.
• {Translated version} Note: All quotes in English translated by this article’s translator.

Publication Dates

• Publication in this collection
  Oct-Dec 2018
• Date of issue
  Dec 2018

History

• Received
  01 Mar 2017
• Accepted
  08 May 2018