Quality assessment of scientific papers:Excellence or legitimization of research practices?

ABSTRACT

This study aims to analyze the relative efficiency of graduate programs in terms of the quality of their scientific publications in journals. To this end, we applied Data Envelopment Analysis to the fields of Public and Business Administration, Accounting, and Tourism, using data from the 2017 Quadrennial Evaluation developed by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [“Coordination for the Improvement of Higher Education Personnel”, CAPES]. The results suggest that efficient graduate programs are concentrated in the center-south axis of Brazil, in the areas of Administration and Accounting. Despite the overall publishing strategy in journals of various ranks (strata), efficient graduate programs allocate their efficiency primarily to Qualis A1 journals, A1 being the excellence rating in the CAPES scale. We conclude that legitimized research practices pressure graduate programs to improve the quality of their publications while threatening the survival of programs that fail to conform to the assessment rules.

KEYWORDS
Institutional isomorphism; Efficiency; Scientific publications; Legitimacy

RESUMO

O objetivo desta pesquisa foi analisar a eficiência relativa dos Programas de Pós-Graduação [PPGs] em termos de qualidade de suas publicações em periódicos. O método utilizado foi o Data Envelopment Analysis, aplicado à área de concentração de Administração Pública e de Empresas, Ciências Contábeis e Turismo, a partir da Avaliação Quadrienal de 2017, desenvolvida pela Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [CAPES]. Os resultados sugerem que os PPGs eficientes estão centralizados no eixo Centro-Sul do país, nas áreas de Administração e Contabilidade. Apesar de a estratégia difundida de publicar em todos os estratos, os PPGs eficientes possuem concentração preponderante de sua eficiência em publicações de Qualis A1, consideradas como as de maior excelência pela CAPES. Conclui-se, portanto, que as práticas de pesquisa legitimadas pressionam os PPGs por uma melhoria na qualidade de suas publicações, ao mesmo tempo em que ameaçam a existência daqueles que não conseguirem se adequar às regras de avaliação.

PALAVRAS-CHAVE
Isomorfismo institucional; Eficiência; Publicações científicas; Legitimidade

1. INTRODUCTION

In the past decades, the university system has been reconfigured to achieve more efficient results in response to uncertainties and external pressures (Jelonek & Mazur, 2020). That process has led Higher Education Institutions [HEIs] to seek legitimacy by replicating research, teaching, and extension practices adopted by their most prestigious peers (Meyer & Rowan, 1977). These practices have become a benchmark for legitimizing and maintaining quality standards for other HEIs and government agencies, rankings, and international accreditations (Xu, Rose & Oancea, 2019). In this context, the publication of scientific papers in high-impact journals, many of which are international, has become recognized as a good research practice (Barrett, Fernandez & Gonzalez, 2020). HEIs perceive these publications as of more significant impact when they have a higher number of citations, which is the indicator adopted to assess their quality (Xu et al., 2019).

In Brazil, the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [Brazilian Federal Agency for Support and Evaluation of Graduate Education - CAPES] is the body responsible for ensuring the quality of graduate programs [GPs] based on the assessment of the National Graduate System [SNPG]. Over the years, CAPES has been appealing for more outstanding excellence in publications through the inclusion of the criterion Publicações Qualificadas do Programa de docente permanente [Qualified publications by permanent professor program] in the 2013-2016 Quadrennial Evaluation (Andrade, Oliveira, Maccari & Hollnagel, 2018; CAPES, 2017b) and, more recently, by showing signs that it may come to include international bibliometric parameters to measure and evaluate the quality of the GPs’ publications (CAPES, 2020a).

Therefore, the appreciation of this type of indicator may represent a quest to legitimize publications by national GPs, based on their degree of compliance with international research practices and standards (Kezar & Bernstein-Sierra, 2019; Vakkayil & Chatterjee, 2017). Along these lines, the CAPES assessment system would imply greater homogeneity among GPs, in the scope of the models aimed at publishing papers in high-impact journals, as indicated by the concept of institutional isomorphism (Meyer & Rowan, 1977; DiMaggio & Powell, 1983). On the other hand, Lozano, Bofarull, Waddock, and Prati-Pubill (2018) point out that although institutional isomorphism suggests a tendency towards homogeneity, collective, and even individual characteristics which could compromise this process have nonetheless persisted. In this context, potential differences in financial and human resources available for a given GP could reflect on the inequality of its results, whereas this, in turn, can lead to substantial gaps in the assessment of its quality (Croucher & Woelert, 2016; Thornton, William & Lounsbury, 2012; Woelert & Croucher, 2018).

CAPES assessment is coercive insofar as it links the survival of GPs to their assessment results (Andrade et al., 2018; Patrus, Shigaki & Dantas, 2018), and normative insofar as it establishes practices regarded as legitimate by members of the academic community working with CAPES. Therefore, these mechanisms create a sort of “iron cage” (DiMaggio & Powell, 1983). For a while, they homogenize the research practices adopted by all GPs, inhibiting or even discouraging initiatives that are not compromised by the CAPES evaluation criteria. Therefore, a decisive influence on the behavior of GP professors and students emerges, aiming to fulfill the required research practices, as well as on the consequences of not complying with them.

The research question that guides this study is: to what extent does the distribution of the quality of scientific papers relate to the efficiency of GPs? To answer it, we proceeded to analyze the relative efficiency of GPs in terms of the quality of their publications in high impact journals (Qualis ratings A1, A2, and B1) and those with lower impact (B2 to B5). GPs in public and business administration, accounting, and tourism were analyzed based on the information available on the 2017 Quadrennial Assessment prepared by CAPES. The relative efficiency was estimated through Data Envelopment Analysis [DEA], while the legitimacy and excellence of research practices were based on institutional isomorphism.

2. THEORETICAL FRAMEWORK

2.1. Institutional isomorphism - excellence and legitimacy

The management of HEIs is influenced by institutional “rules”, “roadmaps”, or “incentives” that lead them towards a greater degree of compliance with the practices adopted by their prestigious peers, regarded as models of excellence. This phenomenon is known as institutional isomorphism (Czarniawska, 2009; DiMaggio & Powell, 1983; Meyer & Rowan, 1977; Thornton et al., 2012). Prestige is understood as the achievement of excellence in established practices, whereas legitimacy consists of conforming to those same practices (Kezar & Bernstein-Sierra, 2019; Vakkayil & Chatterjee, 2017).

This phenomenon creates pressures that influence the assimilation of new practices to be pursued by organizations, which can, in turn, change their values and behaviors (Kezar & Bernstein-Sierra, 2019; Woelert & Croucher, 2018). In this sense, legitimacy and prestige engender the perpetuation of practices and values that gradually become the new “rules of the game”, changing the beliefs and actions of all actors in a given institutional environment (Jelonek & Mazur, 2020). As Kezar and Bernstein-Sierra (2019, p.3) describe, “where many organizations in a field look to a prestigious professional association for guidance, they are likely to adopt the practices embraced by that association. Over time, the practices are legitimated within the field and they become the new norm.”

The institutional pressures join forces to homogenize and converge the institutions to the same conformity environment, based on three mechanisms: coercive, mimetic, and normative (Czarniawska, 2009; DiMaggio & Powell, 1983; Meyer & Rowan, 1977; Thornton et al. 2012). The coercive mechanism is imposed according to legal and regulatory pressures from governmental, social, or cultural institutions. The mimetic mechanism emerges from the replication of strategies adopted by prestigious organizations by their peers, aiming to adapt to environments that contain a high degree of uncertainty and ambiguity. In turn, the normative mechanism operates by adopting professional practices regarded as legitimate by their peers, such as the definition of essential criteria and standards for evaluating the research results in a given field of knowledge. These mechanisms act in an integrated fashion, explaining why organizations adopt practices or strategies with a high degree of similarity (Barrett et al., 2020).

As a result, these pressures lay the foundations for the so-called institutional “iron cage”, which limits the actors’ behaviors and strategies to the scope of incentives to which they are subjected (DiMaggio & Powell, 1983; Meyer & Rowan, 1977; Thornton et al., 2012). On the other hand, this interpretation does not imply that institutional isomorphism should be perceived as deterministic. On the contrary, it is characterized as a dynamic and non-deterministic process resulting in the complete and homogeneous non-convergence of institutions (Woelert & Croucher, 2018).

2.2. Quality assessment of graduate programs by CAPES: the rules of the game

In Brazil, CAPES assesses the quality of GPs through criteria that aim to ensure the programs’ renewal for the subsequent term (CAPES, 2020b). Therefore, quality indicators put institutional pressures on researchers - and consequently the GPs to which they are accredited - so that they come to legitimize themselves according to the “rules of the game”. Those rules can influence their research behaviors and practices (Jelonek & Mazur, 2020), mainly regarding the publication of scientific articles, as these are one of the key results (although not the only one) to be aspired to by GPs according to the current model (Patrus et al., 2018).

The assessment process conducted by CAPES strongly influences the homogeneity of research practices based on two mechanisms, normative and coercive. The first establishes that research practices considered “legitimate” should be disseminated among peers. These rules are determined by scholars and researchers who collaborate with the CAPES board during the assessment process by sharing their beliefs on good research practices. The second operates through the assessment results, which define the support policies and, ultimately, the GPs’ very survival (CAPES, 2020b; Patrus et al., 2018).

CAPES continually seeks to improve its assessment mechanisms and plays a prominent role in the development of GPs in Brazil (Maccari, Rodrigues, Alessio & Quoniam, 2008). However, the current model receives some criticism (Andrade et al., 2018), including the proliferation of “academic productivism” practices (Alcadipani, 2011a, 2011b) stemming from the intellectual production assessment norms in force. In this context, Rego (2014, p. 341) argues that “the scenario is sad: we have never published so much, and we have never had so many journals. But does that mean that we are conducting more and better research?”.

An objective effort undertaken by CAPES (2017b) to overcome such criticism is the incentive to high-quality publications, that is, the publication of articles in journals with a higher Qualis grade, with particular attention to A1 journals, due to their internationalization. In the 2013-2016 quadrennium, a new criterion was incorporated, called “Qualified publications by permanent professor”, in which the three best productions of each permanent professor in line with the program proposal are considered. CAPES (2017b) considers permanent professors who meet some requirements of the GP, such as teaching activities, participation in research projects, supervision of master’s or doctoral students, and management activities. On the other hand, collaborator professors include other members of the PG’s faculty who do not meet those requirements, including postdoctoral fellows. Besides, the number of publications in B4 and B5 journals (CAPES, 2017b), considered to be of lower impact, was limited.

CAPES has shown signs that it may soon adopt international bibliometric criteria to qualify the journals that published the articles developed by the GPs (CAPES, 2020a), which would give greater weight to high-impact international publications. This indication may have consequences for national GPs, for the creation of criteria to assess the quality of publications does not imply that all GPs will automatically adjust or do so homogeneously, as recommended by institutional isomorphism. Indeed, GPs may come to experience this new pressure based on the profile of their researchers (Imasato, Perlin & Borenstein, 2017), in addition to their peculiarities and limitations (Thornton et al., 2012; Woelert & Croucher, 2018). The GPs that have previously adopted this new rule of the game are likely to stand out or achieve prominence. Therefore, it is essential to estimate how GPs comparatively achieve efficiency in academic production, which, in turn, results from the publication of articles in journals of higher and lower strata, based on their respective human and material resources.

3. RESEARCH METHODS

3.1. Efficiency analysis using DEA models

The quality of HEIs has been the subject of empirical research since the mid-1980s (Geva-May, 2001). Since then, these studies have gained greater relevance and their debate has intensified among politicians, faculty members, and other education stakeholders (Witte & López-Torres, 2017). Therefore, in addition to its relevance, the current context is one of continuous and significant reductions in government investments and subsidies, which forces HEIs, on the one hand, to maintain their level of academic excellence and, on the other hand, to use their resources efficiently (Monfared & Safi, 2013). In the literature, studies involving quality and efficiency in education have been developed mainly through Data Envelopment Analysis [DEA] (Liu, Lu, Li & Lin, 2013).

The application of DEA models is helpful in the absence of market prices and when various inputs and outputs are used to calculate the efficiency of the production process of a decision-making unit [DMU], which can be represented by universities, graduate programs (Joumady & Ris, 2005), or many others. For instance, in the case of HEIs, it is known that the selection and, primarily, the measurement of inputs and outputs in the production function of education has been a topic of heated debate (Agasisti & Pérez-Esparells, 2010). The use of DEA has advantages, as it allows each DMU to have its input and output vectors weighted so that its efficiency is maximized and can be compared within the set of DMU analyzed at the same time (Johnes & Johnes, 1993).

This study relied on DEA to fulfill the objective of analyzing the relative efficiency of GPs in the fields of public and business administration, accounting, and tourism, from 2013 to 2016, based on data collected by the 2017 Quadrennial Assessment developed by CAPES. It is important to note that CAPES Quadrennial Evaluation aims to assess the quality of graduate programs. That is, it does not focus on broader aspects of HEIs. Therefore, this study defines GPs in the areas mentioned earlier of knowledge as units of analysis, such as the DMUs in the DEA model.

The model follows the assumption of output-oriented Constant Returns to Scale [CRS]. We chose CRS by comparing the distribution of CRS efficiency scores under CRS models with the scores of Variable Returns to Scale [VRS] models through three statistical tests, by assuming that (i) the distribution of errors is exponential, and (ii) the error distribution is semi-normal, and by applying (iii) the Kruskal-Wallis non-parametric test (Bogetoft & Otto, 2011).

Virtual outputs were also calculated using the multiplier DEA model. For further details on these calculations, see Cook and Zhu (2008, p. 69-89). The analysis of virtual outputs allows identifying the source of the efficiency score of each DMU in terms of their outputs. For instance, if a given data set has three outputs, the sum of all virtual outputs must be equal to one (or 100%). This method demonstrates whether the efficiency is concentrated in a single output or distributed in different output combinations. Given that the efficiency of GPs can have several optimal solutions, we used the method proposed by Thanassoulis (2001, p. 101) to analyze the robustness of the results. After determining the output-oriented DEA model with CRS, we must define the DMUs, the sample, the inputs, and the selected outputs.

3.2. Sample selection

The data were extracted from the CAPES website and refer to the 2017 Quadrennial Assessment, conducted between 2013 and 2016 (CAPES, 2017a), and to the public and business administration, accounting, and tourism areas of concentration. As a criterion for selecting DMUs, data were collected from the GPs that offered doctoral courses in that area of knowledge and were evaluated throughout the four-year coverage. At the end of the process, the sample reached 59 GPs assessed by CAPES (2017b).

3.3. Determining the model inputs and outputs

The number of permanent and collaborator professors, productivity scholarship researchers, doctoral students, master’s students, and financed projects were considered inputs to select the variables. The outputs were the publications of articles, subdivided into three categories of Qualis ratings, or “strata”: A1 (weighted); A2 and B1 (weighted and added); B2 to B5 (weighted and added). The number of variables in the model is adjusted to the number of DMUs analyzed, as recommended by Dyson et al. (2001). Figure 1 lists the model inputs and outputs.

The choice for the number of professors aimed to bring together the permanent and collaborator professors at the disposal of the GPs, highlighting those who have been granted productivity scholarships for their superior academic production performance, recognized by CAPES. In parallel are the students’ contributions, measured by the number of master’s and doctoral students enrolled in the GPs surveyed, as academic publications are typically developed in collaboration between professors and students. The number of projects financed were included due to limitations in finding information on their monetary value and other financial variables in the database. This variable identifies the existence of economic resources available for conducting research.

Figure 1.
List of model inputs and outputs

The selection of article publications as outputs was motivated by the assumption that these are crucial results to be achieved by GPs, although not the only ones (Patrus et al., 2018). The discrimination of outputs in different ratings was based on CAPES assessment rules (2017b), as one of the criteria to attest to the quality of scientific production by the faculty and student bodies is the number of papers published in high-impact journals, which, in this case, are the highest-ranked ones on the CAPES scale. CAPES (2017b) defines the Qualis system ranks journals according to a scale, in which A1 and B1 ratings are regarded as of higher impact, whereas the ratings B2 to B5 are regarded as inferior. According to these criteria, the scores attributed to papers were the ratings A1, A2, B1, B2, B3, B4, and B5, weighted by the scores 100, 80, 60, 50, 30, 20, and 10, respectively (CAPES, 2017b).

Therefore, the division into higher and lower strata highlights the scientific productions published in journals with the highest academic excellence. In this sense, separate publications classified as A1 derive from the more significant impact they have on the academic community. They are represented by international journals in the scope of the researched area, despite strata A2 and B1, which consist of both national and international journals. Therefore, as this research addresses the quality (according to the Qualis scale) of the journals where the articles are published, we opted to exclude publications in Qualis C journals, technical productions, and books.

As the database is a balanced panel with four observations in each GP, one for each year of the 2013-2016 quadrennium, the data was consolidated to analyze the period by calculating the mean value of inputs over four years for the following inputs: the number of faculty members (permanent and collaborator professors, and productivity scholarship researchers), master’s students, doctoral students, and financed projects. We chose to rely on the average of such variables because the data proved to be stable or not cumulative. As the number of articles published in journals (ranked as A1, A2, B1, B2, B3, B4, and B5, weighted by their respective scores) is characterized as cumulative variables, the output values for the period for each one of the three stratum categories were added.

In summary, the outputs were divided into three stratum categories or groups, namely (i) A1, (ii) A2 and B1, and (iii) B2 to B5. The output-oriented CRS model was adopted under the assumption that the academic production of GPs in terms of quality, quantity, or some combination between both will be maximized if the input levels remain constant. A similar procedure was also adopted in Brazil by Silva, Corrêa and Gomes (2016); Costa, Ramos, Souza, Sampaio and Barbosa (2015); Costa, Souza, Ramos and Silva (2012), and Moreira, Cunha, Ferreira and Silveira (2011). DEA with virtual outputs aims to complement the developed analyses insofar as it segregates the optimal solution according to the different output combinations.

4. RESULTS

To analyze sample data distribution, Table 1 shows the descriptive statistics. More specifically, it shows greater representativeness of GPs in administration, not only in absolute terms but in all inputs. In total, there were 45 programs in administration (76% of the sample), followed by accounting, with 10 GPs (17%), and tourism, with 4 GPs (7%).

Table 1.
Data Overview

Descriptive Statistics	Productivity scholarship researchers	Professors	Master’s students	Doctoral students	Projects financed	Publications
						A1	A2	B1	B2	B3	B4	B5
Administration (45 GPs)
Mean	4	17	42	41	26	13	46	56	58	50	43	9
Minimum value	0	11	0	2	1	0	9	18	19	12	8	0
Maximum value	18	48	93	154	64	63	168	172	228	179	230	35
Standard deviation	4	7	19	32	15	17	35	36	41	37	36	9
Accounting (10 GPs)
Mean	3	14	26	18	15	5	60	58	50	50	50	14
Minimum value	1	7	15	2	0	0	15	12	12	16	8	2
Maximum value	7	20	34	70	42	16	148	156	142	121	132	39
Standard deviation	2	4	6	20	11	6	35	45	42	34	40	13
Tourism (4 GPs)
Mean	2	14	33	8	14	1	15	21	30	36	26	11
Minimum value	0	12	25	2	6	0	9	14	20	26	21	9
Maximum value	3	18	38	19	19	1	18	30	47	45	31	12
Standard deviation	1	3	6	8	6	1	4	7	12	10	5	1
Complete sample (59 GPs)
Mean	4	17	39	35	23	11	46	54	55	49	43	10
Minimum value	0	7	0	2	0	0	9	12	12	12	8	0
Maximum value	18	48	93	154	64	63	168	172	228	179	230	39
Standard deviation	4	6	18	31	15	15	35	37	40	35	36	10

• Source: Research data.

As to the outputs, the analysis revealed greater representativeness of GPs in administration and accounting in high-impact publications. For instance, GPs in administration showed higher mean and maximum values in A1 journals, while GPs in accounting showed a higher mean value in A2 journals. Regarding the articles published in journals of lower strata, both courses showed similar mean, minimum, maximum, and standard deviation values. Finally, it appears that the number of articles published by GPs in Tourism is lower compared to the other areas analyzed, except for publications in B5 journals, which showed similar levels for all courses.

4.1. Analysis of the relative efficiency of graduate programs

The estimation of the output-oriented DEA model with CRS included analyzing five inputs and three outputs (see Figure 1) for the 59 GPs analyzed. The relative efficiency of GPs can be interpreted based on the comparison between how human and financial resources (inputs) result into publications (outputs). We expect that the most efficient GPs will be those that can produce the most publications using fewer resources (relative efficiency). A GP will be 100% efficient if, and only if, none of its inputs or outputs can be improved (comparatively with other GPs) without worsening any other inputs or outputs.

To understand the results presented in Table 2, we must refer to two fundamental concepts of DEA models: (i) benchmark and (ii) source of efficiency. First, we infer that inefficient GPs use GPs located on the efficient frontier formed by DEA as their efficiency benchmarks. This domination relationship (in mathematical terms) of inefficient GPs by efficient GPs considers the proportion of the combination between outputs and inputs (see illustrative example in ^{Appendix 1}). In other words, efficient GPs used the best combination of the proportion between inputs and outputs in the sample analyzed, thus serving as a benchmark for the inefficient GPs closest to them. Second, the source of efficiency is calculated from the virtual outputs. This procedure allows us to identify how much each output stratum contributed to the final GP efficiency score and serve as a benchmark for the quality of scientific publications. This approach allows identifying the proportion of each stratum in calculating the efficiency of GPs so that the outputs of efficient GPs can be compared against inefficient ones.

Table 2
GP efficiency by region, area, and legal status

Criterion	GPs #	Average Efficiency	Efficient GPs #	Benchmark for other GPs #	Source of efficiency (mean)
					A1	A2 and B1	B2 to B5
Geographic Region
Central-West	3	99%	2	8	16%	65%	18%
Northeast	9	61%	0	0	11%	46%	44%
North	1	93%	0	0	0%	7%	93%
Southeast	27	83%	15	108	33%	44%	23%
South	19	75%	7	60	23%	44%	33%
Discipline
Administration	45	74%	15	85	30%	41%	29%
Accounting	10	95%	8	88	10%	71%	19%
Tourism	4	78%	1	3	6%	20%	73%
Legal status
State	7	66%	2	23	15%	57%	28%
Federal	23	80%	10	67	19%	56%	25%
Municipal	2	100%	2	29	25%	38%	38%
Private	27	77%	10	57	29%	44%	27%

• Source: Research data.

Table 2 shows that the average efficiency of GPs is highest in the Central-West region of Brazil, followed by the North, Southeast, South, and Northeast regions. It is noteworthy that the first two regions are represented with fewer DMUs compared to the rest, with three and one GPs, respectively. The Southeast region has greater representativeness in the absolute number of efficient GPs and the number of times that these programs serve as benchmarks for other GPs, followed by the South and Central-West regions. Therefore, the analysis of the GPs efficiency by geographic region reveals that efficient programs are concentrated in Brazil’s center-south axis, whereas other regions did not present efficient GPs.

Table 2 also reveals that this concentration of efficient GPs in some regions is also present when high-impact publications are examined, that is, those published in journals of a higher rating. The Southeast region has the highest proportion of its efficiency attributed to Qualis A1 (33%), followed by the South (23%), Central-West (16%), and Northeast (11%) regions. In terms of publications in A2 and B1 journals, the efficiency of the Central-West region is more concentrated in this indicator (65%), and the other regions showed similar rates (44% - 46%), except for the North region (7%). Indeed, this region showed a higher concentration of publications in journals of lower strata (93%), followed by the Northeast (44%), South (33%), Southeast (23%), and Central-West (18%) regions.

Regarding efficiency by discipline, GPs in accounting reached higher average efficiency compared to the programs in administration and tourism. Proportionally, 80% of GPs in accounting are efficient, against 33% of administration programs. Even in smaller quantities, GPs in accounting are benchmarks for more GPs than administration programs. The primary source of efficiency of GPs in administration and accounting are in A2 and B1 publications. Programs in accounting reached an average of 71%, compared to 41% in administration. As for the administration programs, 30% of their average efficiency is allocated in the A1 stratum. That is, it is higher than the other areas of knowledge surveyed. The efficiency of tourism programs is concentrated in publications of lower strata (73%), and only a single GP was rated efficient.

According to the legal status of the GPs, we identified that the municipal university programs were the most efficient, followed by private, federal, and state universities, respectively. Federal and private GPs account for the most significant number of efficient GPs. In general, the source of efficiency of the different legal statuses is distributed in a similar way, in which A2 and B1 are the main indicators of the efficiency of these GPs.

4.2. Analysis of the source of efficiency based on the participation of publication ratings

To demonstrate the source of the relative efficiency of GPs, we chose to divide them into efficient and inefficient. Efficient programs showed a result equal to one in the DEA model estimation, and these totaled 24 GPs. In turn, programs with a value less than one were considered inefficient (35 GPs).

According to the Qualis scale, the outputs structuring followed the same division based on publications in journals of higher (A1; A2 and B1) and lower rating (B2 to B5). To this end, two levels of participation were defined for each rating in efficiency: preponderant participation, in which the rating represents at least half (50% or more) of the GP’s efficiency rate; and high participation, in which the rating’s participation was between a quarter and half of the GP’s efficiency rate (equal to or greater than 25% and less than 50%). Therefore, we present data on how many efficient and inefficient GPs have a high or preponderant source of efficiency in each publication rating (see Figure 2 and ^{Appendix 4}). The data were organized in percentage values so that the total count of efficient (24 programs) and inefficient groups (35 programs) could be compared.

In general, the results revealed that the efficiency of efficient GPs derives from a combination of the three rating categories analyzed, that is, publications in journals of higher (A1; A2 and B1) and lower strata (B2 to B5). This indicates that efficient GPs adopt a hybrid strategy by publishing in journals of different strata. As shown in Figure 2, 29% of the efficient GPs have their source of efficiency predominantly allocated in A1 publications, as well 29% in A2 and B1. The preponderance of efficiency concentration in lower Qualis grades (B2 to B5) is present in 42% of all GPs. Also, 4% of the programs have a high-efficiency rate in the A2 and B1 strata, whereas 4% have a high-efficiency rate in the B2 to B5 category.

The primary source of efficiency (63% of all inefficient GPs) is in the higher Qualis stratum (A2 and B1). Journals fitting in this grade of the scale, divided into international and national ones, have less impact compared to A1 journals, even though they are of high quality. Respectively, 9% and 26% of GPs concentrated their efficiency predominantly or highly in A1 publications. As to the lower strata, inefficient GPs showed 23% and 11% of predominant and high concentration, respectively. These results suggest that most of these programs seek to adapt to research practices regarded as excellent, as they allocate part of their efficiency to publishing papers in journals of a higher rating, particularly A2 and B1. However, the results also reveal that the international prominence of these very programs, represented here by publications in A1 journals, is lower than efficient GPs.

Figure 2.
Source of efficiency according to the participation of the GPs virtual outputs

4.3. Analysis of other optimal DEA solutions

Linear programming and, consequently, DEA models allow for the existence of more than a single optimal solution for efficient DMUs. In the case analyzed, this means that efficient GPs can remain efficient with other combinations of virtual outputs. Therefore, we must analyze a model’s virtual outputs that rely on methods to certify other potential optimal solutions. To test the robustness of the results found through the analysis of virtual outputs, the exact implementation of the model adopted in this study was carried out but another linear programming solver was used [LpSolve].

Also, the method suggested by Thanassoulis (2001) was applied to verify the maximum value of virtual outputs to be achieved by efficient GPs. This method assumes that a given GP could have its efficiency distributed by one-third for each rating category if it had a volume of publications large enough for these considerations to confirm their efficiency compared to other programs. Otherwise, the one or two strata will continue to be preponderant in this GP efficiency rate composition, as previously demonstrated. These results are shown in Table 3. In addition to preponderant and high levels of participation, the table also shows the null level, in which the efficiency is completely absent from a certain stratum, and the low-intermediate level, in which the participation of each stratum in the efficiency is greater than zero and less than a quarter (25 %).

Table 3.
Number of efficient GPs and contribution of each publication rating when virtual outputs are maximized

	Publication rating
Contribution level of each publication rating	A1	A2 and B1	B2 to B5
Null (0)	4	0	2
Low - Intermediate (>0% & <=25%)	4	2	6
High (>25% & <= 50%)	13	19	15
Preponderant (>50%)	3	3	1
Total GPs	24	24	24

• Source: Research data.

In general, the results show that neither the implementation of the same model in another solver nor the maximization of virtual outputs changed the results for inefficient GPs. This conclusion was expected, for the optimal combination of weights and publications had already been achieved to rank the GP as the most efficient as possible. After the estimates were performed using LpSolve, the results showed that some optimal solutions differed for efficient GPs, compared to previously obtained results. However, the same trend found in the first solution was maintained. Table 3 presents the results compiled after maximizing the virtual outputs of the model in this study for efficient GPs.

The results show that the most efficient GPs manage to achieve efficiency by maximizing their participation in all three publication ratings. Only four out of the 24 efficient GPs have no (zero) participation in A1, zero in A2 and B1, and two in B2 to B5. Table 3 also shows that most of these programs fit in the high participation range of each rating. This indicates that minimally, all programs that have proven to be efficient attribute their efficiency to higher impact publications while maintaining a high incidence of publications in lower stratum journals. These results are in line with previous interpretations stating that efficient GPs generally adopt a hybrid publishing strategy.

5. DISCUSSION

The results suggest that efficient GPs are more concentrated in the center-south axis of Brazil and in the areas of administration and accounting. Regarding concentration by discipline, graduate programs in accounting have increased in number since 2005. Besides, despite having sparse social networks of scientific production (Nascimento & Beuren, 2011), they can publish their papers in journals of higher strata and serve as benchmarks for more GPs when compared to the programs in administration, whereas GPs in tourism have begun to emerge more recently (Holanda, Widmer & Leal, 2014). On the other hand, not only did GPs in administration reach the highest absolute number, but they also achieved the most significant presence in cooperation networks between programs and professors, thus increasing their productivity (Rossoni & Guarido Filho, 2009).

As we investigate the geographical concentration and the number of GPs, we conclude that these results are generally in line with the assessment developed by CAPES (2017b). This is the same scenario for GPs that have achieved academic excellence, and that stand out for their higher international prominence, among other features (Ramos, 2018). We conclude that regional disparities impact the quantity and quality of results presented by Brazilian GPs.

Regarding relative efficiency analysis, the results showed that GPs publish their articles in journals with higher impact factors. However, the proportion of GPs whose source of efficiency is predominantly allocated in A1 publications is higher among efficient GPs, whereas inefficient GPs tend to emphasize A2 and B1 journals. Therefore, the international prominence of their research was identified as a distinctive aspect of efficient GPs, a fact that had been pointed out in Ramos (2018), regarding excellent GPs.

This prominence is part of a legitimation process (Kezar & Bernstein-Sierra, 2019; Vakkayil & Chatterjee, 2017; Woelert & Croucher, 2018) proposed by the CAPES assessment system through normative and coercive pressures by the National Graduate System, based on the adherence of Brazilian papers to international research practices. Programs are pressed so that the quality of their academic production becomes a priority, and special attention is paid to international publications in journals accepted for Qualis A1 journals.

Therefore, the GPs whose outputs were allocated predominantly in journals of lower strata may have to review their research practices soon, as CAPES has already hinted that the quality assessment of publications may be grounded on international bibliometric criteria from the 2021-2024 Quadrennium onwards (CAPES, 2020a). However, it is doubtful which GPs would publish their papers in Qualis A1 journals.

As discussed in DiMaggio and Powell (1983), institutional homogenization may not occur or may occur only partially. This would happen because the GPs have different characteristics and count on different resources, such as the funds available for research, intra and inter-university cooperation networks, and international partnerships with professors and HEIs, among others. In turn, this discrepancy culminates in the regional and efficiency concentrations found by this study.

Lozano et al. (2018) argue that isomorphism may not occur in universities that pursue the rankings developed by business schools. Some of these choose to distinguish themselves by resorting to other ways of achieving excellence not computed by this evaluation system. Taking this argument into account, perhaps the GPs assessed here as inefficient in terms of their publications will perform well in other aspects, such as their social impact, for instance (Wood Jr, Costa, Lima, & Guimarães, 2016), made possible by their actions at the local and regional spheres.

Unlike these rankings, the “rules of the game” outlined by CAPES are based on coercive and normative mechanisms, and its assessment system may determine the very survival of graduate programs (Andrade et al., 2018; Patrus et al., 2018). Any eventual non-compliance with new criteria could, in the future, lead to the discontinuation of less prestigious GPs or those that have failed to achieve high performance in terms of their publications in case they fail to fulfill the new goals. In this sense, the homogenization of research practices could derive not from conforming with the new criteria but from the sole maintenance of GPs that managed to adapt to them. This outcome could further aggravate the area’s concentration and deepen the regional gaps in terms of scientific development.

6. CONCLUSION

This paper analyzed the relative efficiency of graduate programs [GPs] in terms of the quality of their publications in journals of higher (A1, A2, and B1) and lower ratings (B2 to B5). To this end, we applied Data Envelopment Analysis to the Public and Business Administration, Accounting, and Tourism area of concentration, using data from the 2017 Quadrennial Evaluation developed by CAPES. The results showed that efficient GPs are concentrated in the center-south axis of Brazil and the areas of administration and accounting. Regarding the characteristics of the GPs’ efficiency, the programs that stood out optimally diversified their scientific publications among journals in all strata analyzed. We must highlight that efficient GPs predominantly allocated their efficiency in A1 journals, those regarded as of higher impact by CAPES due to their prominence and high international prestige.

According to the results, this study advances the understanding of the consequences of institutional isomorphism in Brazilian GPs and HEIs, through an integrated relationship between the coercive and normative mechanisms put forth by CAPES. In the Brazilian context, the current model, developed jointly between the government and academic representatives, leads to a single form of assessment, which tries to reconcile the legitimacy of research practices with the high-impact indicators adopted internationally, and therefore ensures the survival of GPs. This contradictory and complex scenario contributes to the development of a heterogeneous environment where GPs may prosper. In this sense, institutional isomorphism has not been achieved equitably among the GPs analyzed herein.

CAPES (2020a) has pointed in that direction by proposing a multidimensional assessment model that includes other quality criteria to adjust to new contexts and improve their current standards. High-impact scientific journals continue to benchmark for the highest quality scientific production, so GPs should try to adapt to this reality. Therefore, it is intended that these research practices, internationally legitimized, will be implemented here equally.

The GPs that come to achieve this, or those most likely to move forward by incorporating these new rules, will survive. As their scientific product can achieve international prominence, these GPs would become dominant in the national context, just as their research practices would be regarded as models of excellence to be adopted and legitimized by other programs. In this regard, these GPs are closer to becoming internationally recognized centers, but does that imply that only they should be encouraged?

The results showed limitations that can pave the way for future research. The model proposed herein evaluated only a single quadrennium, so it would be interesting to understand how the efficiency of GPs changes due to the quality criterion in the following quadrennium; that is, how this change influences the research practices in the long run. New factors could be explored, such as the impact of the value of projects financed in publications or the impact of teaching and the quality of the training of masters and doctors in the research results of graduate programs. Furthermore, the context surrounding HEIs on their research efficiency can also be investigated based on the impact of having their own financial resources, their level of international prominence, or even the presence of research and national and international cooperation networks among GPs in several areas of knowledge. We also suggest that future studies address how normative mechanisms become coercive during the assessment; that is, how the values of ad hoc consultants could be incorporated into the outlining of evaluation criteria, which could potentially become CAPES’ “rules of the game” in the future.

ACKNOWLEDGEMENT

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001 and by Conselho Nacional de Desenvolvimento Científico e Tecnológico - CNPq.

REFERENCES

• Agasisti, T., & Pérez-Esparrells, C. (2010). Comparing efficiency in a cross-country perspective: the case of Italian and Spanish state universities. Higher Education, 59, 85-103. https://doi.org/10.1007/s10734-009-9235-8
  » https://doi.org/10.1007/s10734-009-9235-8
• Alcadipani, R. (2011a). Academia e a fábrica de sardinhas. Organizações & Sociedade, 18(57), 345-348. https://portalseer.ufba.br/index.php/revistaoes/article/view/11155/8067
  » https://portalseer.ufba.br/index.php/revistaoes/article/view/11155/8067
• Alcadipani, R. (2011b). Resistir ao produtivismo: uma ode à perturbação acadêmica. Cadernos EBAPE.BR, 9(4), 1174-1178. http://dx.doi.org/10.1590/S1679-39512011000400015
  » http://dx.doi.org/10.1590/S1679-39512011000400015
• Andrade, E. F. S., Oliveira, J., Maccari, E. A., & Hollnagel, H. (2018). Inovação em um contexto isomórfico nos Programas de Pós-Graduação Stricto Sensu em Administração. Revista Gestão & Tecnologia, 18(2), 100-127. http://revistagt.fpl.edu.br/get/article/view/1415/847
  » http://revistagt.fpl.edu.br/get/article/view/1415/847
• Barrett, B., Fernandez, F., & Gonzalez, E. M. (2020). Why universities voluntarily pursue US accreditation: the case of Mexico. Higher Education, 79(3-4), 1-17. https://doi.org/10.1007/s10734-019-00427-y
  » https://doi.org/10.1007/s10734-019-00427-y
• Bogetoft, P., Otto, L. (2011). Benchmarking with DEA, SFA, and R. Media New York: Springer.
• Cook, W. D., & Zhu, J. (2008) Data Envelopment Analysis: Modeling Operational Processes and Measuring Productivity Boston: CreateSpace.
• CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. (2017a). Avaliação Quadrienal 2017 Recuperado de http://avaliacaoquadrienal.capes.gov.br/home
  » http://avaliacaoquadrienal.capes.gov.br/home
• CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. (2017b). Documento de Área 2017 - Administração, Ciências Contábeis e Turismo Recuperado de http://avaliacaoquadrienal.capes.gov.br/resultado-da-avaliacao-quadrienal-2017-2
  » http://avaliacaoquadrienal.capes.gov.br/resultado-da-avaliacao-quadrienal-2017-2
• CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. (2020a). Sobre a Avaliação Recuperado de https://www.capes.gov.br/images/novo_portal/documentos/PNPG/25052020_Relat%C3%B3rio_Final__2019_Comiss%C3%A3o_PNPG.pdf
  » https://www.capes.gov.br/images/novo_portal/documentos/PNPG/25052020_Relat%C3%B3rio_Final__2019_Comiss%C3%A3o_PNPG.pdf
• CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. (2020b). Sobre a Avaliação Recuperado de https://www.capes.gov.br/perguntas-frequentes
  » https://www.capes.gov.br/perguntas-frequentes
• Costa, E. M., Ramos, F. S., Souza, H. R., Sampaio, L. M. B., & Barbosa, R. B. (2015). Dinâmica da eficiência produtiva das instituições federais de ensino superior. Planejamento e Políticas Públicas, (44), 51-84.
• Costa, E. M., Souza, H. R. de, Ramos, F. de S., & Silva, J. L. M. da. (2012). Eficiência e desempenho no ensino superior: uma análise da fronteira de produção educacional das IFES brasileiras. Revista de Economia Contemporânea, 16(3), 415-440. https://doi.org/10.1590/S1415-98482012000300003
  » https://doi.org/10.1590/S1415-98482012000300003
• Croucher, G., & Woelert, P. (2016). Institutional isomorphism and the creation of the Unified National System of Higher Education in Australia: An empirical analysis. Higher Education, 71(4), 439-453. https://doi.org/10.1007/s10734-015-9914-6
  » https://doi.org/10.1007/s10734-015-9914-6
• Czarniawska, B. (2009). Emerging institutions: pyramids or anthills?Organization Studies,30(4), 423-441. https://doi.org/10.1177/0170840609102282
  » https://doi.org/10.1177/0170840609102282
• DiMaggio, P. J., & Powell, W. W. (1983). The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields. American sociological review, 48(2), 147-160. https://doi.org/10.2307/2095101
  » https://doi.org/10.2307/2095101
• Dyson, R. G., Allen, R., Camanho, A. S., Podinovski, V. V., Sarrico, C. S., & Shale, E. A. (2001). Pitfalls and protocols in DEA. European Journal of Operational Research, 132(2), 245-259. https://doi.org/10.1016/S0377-2217(00)00149-1
  » https://doi.org/10.1016/S0377-2217(00)00149-1
• Geva-May, I. (2001). Higher education and attainment of policy goals: Interpretations for efficiency indicators in Israel. Higher Education, 42(3), 265-305. https://doi.org/10.1023/A:1017550802162
  » https://doi.org/10.1023/A:1017550802162
• Holanda, L. A., Widmer, G. M., & Leal, S. (2014). A produção científica em turismo no Brasil: reflexões e proposições a partir de um estudo revisional. Anais Brasileiros de Estudos Turísticos, 4(1), 72-79.
• Imasato, T., Perlin, M. S., & Borenstein, D. (2017). Análise do perfil dos acadêmicos e de suas publicações científicas em Administração.Revista de Administração Contemporânea,21(1), 62-83. https://doi.org/10.1590/1982-7849rac2017150329
  » https://doi.org/10.1590/1982-7849rac2017150329
• Jelonek, M., & Mazur, S. (2020). Necessary changes, adverse effects? The institutional patterns of adaptation of economics universities to changes prompted by the reform of Poland’s science and higher education system. Management Learning, 51(4), 472-490. https://doi.org/10.1177/1350507620913896
  » https://doi.org/10.1177/1350507620913896
• Johnes, G., & Johnes, J. (1993). Measuring the research performance of UK economics departments: an application of data envelopment analysis. Oxford Economic Papers, 45(2) 332-347.
• Joumady, O., & Ris, C. (2005). Performance in European higher education: a non-parametric production frontier approach. Education Economics, 13(2), 189-205. https://doi.org/10.1080/09645290500031215
  » https://doi.org/10.1080/09645290500031215
• Kezar, A., & Bernstein-Sierra, S. (2019). Examining processes of normative isomorphism and influence in scaled change among higher education intermediary organizations. AERA Open, 5(4), 1-16. https://doi.org/10.1177/2332858419884905
  » https://doi.org/10.1177/2332858419884905
• Liu, J., Lu, L., Li, W., & Lin, B. (2013). A survey of DEA applications. Omega, 41(5), 893-902. https://doi.org/10.1016/j.omega.2012.11.004
  » https://doi.org/10.1016/j.omega.2012.11.004
• Lozano, J. M., Bofarull, I., Waddock, S., & Prat-I-Pubill, Q. (2018). Avoiding the iron cage of business school rankings. Higher Education Policy, 33(1), 135-157. https://doi.org/10.1057/s41307-018-0107-7
  » https://doi.org/10.1057/s41307-018-0107-7
• Maccari, E. A., Rodrigues, L. C., Alessio, E. M., & Quoniam, L. M. (2008). Sistema de avaliação da pós-graduação da Capes: pesquisa-ação em um programa de pós-graduação em Administração. Revista Brasileira de Pós-Graduação, 5(9). https://doi.org/10.21713/2358-2332.2008.v5.147
  » https://doi.org/10.21713/2358-2332.2008.v5.147
• Meyer, J. W., & Rowan, B. (1977). Institutionalized organizations: Formal structure as myth and ceremony. American Journal of Sociology, 83(2), 340-363. https://www.jstor.org/stable/2778293
  » https://www.jstor.org/stable/2778293
• Monfared, M. A. S., & Safi, M. (2013). Network DEA: an application to analysis of academic performance. Journal of Industrial Engineering International, 9(1), 1-10. https://doi.org/10.1186/2251-712X-9-15
  » https://doi.org/10.1186/2251-712X-9-15
• Moreira, N. P., Cunha, N. R. da S., Ferreira, M. A. M., & Silveira, S. de F. R. (2011). Fatores determinantes da eficiência dos programas de pós-graduação acadêmicos em Administração, Contabilidade e Turismo. Avaliação: Revista da Avaliação da Educação Superior, 16(1), 201-230. https://doi.org/10.1590/S1414-40772011000100011
  » https://doi.org/10.1590/S1414-40772011000100011
• Nascimento, S. do, & Beuren, I. M. (2011). Redes sociais na produção científica dos programas de pós-graduação de ciências contábeis do Brasil. Revista de Administração Contemporânea, 15(1), 47-66. https://doi.org/10.1590/S1415-65552011000100004
  » https://doi.org/10.1590/S1415-65552011000100004
• Patrus, R., Shigaki, H. B., & Dantas, D. C. (2018). Quem não conhece seu passado está condenado a repeti-lo: distorções da avaliação da pós-graduação no Brasil à luz da história da Capes. Cadernos EBAPE.BR, 16(4), 642-655. https://dx.doi.org/10.1590/1679-395166526
  » https://dx.doi.org/10.1590/1679-395166526
• Ramos, M. (2018). Internacionalização da pós-graduação no Brasil: lógica e mecanismos. Educação e Pesquisa, 44, 1-10. https://doi.org/10.1590/s1517-9702201706161579
  » https://doi.org/10.1590/s1517-9702201706161579
• Rego, T. C. (2014). Produtivismo, pesquisa e comunicação científica: entre o veneno e o remédio. Educação e Pesquisa, 40(2), 325-346. https://doi.org/10.1590/S1517-97022014061843
  » https://doi.org/10.1590/S1517-97022014061843
• Rossoni, L., & Guarido Filho, E. R. (2009). Cooperação entre programas de pós-graduação em administração no Brasil: evidências estruturais em quatro áreas temáticas. Revista de Administração Contemporânea, 13(3), 366-390. https://doi.org/10.1590/S1415-65552009000300003
  » https://doi.org/10.1590/S1415-65552009000300003
• Silva, J. S., Corrêa, C. R. & Gomes, A. P. (2016). Determinantes da eficiência dos programas de pós-graduação em Economia do Brasil. Reflexões Econômicas, 2(2), 55-75.
• Thanassoulis, E. (2001). Introduction to the Theory and Application of Data Envelopment Analysis New York: Springer .
• Thornton, P. H., William, O., & Lounsbury, M. (2012). The institutional logics perspective: A new approach to culture, structure, and process Oxford: Oxford University Press.
• Woelert, P., & Croucher, G. (2018). The multiple dynamics of isomorphic change: Australian law schools 1987-1996. Minerva: a review of science, learning and policy, 56(4), 479-503. https://doi.org/10.1007/s11024-018-9350-8
  » https://doi.org/10.1007/s11024-018-9350-8
• Vakkayil, J., & Chatterjee, D. (2017). Globalization routes: The pursuit of conformity and distinctiveness by top business schools in India. Management Learning, 48(3), 328-344. https://doi.org/10.1177/1350507616679347
  » https://doi.org/10.1177/1350507616679347
• Witte, K., & López-Torres, L. (2017). Efficiency in education: a review of literature and a way forward. Journal of the Operational Research Society, 68(4), 339-363. https://doi.org/10.1057/jors.2015.92
  » https://doi.org/10.1057/jors.2015.92
• Wood T. Jr., Costa, C. C. M., Lima, G. D. M. R., & Guimarães, R. C. (2016). Impacto social: Estudo sobre programas brasileiros selecionados de pós-graduação em administração de empresas. Revista de Administração Contemporânea, 20(1), 21-40. https://doi.org/10.1590/1982-7849rac20161842
  » https://doi.org/10.1590/1982-7849rac20161842
• Xu, X., Rose, H., & Oancea, A. (2019). Incentivising international publications: institutional policymaking in Chinese higher education. Studies in Higher Education, 1-14. https://doi.org/10.1080/03075079.2019.1672646
  » https://doi.org/10.1080/03075079.2019.1672646

Appendix 1.

Simplified illustrative example of a CRS frontier with only one input (Number of professors) and one output (A1 publications)

Appendix 2.

Efficiency, virtual outputs, maximized virtual outputs, and the number of times each efficient GP is a benchmark for other programs

		Virtual outputs			Maximized virtual outputs			Benchmark for
Acronym	Efficiency	A1	A2 and B1	B2 to B5	A1	A2 and B1	B2 to B5
FUCAPE (C)	1.00	0.47	0.53	0.00	0.37	0.37	0.25	9
UFRJ (A)	1.00	1.00	0.00	0.00	0.66	0.34	0.00	10
UFRJ (C)	1.00	0.10	0.90	0.00	0.19	0.74	0.07	1
PUC-RIO (A)	1.00	0.78	0.00	0.22	0.77	0.01	0.22	1
FGV/RJ (A)	1.00	1.00	0.00	0.00	0.45	0.45	0.11	8
UFV (PA)	1.00	0.00	0.00	1.00	0.26	0.37	0.37	3
UFU (C)	1.00	0.00	1.00	0.00	0.00	0.50	0.50	18
USP (C)	1.00	0.00	1.00	0.00	0.07	0.58	0.35	18
USP/RP (A)	1.00	1.00	0.00	0.00	0.59	0.41	0.00	5
PUC/SP (A)	1.00	1.00	0.00	0.00	0.38	0.24	0.38	1
UNIMEP (A)	1.00	0.00	0.00	1.00	0.33	0.33	0.33	1
FEI (A)	1.00	0.95	0.00	0.05	0.44	0.44	0.12	5
UNINOVE (A)	1.00	0.11	0.00	0.89	0.33	0.33	0.33	18
UAM (T)	1.00	0.00	0.00	1.00	0.00	0.42	0.58	3
USCS (A)	1.00	0.00	0.00	1.00	0.33	0.33	0.33	7
UFPR (C)	1.00	0.00	1.00	0.00	0.00	0.97	0.03	1
UFSC (C)	1.00	0.21	0.12	0.67	0.33	0.33	0.33	15
FURB (C)	1.00	0.00	0.57	0.43	0.16	0.42	0.42	22
UFRGS (A)	1.00	1.00	0.00	0.00	0.44	0.44	0.12	2
UFSM (A)	1.00	0.00	0.00	1.00	0.19	0.41	0.41	9
PUC/RS/UCS (A)	1.00	0.32	0.00	0.68	0.33	0.33	0.33	2
UCS (A)	1.00	0.44	0.00	0.56	0.33	0.33	0.33	9
UFMS (A)	1.00	0.00	0.45	0.55	0.00	0.50	0.50	4
UNB (C)	1.00	0.00	1.00	0.00	0.33	0.33	0.33	4
Mean		0.35	0.27	0.38	0.30	0.41	0.28

• Source: Research data.
• Note. (A) = Graduate program in Administration (C) = Graduate program in Accounting (T) = Graduate program in Tourism

Appendix 3.

Efficiency and virtual outputs of inefficient GPs

		Virtual outputs
Acronym	Efficiency	A1	A2 and B1	B2 to B5
UFRN (T)	0.99	0.25	0.00	0.75
FGV/SP (A)	0.96	0.53	0.47	0.00
UNB (PA)	0.96	0.49	0.51	0.00
UFPE (C)	0.95	0.15	0.01	0.84
UNAMA (A)	0.93	0.00	0.07	0.93
UNISINOS (A)	0.88	0.47	0.53	0.00
PUC/RS (A)	0.85	1.00	0.00	0.00
FGV/SP (PA)	0.83	0.50	0.50	0.00
UCS (T)	0.80	0.00	0.00	1.00
UFC (A)	0.77	0.13	0.58	0.28
PUC/PR (A)	0.67	0.06	0.87	0.08
ESPM (A)	0.66	0.18	0.82	0.00
UFES (A)	0.65	0.28	0.65	0.07
USP (A)	0.65	0.28	0.72	0.00
UFPR (A)	0.63	0.06	0.94	0.00
UFLA (A)	0.62	0.00	0.97	0.03
UNIVALI (A)	0.61	0.34	0.00	0.66
UNISINOS (C)	0.60	0.28	0.72	0.00
UFSC (A)	0.59	0.00	0.77	0.23
USP/RP (C)	0.56	0.03	0.97	0.00
UFBA (PA)	0.55	0.34	0.66	0.00
FUMEC (A)	0.54	0.38	0.53	0.08
UECE (A)	0.53	0.00	0.24	0.76
UPM (A)	0.52	0.23	0.42	0.35
UDESC (A)	0.50	0.07	0.09	0.84
UNIFOR (A)	0.47	0.00	0.12	0.88
UFRN (A)	0.46	0.00	1.00	0.00
UNIGRANRIO (A)	0.46	0.00	1.00	0.00
UFMG (A)	0.45	0.11	0.39	0.49
UFPB/J.P. (A)	0.45	0.10	0.90	0.00
UP (A)	0.41	0.08	0.92	0.00
UEM (A)	0.41	0.00	1.00	0.00
PUC/MG (A)	0.37	0.00	0.96	0.04
UNIVALI (T)	0.33	0.00	0.82	0.18
UFPE (A)	0.30	0.00	0.57	0.43
Mean	0.63	0.18	0.56	0.25

• Source: Research data.
• Note. (A) = Graduate program in Administration (PA) = Graduate program in Public Administration (C) = Graduate program in Accounting (T) = Graduate program in Tourism

Appendix 4.

Number of GPs classified at different levels of participation in the publication strata

	Number of efficient GPs			Number of inefficient GPs
Participation of the type of publication in the efficiency score	A1	A2 and B1	B2 to B5	A1	A2 and B1	B2 to B5
Null (0)	11	15	11	12	4	16
Low - Intermediate (>0% & <=25%)	3	1	2	11	5	7
High (>25% & <= 50%)	3	1	1	9	4	4
Preponderant (>50%)	7	7	10	3	22	8
Total GPs	24	24	24	35	35	35

• Source: Research data.

Publication Dates

• Publication in this collection
  12 Jan 2022
• Date of issue
  Nov-Dec 2021

History

• Received
  07 Oct 2020
• Reviewed
  27 Jan 2021
• Accepted
  22 Mar 2021
• Published
  05 Nov 2021