Self-assessment in higher education institutions: literature analysis and research opportunities

Autoavaliação em instituições de ensino superior: análise de literatura e oportunidades de pesquisa

Abstract

Self-assessment in higher education institutions is a topic of increasing interest for the academic community, researchers, the government, and society. This process allows us to map the reality of the institution, becoming the foundation for proposing improvement actions. Self-assessment offers the potential to manage changes if they are considered in decision-making; universities, however, have difficulty using self-assessment as an management instrument; the step-by-step of this process and the use of its results are not effectively presented in the literature. Thus, this study aims to analyze the guidelines for self-assessment methods in higher education institutions, considering their processes, results, and effectiveness. The analysis of previously proposed attributes help to present the potential for developing a better practice in this area. A systematic review of the literature was conducted in several databases, including the Brazilian Digital Library of Theses and Dissertations. A total of 119 documents, relevant to this research were found, in which 62 presented some guideline that considered planning and management for decision-making. These guidelines may be considered as a possible systematic approach for institutional self-assessment. Additionally, research gaps were identified, supporting the appointment of opportunities for researchers and managers within the scope of process management, work routine, and human resources.

Keywords
Institutional self-assessment; Higher education institution; Decision-making

Resumo

A autoavaliação nas instituições de ensino superior é um tema que vem atraindo cada vez mais interesse da comunidade acadêmica, de pesquisadores, do governo e da sociedade. É um processo por meio do qual se busca mapear a realidade da instituição, sendo a base para propostas de ações de melhoria. A autoavaliação oferece potencial de gerenciar mudanças, desde que seja considerada na tomada de decisões; há, porém, dificuldade em se utilizar a autoavaliação como instrumento na gestão universitária. Além disso, o passo a passo desse processo e o uso de seus resultados não são apresentados efetivamente na literatura. Assim, este artigo tem por objetivo analisar diretrizes para métodos de autoavaliação em instituições de ensino superior sob o viés de processos, resultados e eficácia. A análise dos atributos já propostos auxilia na apresentação dos potenciais existentes para desenvolver uma possível melhor prática nessa área. Uma revisão sistemática da literatura foi realizada em bases de dados e na Biblioteca Digital Brasileira de Teses e Dissertações. Sem limitação de tempo, foram encontrados 119 documentos relevantes para esta pesquisa, sendo que 62 deles apresentaram alguma diretriz que considerou planejamento e gestão para tomada de decisão. Como resultado, foram obtidas diretrizes para serem consideradas em uma possível sistemática para a autoavaliação institucional. A partir disso, identificaram-se lacunas que apoiaram a identificação de oportunidades para pesquisadores e gestores no âmbito da gestão de processos, rotina de trabalho e recursos humanos.

Palavras-chave
Autoavaliação institucional; Instituição de ensino superior; Tomada de decisão

Introduction

The quality of education is important for the development of a country and can be considered as a form of public asset (SANTOS, 2011). Therefore, the pursuit for quality in education has been the object of study within several research fields, with the field of institutional assessment currently gaining strength, emphasizing the challenges that involve the diagnosis of the inner workings of an educational institution (BALDIGEN, 2018).

In Brazil, the National Higher Education Assessment System ( Sistema Nacional de Avaliação da Educação Superior – SINAES) was created to improve the quality of higher education; guide its expansion; permanently increase its institutional, academic, and social effectiveness; and, especially, to promote the deepening of the social commitments and responsibilities of higher education institutions (HEI). For the evaluation of the institutions, diversified procedures and instruments are used, including self-assessment and external evaluation in loco (BRASIL, 2004). The former is the focus of this study.

Institutional self-assessment (ISA), a component of SINAES, is a continuous process in which the organization builds knowledge on its own reality, seeking to understand the meanings of its activities to achieve greater social relevance. To this end, the evaluation systematizes information; collectively analyzes the meanings of its achievements; unravels forms of organization, management, and action; identifies weaknesses, as well as strengths and potential; and establishes strategies to overcome problems. Such assessments are legally required to be conducted by the HEI. Based on the principle of university autonomy, each institution is responsible for developing the process in the manner most appropriate to its context. The evaluation has a formative character and will constitute the basic framework for the regulation and supervision processes within higher education, thus promoting the improvement of its quality.

Considering the role of the self-assessment processes within the context of higher education regulations, we observed that many HEI perform the assessment only to meet the requirements of regulatory bodies, instead of using it as a management and planning instrument, making the self-assessment process a bureaucratic issue (LIMA, C. I., 2010) rather than an institutional management policy. This process offers the potential to manage transformations since it can be used to support managers’ decision-making process ( NARDELLI, 2019; GONÇALVES FILHO, 2016). Universities, however, have difficulty in using self-assessment as a management tool ( CARVALHO; OLIVEIRA; LIMA, 2018; AOKI, 2017; ROSA et al., 2011 ) the step by step of this process and the effective use of its results are not presented in the literature. Additionally, this study presents research gaps, seeking challenging opportunities for researchers to establish new research flows and for managers and decision-makers to formulate effective implementation strategies. Thus, we seek to answer the following research question: What guidelines are used for self-assessment in higher education institutions? Identifying the current attributes of this process may assist in the development of better practices in this area.

To systematize elements that may assist in solving the research problem, we sought to analyze the guidelines for self- assessment methods in higher education institutions, considering their processes, results, and effectiveness. Guidelines are instructions that support the development of an institutional self-assessment systematicapp which considers the processes and results for effective decision-making in university management.

Self-assessment in higher education institutions

Evaluations became important as a management tool in social organizations, mainly with the publication on the principles of scientific administration by Frederic Taylor, in 1911 ( Figure 1). This work criticizes administrators for making decisions based on intuitions and experiences, without using performance standards.

There is not a single definition for evaluation. The classic concept – proposed, and later complemented, by Scriven ( SCRIVEN, 1967, 1994) – states that evaluation is the judgment of importance, merit, and value. The evaluation must be contextualized, conversing with the object to be evaluated, its theoretical assumptions, its political articulations, and the actors involved (ARRUDA; PASCHOAL; DEMO, 2019). Evaluating is not just about raising numbers, but raising actions based on those numbers. The assessment must be credible, but it cannot claim to be the conclusive demonstration of the truth. Its function is not to reveal nor determine the truth, but rather to ground the possibilities for reflective social processes that produce meanings on actions, relationships, and educational productions (DIAS SOBRINHO, 2008).

According to Souza (2010), the Organization for Economic Cooperation and Development (OECD) was the only international institution in the 1960s that evaluated educational systems worldwide, providing information for the formulation and monitoring of policies for socioeconomic development. The OECD has been operating since then (OECD, 2011), and they annually present an overview of world education within its member countries, Brazil included.

The first discussions about institutional evaluation in Brazil started in the 1980s (BARREYRO; ROTHEN, 2006). Brazilian public universities argued that the evaluation was necessary so that the principle of transparency could be fulfilled, that is, of accountability to society. During this period, the discussion on evaluation gained another perspective: not only was it an instrument of transparency but also a concern with quality and autonomy (BALZAN; DIAS SOBRINHO, 1995). Since the early 1990s, the theme of institutional evaluation has been gaining consistency regarding the principles of university autonomy and quality. Several attempts to evaluate higher education institutions have been made. In 2004, however, institutional self-assessment became mandatory in Brazil, subject to the National Higher Education Evaluation System, with the aim of ensuring the national evaluation process in higher education institutions.

Figure 1
Brief history of evaluations with higher education institution

National higher education assessment system

Established by Law No. 10,861 of 2004, SINAES aimed to improve the quality of higher education; guide the expansion of its offer; permanently increase its institutional, academic, and social effectiveness; and, especially, promote the deepening of social commitments and responsibilities of higher education institutions, by promoting their public mission, their democratic values, the respect for difference and diversity, the affirmation of autonomy, and their institutional identity (BRASIL, 2004).

The evaluation of higher education institutions aims to identify their profile and the meaning of their performance by their activities, courses, programs, projects, and sectors, considering different institutional dimensions, such as: I – mission and institutional development plan; II – the policy for teaching, research, and extension; III – the social responsibility of the institution; IV – communication with society; V – personnel policies; VI – organization and management of the institution; VII – physical infrastructure; VIII – planning and evaluation, especially the processes, results, and effectiveness of institutional self-evaluations; IX – student service policies; and X – financial sustainability. For the assessment of institutions, diversified procedures and instruments are used, among which is the self-assessment (BRASIL, 2004).

Each university develops the best way to conduct its self-assessment process. This study is developed from the perspective of institutional self-assessment and, in section 4.2, we show the systemic analysis for the dimension of planning and evaluation. The ISA contains analyses, criticisms, and suggestions and is configured as an important instrument for decision-making insofar as it conceives a collective discussion about the institution with the subjects that compose it, giving legitimacy, autonomy, and meaning to this analysis. This process enables transparency and, thus, allows for decision-making that makes public spending more efficient (ARRUDA; PASCHOAL; DEMO, 2019). However, the scenario identified by Cunha (2010) reveals that the use of ISA results represents a challenge for Brazilian HEI, contributing little to university management. According to Arruda, Paschoal, and Demo (2019), the strongest criticism made of the evaluation is that its results are not used for decision-making, which raises the question of the relevance of the evaluation itself.

Methodological procedures

This subsection presents the methodological framework of the investigation to inform the assumptions that guided its execution. From the point of view of nature, this qualitative and quantitative research is classified as basic since it aims to generate new knowledge useful for the advancement of science without an expected practical application. It has an exploratory-descriptive character: it is exploratory since the main agent is the researcher who generates knowledge on self-assessment in higher education institutions with the purpose of providing more information on the subject to be investigated; it is descriptive since the researcher only records and describes the facts observed without interfering with them (PRODANOV; FREITAS, 2013).

The technical procedures, that is, the way in which the necessary data were obtained for the elaboration of the research, was elaborated from material already published, consisting of scientific articles and national theses and dissertations. Regarding data collection, both primary and secondary data were used (PRODANOV; FREITAS, 2013). The stage of selection of the bibliographic portfolio (BP) made use of primary and secondary data.

ProKnow-C (ENSSLIN; ENSLLIN; DUTRA, 2015), the instrument used to assist in the operationalization of this research, is composed of four phases ( Figure 2).

Figure 2
Phases of ProKnow-C

The selection process of the bibliographic portfolio, the first stage of ProKnow-C, begins with the definition of research axes and their keywords ( Figure 3). For searching the databases, keywords in English were used, in addition to their respective translations and adaptations to the Portuguese language. Our study focuses on studies on institutional self-assessments in higher education institutions.

Figure 3
Axes of search and keywords

The databases selected for this research were Scopus, Web of Science, EBSCOhost, Engineering Village (Compendex), and SciELO since they are those of the highest priority in the area. The Brazilian Digital Library of Theses and Dissertations (BDTD) was also searched, since this is a topic with specificities in Brazil, in addition to searches conducted on Google and Google Scholar. There was no time limitation in any of the filter searches and the search was carried out looking for the defined keywords within titles, abstract, and/or keywords of the sources. The search was limited to articles published in English and Portuguese. However, there was no distinction in the type of article, that is, articles from both journals and congresses were considered. The database search resulted in 608 documents, totaling 327 after duplicates were eliminated.

A total of 970 theses and dissertations were obtained, 279 publications were identified as duplicates by EndNote X9 and were subsequently excluded, resulting in 328 references. In the filtering phase of the raw article bank, the title of each article was examined and those that were clearly misaligned were excluded for not contributing to this research, according to our pre-defined criteria (self-assessment and higher education institution). The search on the databases resulted in 196 documents with eligible titles. These same criteria were used for the other sources of this research, resulting in 106 (33 theses and 73 dissertations) documents with eligible titles in the BDTD. The documents with eligible titles found at Google and Google Scholar totaled 39. Three of these were duplicated from the databases and six derived from BDTD (three theses and three dissertations).

After completing the next step – which aimed to identify the alignment of the abstracts with the author’s interest (presenting characteristics of the self-assessment processes in higher education institutions) – 44 documents were obtained from the databases; 23 theses; 37 dissertations and; nine documents from Google and Google Scholar; eight articles (one from a congress and seven from journals); and one thesis. For organizational reasons to assist in the analysis, these last two were added to the database and BDTD and resulted in 113 documents.

The references of the theses and dissertations and articles with more than 10 citations were analyzed, since these comprised 85% of the total citations of the articles already selected. This process was an opportunity to check if there was any relevant work according to the same pre-established criteria. Six new documents were found (a thesis, a dissertation, and four articles), totaling 119 documents which make up the study portfolio of this article.

Based on this portfolio on self-assessment in higher education institutions, 62 documents described some forms of guidelines that considered planning and management for decision-making, these were the basis for the results of the bibliometric and systemic analysis of the portfolio.

Presentation and discussion of results

The bibliometric analysis of the results, the second stage of ProKnow-C, aims to highlight the variables identified in the bibliographic portfolio. From this information, complementary data is searched and analyzed to build knowledge on the subject. This procedure is performed by counting the occurrence of a certain variable within the bibliographic portfolio, as well as in its references. This section also presents the systemic analysis of the results and, finally, presents research opportunities.

Bibliometric analysis

The bibliometric analysis of the results, considering the basic variables of the articles from the databases, shows: (1) number of articles per year of publication; and (2) production by country in which the study was developed. In relation to theses and dissertations, it exposes: (3) quantity of publication of theses and dissertations per year; and (4) concentration area of the author’s graduate program.

Figure 4 corroborates Balzan and Dias Sobrinho (1995), since it points to the first discussions about institutional evaluation in the early 1980s, although more articles have been published in the last five years.

Figure 4
Number of articles per year of publication

Figure 5 shows that Brazil was the country with the most articles published on the selected subject, followed by the United States of America and Argentina. The other countries had a small number of studies carried out on the subject.

Figure 5
Productions by country

From the BDTD documents – theses and dissertations – an increase in publications in the last decade is noticeable ( Figure 6).

Figure 6
Number of publication of theses and dissertations per year

More than half of the theses and dissertations in the bibliographic portfolio were from the area of Education, followed by Production Engineering and Administration, respectively ( Figure 7).

Figure 7
Concentration area of the author’s graduate program

Systemic analysis

The systemic analysis of the results, the third stage of ProKnow-C, consists in analyzing the content and verifying the characteristics of the bibliographic portfolio on the subject to identify knowledge gaps. From this stage on, the portfolio analysis becomes one, that is, the articles and the theses and dissertations are grouped together. All dimensions of SINAES must be considered in the institutional self-assessment process (RIBEIRO, J., 2015). Our study, however, is limited to the analysis of the portfolio from the perspective of planning and assessment, especially the processes, results, and effectiveness of the self-assessment.

This decision is based on the importance of that dimension in the evaluation process in higher education institutions. The analysis from this perspective implies subsidies for university management, especially regarding decision- making. Thus, Table 1 points out guidelines for self-assessment in higher education institutions. Some guidelines were proposed in this system and corroborated in the bibliographic portfolio, others came from specific experiences of the authors of the 62 documents.

Table 1
Guidelines for ISA regarding processes, results, and effectiveness

PROCESS	ISA planning	Andriola and Araújo (2016); Andrade (2014); J. P. Lima (2018); Matoski Júnior (2008); Oliveira (2013); Souza (2010)
	Institutional mission, vision, and objectives	Donato and Ladeia (2019); Goulart (2018); E. A. Ribeiro (2010); Rodrigues (2003); Rosa et al.(2011); Soares (2016); Souza (2010)
	Contextualization	Abreu Júnior (2009); Andrade (2014); Andriola and Araújo (2016); Brito (2006); Cardoso (2017); Carvalhaes (2018); Dias (2007); Donato and Ladeia (2019); Gonçalves (2016); Gonçalves Filho (2016); Goulart (2018); Holzweiss, Bustamante, and Fuller, (2016); L. M. Lima (2011); Martins (2010); E. A. Ribeiro (2010); Rosa et al.(2011); Sais (2017); Soares (2016); Souza (2010)
	Democratic management	Andrade (2014); Andriola and Araújo (2016); Goulart (2018); L. M. Lima (2011); Lins et al. (2017); Lopes (1994); Menezes (2012); E. A. Ribeiro (2010); A. L. Silva (2015)
	Active participation of the academic community	Abreu Júnior (2009); Amâncio (2019); Andrade (2014); Andriola and Araújo (2016); Angst and Alves (2018); Arruda, Paschoal, and Demo (2019); Bernardes and Rothen (2016); Brito (2006); Cunha (2010); Dias (2007); Donato and Ladeia (2019); Gonçalves (2016); Gonçalves Filho (2016); Goulart (2018); Hamill (2015); Lehfeld et al. (2017); C. I. Lima (2010); L. M. Lima (2011); J. P. Lima (2018); Lins et al. (2017); Matoski Júnior (2008); Menezes (2012); Nunes, Duarte, and Pereira (2017); Oliveira (2013); Peixoto (2009); E. A. Ribeiro (2010); Rodrigues (2003); Rosa et al. (2011); Rosas (2014); A. L. N. Silva et al. (2019); J. R. C. Silva et al. (2019); Venturini et al.(2010); Vieira and Freitas (2010)
	10 SINAES dimensions	J. L. L. S. Ribeiro (2015)
	Several instruments	Andriola and Araújo (2016); Oliveira (2013)
	Innovation	Baldigen (2018); Holzweiss, Bustamante, and Fuller (2016); A. L. Silva (2015)
	Legitimacy	Abreu Júnior (2009); Angst and Alves (2018); Arruda, Paschoal, and Demo (2019); Bernardes and Rothen (2016); Cardoso (2017); Cunha (2010); Donato and Ladeia (2019); Goulart (2018); Lehfeld et al. (2017); Martins (2010); Matoski Júnior (2008); Nardelli (2019); Peixoto (2009); Souza (2010); Vieira and Freitas (2010)
	Flexibility	Abreu Júnior (2009); Andriola and Araújo (2016); Cardoso (2017); Lopes (1994)
	Simplicity	Brito (2006); Lehfeld et al. (2017)
	Timely information	Goulart (2018); Menezes (2012); Pinheiro (2018)
	Brief questionnaires / interviews	A. L. N. Silva et al. (2019)
RESULTS	Integration of results to the Institutional Development Plan (IDP)	Castellanelli (2018)
	Indicators	Andriola and Araújo (2016); Brito (2006); L. M. Lima (2011); Oliveira (2013)
	Self-referral	Chinta, Kebritchi, and Ellias (2016); Gonçalves Filho (2016); Martins (2010)
	Aspirational reference	Chinta, Kebritchi, and Ellias (2016); Gonçalves Filho (2016); Martins (2010)
	Transparency	Angst and Alves (2018); Campos (2019); L. M. Lima (2011); Lins et al. (2017); E. A. Ribeiro (2010); Rodrigues (2003); J. R. C. Silva et al. (2019); Vieira and Freitas (2010)
	Interpretation and incorporation of results	Abreu Júnior (2009); Aoki (2017); Arruda, Paschoal, and Demo (2019); Campos (2019); Carvalhaes (2018); Cunha (2010); Dias (2007); Gonçalves (2016); Goulart (2018); Holzweiss, Bustamante, and Fuller (2016); Kwecko (2017); L. M. Lima (2011); J. P. Lima (2018); Martins (2010); Matoski Júnior (2008); Menezes (2012); Nardelli (2019); Pinheiro (2018); Rosas (2014); E. A. Ribeiro (2010); J. L. L. S. Ribeiro (2015); Rodrigues (2003); Sais (2017); Soares (2016); Vieira and Freitas (2010)
	Feedback	Andrade (2014); Cardoso (2017)
	Qualitative emphasis	Andrade (2014); Arruda, Paschoal, and Demo (2019); Brito (2006); Donato and Ladeia (2019); Holzweiss, Bustamante, and Fuller (2016); C. I. Lima (2010); Martins (2010); Matoski Júnior (2008); E. A. Ribeiro (2010)
	Time for analysis	Vieira and Freitas (2010)
	Not punitive, nor rewarding	Cardoso (2017); Carvalhaes (2018); Donato and Ladeia (2019); Martins (2010); Matoski Júnior (2008); Nardelli (2019)
EFFECTIVENESS	Communication	Andriola and Araújo (2016); Arruda, Paschoal, and Demo (2019); Bernardes and Rothen (2016);Cunha (2010); Holzweiss, Bustamante, and Fuller (2016); Kwecko (2017); E. A. Ribeiro (2010); Vieira and Freitas (2010)
	Evaluative culture	Angst and Alves (2018); Aoki (2017); Arruda, Paschoal, and Demo (2019); Bernardes and Rothen (2016); Carvalhaes (2018); Hamill (2015); Holzweiss, Bustamante, and Fuller (2016); Goulart (2018); C. I. Lima (2010); Lins et al. (2017); Matoski Júnior (2008); Nardelli (2019); Nunes, Duarte, and Pereira (2017); Rosas (2014); Sais (2017); A. L. Silva (2015); J. R. C. Silva et al.(2019); Vieira and Freitas (2010)
	Freedom of expression	Andriola and Araújo (2016)
	Continuity	Abreu Júnior (2009); Andriola and Araújo (2016); Bernardes and Rothen (2016); Brito (2006); Cardoso (2017); Donato and Ladeia (2019); Goulart (2018); Martins (2010); E. A. Ribeiro (2010)
	Systemic view	Andriola and Araújo (2016); Cardoso (2017); Donato and Ladeia (2019); Goulart (2018); C. I. Lima (2010); Martins (2010); Nunes, Duarte, and Pereira (2017); Oliveira (2013)
	Change implementation actions	Abreu Júnior (2009); Angst and Alves (2018); Arruda, Paschoal, and Demo (2019); Bernardes and Rothen (2016); Brito (2006); Campos (2019); Cardoso (2017); Carvalhaes (2018); Cunha (2010); Donato and Ladeia (2019); Gonçalves (2016); Gonçalves Filho (2016); Goulart (2018); C. I. Lima (2010); Nunes, Duarte, and Pereira (2017); J. L. L. S. Ribeiro (2015); A. L. Silva (2015); Soares (2016); Venturini et al. (2010); Vieira and Freitas (2010)
	Meta-evaluation	Andriola and Araújo (2016); Abreu Júnior (2009); Andrade (2014); Brito (2006); Cardoso (2017); Carvalhaes (2018); Goulart (2018); Lehfeld et al. (2017); Pinheiro (2018); Rodrigues (2003)
	Feedback from institutional planning	Andrade (2014); Aoki (2017); Baldigen (2018); Brito (2006); Campos (2019); Goulart (2018); Kwecko (2017); Lopes (1994); Matoski Júnior (2008); Peixoto (2009); Pinheiro (2018); E. A. Ribeiro (2010); J. L. L. S. Ribeiro (2015); Rodrigues (2003); Sais (2017); Souza (2010); Vieira and Freitas (2010)
	Technical qualification of those involved	Andriola and Araújo (2016); Angst and Alves (2018); Aoki (2017); Campos (2019); Cardoso (2017); Gonçalves Filho (2016); Kwecko (2017); C. I. Lima (2010); Venturini et al. (2010); Vieira and Freitas (2010)
	Society development	Brito (2006); J. P. Lima (2018); Lopes (1994); A. L. Silva (2015); Souza (2010)
	Bias of the new public management	Castellanelli (2018)
	Concatenation with external evaluation	Cardoso (2017); C. I. Lima (2010)
	Integrated management cycle	Kwecko (2017)

• Source: Elaborated by the authors.

The planning phase – predominantly preceding the institutional self-assessment process – comprises the definition of the organization’s goals, the establishment of a strategy, and the development of an action plan to coordinate activities. Souza (2010) states that the assessment articulated with planning allows the university to deal better with reality. Nevertheless, Campos (2019) highlights the lack of integration of assessment with institutional objectives and planning. Therefore, the characteristics of the institution, its mission, vision, and objectives (GOULART, 2018), its size, and the existence or not of previous evaluative experiences (AOKI, 2017) must be considered. The assessment must enable the educational institution to meet the demand for higher education in the social environment in which it is inserted; it must respect the institutional identity, history, and culture, by using historical series of indicators (ANDRIOLA; ARAÚJO, 2016).

For the evaluation to be used as subsidy for decision-making processes, the results must be analyzed beyond measurement and classification, thus allowing the occurrence of judgment. Therefore, for the assessment to make sense, it needs to be transparent (ANGST; ALVES, 2018), interpreted, discussed, and incorporated by the academic community (NARDELLI, 2019), generating feedback on actions ( CARDOSO, 2017; ANDRADE, 2014).

The use of the results generated by the evaluation process offers conditions for decision-making that can lead to improvements in HEI. However, empirical data showed the lack of commitment to the evaluation process and the non- use of the results (BOTELHO, 2016). The reflection and the judgment of the results have been a complicated task to be performed, according to Carvalho, Oliveira, and Lima (2018). The results of the actions must encompass all those involved, making awareness crucial for this process. Thus, organizational management can continuously improve, as long as it considers the needs of those involved, showcasing the implementation of improvements, generating transformation.

Using the results from the institutional self-assessment process is effective due to several factors, including the creation of an evaluative culture (SAIS, 2017) in the academic community; satisfactory communication during the process (ARRUDA; PASCHOAL; DEMO, 2019); promotion of freedom of expression (ANDRIOLA; ARAÚJO, 2016); continuity so that these improvement practices are not isolated moments (BERNARDES; ROTHEN, 2016); and permission for a global and systemic view (ANGST; ALVES, 2018) of institutional actions.

The discourse of managers uncovers a conception of formative self-assessment and provides opportunities for institutional improvement from their results. The actions practiced by managers do not confirm the effectiveness of SINAES regarding the use of evaluation results to produce improvement actions that aim at increasing institutional quality. The bibliographic review showed the importance of self-assessment; the process, however, needs to mature in order to lead the promotion of the institution’s autonomy, in a reflective and transforming bias. Therefore, the result of this research corroborates portfolio studies that argues that the academic community needs to award new meaning to institutional self-assessment.

Research opportunities

In the fourth stage of ProKnow-C – analyses of the 62 documents with described guidelines – we perceived knowledge gaps within self-assessment in which new research flows can be established for the development of knowledge aimed at manager and decision-makers to help them formulate more effective implementation strategies. Along with the gaps, the guidelines for self-assessment identified in the literature can be considered when developing a systematic approach that aims to formalize processes and use results effectively. Table 2 shows possible research opportunities to address the identified gaps, which are considered in conjunction with the guidelines for the further development of a systematic approach.

Table 2
Research gaps and opportunities

	Gaps	Opportunities
PROCESS	- Lack of coordination between processes and agents involved	- Modeling of processes, in which the tasks, mode of operation, deadlines, and those responsible are evident
	- Lack of integration of ISA with institutional objectives and planning	- Integration of ISA with other institutional instruments, such as the IDP, allowing a systematic view of the institution
RESULTS	- Lack of reflection on the results	- Proposition of actions for those involved based on the analysis of results
	- Lack of qualitative results	- Use of indicators, not only quantitative, but also qualitative
	- Lack of clarity in the process that leads to inadequate results	- Use of the principles of the new public management, such as transparency, objectivity, and clarity
	- Punctual results, without monitoring improvements	- Insertion of a continuous flow of information in the work routine
EFFECTIVENESS	- Data obtained are often not essential in the demands of each academic segment	- Research on the expectations of each segment of the academic community
	- Insufficient impact of the evaluation on management	- Change of mental model, in which people see the value in the institutional self-assessment process
	- Main regulatory aspects in ISA

• Source: Elaborated by the authors.

Although this issue has been widely discussed, in practice, little has been done regarding models or systematics of institutional self-assessment that present the processes in a formal way, with a presentation of tasks, mode of operation, deadlines, and of those responsible. Process modeling helps in planning and monitoring actions. We also observed a difficulty in creating an evaluative culture in which both those responsible for conducting ISA and the overall academic community use the process as a work routine in a cycle of continuous improvement, directly impacting the value of the service provided to society. Thus, this article offers opportunities to discuss future interventions in the ISA process and contributes to filling some of the gaps regarding the use of evaluation results, offering practical and theoretical-academic contributions.

Based on the opportunities described in Table 2 and considering the proposition of a model of institutional self- assessment based on processes, results, and effectiveness, we suggest future studies focus on the application of two approaches: new public management or lean management. New public management emerged in the 1990s (HOOD, 1991) and uses the principles of transparency, objectivity, and clarity in the process. As with other market segments, education requires customer focus, responsive management, and greater production with less cost. We also suggest the use of concepts derived from lean management in the development of a model of institutional self-assessment that aims to improve work and develop people. According to the Lean Institute Brasil (2016), this approach can help in the management of processes since it can systematically eliminate waste and solve problems in people’s day-to-day lives. Lean thinking prioritizes the overall mentality of the system, guiding those involved to look at the internal factors of the organization’s development, giving preference to the efficient use of resources, guiding the maximization of the results and the involvement of teams in search of continuous improvement.

Final remarks

We can observe a change in behavior in the literature; ideas are currently more focused on issues related to human resources and the performance of organizations, while, in the 1980s, the evaluation had a control function. This study highlights the need to go beyond evaluations and monitoring of performance management, especially in complex organizations like universities, in which academics, technicians, teachers, and managers have goals. Gathering the information is easy; the application of the results, however, is challenging for management. Thus, there is no pattern that can be replicated successfully in this system, it is necessary to fit the environment and reality in which the organization operates.

Thus, this study analyzed the literature guidelines for self-assessment methods in higher education institutions, regarding processes, results, and effectiveness. The analysis of attributes already proposed for this process helps in the development of a best practice in this area. The use of both the new public management and the lean management approaches can achieve successful results and is indicated in this study. The fact that much of the information on self-assessment with little practical value is available may be linked to the obligation that SINAES imposes on HEI, while higher education institutions are adhering to the principle of “voluntary adhesion,” that is, it is allowed for the academic community to participate in this process.

The information collected shows that, in Brazil, social and geographical differences are prevalent, resulting in differences among HEI. This study corroborates Andrade (2014); although the process of self-assessment in higher education has potential and can be taken as an indication of advancing a culture of self-assessment, not enough space is given to debate the results of the evaluations, much less to planning new actions. This study is expected to promote the self-assessment process and this will lead to constant improvement.

Discussions on the evaluation process and its results, by the internal academic community, is essential for the development of management policies and institutional planning. This is an uninterrupted quality search procedure and requires predisposition for transformation. Distancing the process from the culture of change is unconceivable. Therefore, it must be a constant object of reflection, making it an instrument that generates valid and necessary information that supports the academic community in the planning and management of activities. Self-assessment must be formative, continuous, and permanent. Moreover, with the current scenario of higher education, it should impose important changes in the methods of operation and management of universities.

References

• ABREU JÚNIOR, Nelson de. Brazilian higher education assessment system(s). Cadernos Cedes, Campinas, v. 29, n. 78, p. 257-269, 2009.
• AMÂNCIO, Weslei Trevizan. Participação da comunidade na autoavaliação institucional em universidades da Argentina, Brasil e Paraguai. 2019. Dissertação (Mestrado em Ciências Sociais) – Universidade Estadual do Oeste do Paraná, Toledo, 2019.
• ANDRADE, Nelson Lambert de. Autoavaliação a partir da ótica dos gestores de uma instituição de ensino superior. 2014. Tese (Doutorado em Educação) – Pontifícia Universidade Católica de São Paulo, São Paulo, 2014.
• ANDRIOLA, Wagner Bandeira; ARAÚJO, Adriana Castro. Relevance of using management indicators for self-assessment and strategic planning at higher education institutions. Meta, Rio de Janeiro, v. 8, n. 24, p. 515-533, 2016.
• ANGST, Felipe André; ALVES, José Matias. Higher education quality evaluation in Mozambique: a case study. Revista Portuguesa de Investigação Educacional, Lisboa, n. esp., p. 117-151, 2018.
• AOKI, Fabiano Kenji. Processo de autoavaliação institucional: produção de um manual para apoio à Comissão Própria de Avaliação. 2017. Dissertação (Mestrado em Educação e Tecnologias Aplicadas em Instituições Educacionais) – Universidade Federal dos Vales do Jequitinhonha e Mucuri, Diamantina, 2017.
• ARRUDA, June Alves de; PASCHOAL, Tatiane; DEMO, Gisela. Uso dos resultados da autoavaliação institucional pelos gestores da Universidade de Brasília. Avaliação, Campinas, v. 24, n. 3, p. 680-698, 2019.
• BALDIGEN, Francine Adriane. Maturidade e inovação na avaliação institucional nas instituições de ensino superior do setor público. 2018. Dissertação (Mestrado em Engenharia de Produção) – Escola de Engenharia, Universidade Federal do Rio Grande do Sul, Porto Alegre, 2018.
• BALZAN, Newton Cesar; DIAS SOBRINHO, José (org.). Avaliação institucional: teoria e experiências. São Paulo: Cortez, 1995.
• BARREYRO, Gladys Beatriz; ROTHEN, José Carlos. Contradictory “SINAES”: considerations on the elaboration and implantation of the National Higher Education Evaluation System. Educação & Sociedade, Campinas, v. 27, n. 96, p. 955-977, 2006.
• BERNARDES, Joelma dos Santos; ROTHEN, José Carlos. The field of higher education evaluation: focus on the Evaluation Committee. Meta, Rio de Janeiro, v. 8, n. 23, p. 248-277, 2016.
• BOTELHO, Arlete de Freitas. Intencionalidades e efeitos da autoavaliação institucional na gestão de uma universidade multicampi. 2016. Tese (Doutorado em Educação) – Universidade de Brasília, Brasília, DF, 2016.
• BRASIL. Lei nº 10.861/05, de 14 de abril de 2004. Institui o Sistema Nacional de Avaliação da Educação Superior – Sinaes e dá outras providências. Diário Oficial da União, Brasília, DF, Seção 1, v. 141, n. 72, p. 3-4, 15 abr. 2004.
• BRITO, Sandra Lopes Estrela. Modelo conceitual de avaliação institucional de universidades: a contribuição da avaliação institucional de universidades para o desenvolvimento socioeconômico de um país – um estudo da realidade de Moçambique. 2006. Tese (Doutorado em Engenharia de Produção) – Universidade Federal de Santa Catarina, Florianópolis, 2006.
• CAMPOS, Valter Gomes. As relações e os desdobramentos da dupla avaliação (CEE/GO e Sinaes) na administração central da Universidade Estadual de Goiás. 2019. Tese (Doutorado) – Faculdade de Educação, Universidade Federal de Goiás, Goiânia, 2019.
• CARDOSO, António Pedro Barbosa. Políticas de avaliação institucional da educação superior: criação e implementação do sistema de avaliação do ensino superior de Cabo Verde. 2017. Tese (Doutorado em Educação) – Pontifícia Universidade Católica do Rio Grande do Sul, Porto Alegre, 2017.
• CARVALHAES, Júlia Flávia Araújo. Comissões próprias de avaliação de universidades do sudeste brasileiro: o que dizem os documentos e o que ocorre na prática. 2018. Dissertação (Mestrado em Educação) – Faculdade de Educação, Universidade Federal de Minas Gerais, Belo Horizonte, 2018.
• CARVALHO, Hilda Alberton; OLIVEIRA, Oséias Santos de; LIMA, Isaura Alberton de. Avaliação institucional em uma universidade pública brasileira multicâmpus: processos e desafios na qualificação da gestão. Avaliação, Campinas; Sorocaba, v. 23, n. 1, p. 217-243, 2018.
• CASTELLANELLI, Luise Medina Cunha. A interação entre avaliação interna e planejamento estratégico na Universidade Federal de Santa Maria. 2018. Dissertação (Mestrado em Gestão de Organizações Públicas) – Centro de Ciências Sociais e Humanas, Universidade Federal de Santa Maria, Santa Maria, 2018.
• CHINTA, Ravi; KEBRITCHI, Mansureh; ELLIAS, Janelle. A conceptual framework for evaluating higher education institutions. International Journal of Educational Management, Bingley, v. 30, n. 6, p. 989-1002, 2016.
• CUNHA, Débora Alfaia da. Avaliação da educação superior: condições, processos e efeitos da autoavaliação nos cursos de graduação da UFPA. 2010. Tese (Doutorado em Educação) – Faculdade de Educação, Universidade de Brasília, Brasília, DF, 2010.
• DIAS, Tereza Cristina. Auto-avaliação institucional no ensino superior: uma análise comparativa do processo realizado em uma instituição pública e em uma instituição privada. 2007. Dissertação (Mestrado) – Universidade Federal de Viçosa, Viçosa, 2007.
• DIAS SOBRINHO, José. Avaliação educativa: produção de sentidos com valor de formação. Avaliação, Campinas, v. 13, n. 1, p. 193-207, mar. 2008.
• DONATO, Jânio Oliveira; LADEIA, James. Avaliação institucional: uma nova proposta de avaliação interna e externa. Revista Jus Navigandi, Teresina, v. 24, n. 5946, 2019. Disponível em: https://jus.com.br/artigos/65883 Acesso em: 30 jun. 2022.
  » https://jus.com.br/artigos/65883
• ENSSLIN, Sandra Rolim; ENSSLIN, Leonardo; DUTRA, Ademar. PROKNOW-C: um processo para geração de conhecimento e identificação de oportunidades de pesquisa científica. Florianópolis: UFSC, 2015. Apostila da disciplina Avaliação de Desempenho do Programa de Pós-Graduação em Engenharia de Produção da Universidade Federal de Santa Catarina.
• GONÇALVES, Lukelly Fernanda Amaral. A autoavaliação na Universidade de Brasília: entre a proposta do Sinaes e os sinais da prática. 2016. Dissertação (Mestrado em Educação) – Faculdade de Educação, Universidade de Brasília, Brasília, DF, 2016.
• GONÇALVES FILHO, Francisco. A avaliação institucional na Universidade Federal do Tocantins (2004-2010). 2016. Tese (Doutorado) – Universidade Federal do Pará, Belém, 2016.
• GOULART, Joana Corrêa. Avaliação institucional interna da Universidade Estadual de Goiás: um estudo meta-avaliativo. 2018. Tese (Doutorado em Educação Escolar) – Universidade Estadual Paulista Júlio de Mesquita Filho, Araraquara, 2018.
• HAMILL, Sharon Boland. Evaluating and redesigning a college assessment system to close the loop. Journal of Assessment and Institutional Effectiveness, State College, v. 5, n. 1, p. 34-57, 2015.
• HOLZWEISS, Peggy C.; BUSTAMANTE, Rebecca; FULLER, Matthew B. Institutional cultures of assessment: a qualitative study of administrator perspectives. Journal of Assessment and Institutional Effectiveness, State College, v. 6, n. 1, p. 1-27, 2016.
• HOOD, Christopher. A public management for all seasons? Public Administration, Hoboken, v. 69, n. 1, p. 3-19, 1991.
• KWECKO, Fabio Rios. Os desafios do diálogo entre autoavaliação institucional e gestão estratégica do IFRS – Campus Rio Grande. 2017. Dissertação (Mestrado Profissional) – Universidade do Vale do Rio dos Sinos, Porto Alegre, 2017.
• LEAN INSTITUTE BRASIL. Definição e aplicações. Lean Institute Brasil, São Paulo, 2016. Disponível em: https://www.lean.org.br/o-que-e-lean.aspx Acesso em: 18 ago. 2020.
  » https://www.lean.org.br/o-que-e-lean.aspx
• LEHFELD, Neide Aparecida de Souza et al. Evaluation processes and the quality of Brazilian higher education: a case study. Espacios, Caracas, v. 38, n. 30, p. 4, 2017.
• LIMA, Cláudia Ibiapina. Autoavaliação nas instituições de ensino superior (IES) do Ceará sob a égide do Sistema Nacional de Avaliação da Educação Superior (Sinaes). 2010. Dissertação (Mestrado em Educação) – Faculdade de Educação, Universidade Federal do Ceará, Fortaleza, 2010.
• LIMA, Jéssyka Pereira de. Auto avaliação e gestão institucional: análise da experiência do Instituto Federal da Paraíba. 2018. Dissertação (Mestrado em Gestão Pública) – Centro de Ciências Sociais Aplicadas, Universidade Federal do Rio Grande do Norte, Natal, 2018.
• LIMA, Lucinete Marques. O processo de auto-avaliação da UFMA (2004-2006) no contexto regulatório da educação superior. 2011. Tese (Doutorado em Educação) – Faculdade de Filosofia e Ciências, Universidade Estadual Paulista Júlio de Mesquita Filho, Marília, 2011.
• LINS, Juliana Medeiros de Omena et al. Institutional evaluation: a case study in a particular heir in Brazil. Espacios, Caracas, v. 38, n. 30, p. 20, 2017.
• LOPES, Fernando Dias. Elementos para um modelo integrado de planejamento e avaliação institucional na universidade. 1994. Dissertação (Mestrado em Administração) – Centro Socioeconômico, Universidade Federal de Santa Catarina, Florianópolis, 1994.
• MARTINS, Francisco Miguel. Autoavaliação institucional da educação superior: uma experiência brasileira e suas implicações para a educação superior de Timor-Leste. 2010. Tese (Doutorado em Educação) – Faculdade de Educação, Universidade Federal da Bahia, Salvador, 2010.
• MATOSKI JÚNIOR, Arivonil dos Santos. Políticas de avaliação institucional: regulamentação e/ou emancipação na auto-avaliação institucional. 2008. Dissertação (Mestrado em Educação) – Pontifícia Universidade Católica do Paraná, Curitiba, 2008.
• MENEZES, Ângela Maria de. Autoavaliação como instrumento de gestão na educação superior: o caso do Instituto Federal de Educação, Ciência e Tecnologia de Goiás – IFG. 2012. Dissertação (Mestrado em Educação) – Faculdade de Educação, Universidade de Brasília, Brasília, DF, 2012.
• NARDELLI, Nelci Janete dos Santos. Autoavaliação institucional e processos de identidade no horizonte da cultura avaliativa: estudo de caso em uma instituição estadual de educação superior. 2019. Tese (Doutorado em Ciências Sociais) – Universidade do Vale do Rio dos Sinos, São Leopoldo, 2019.
• NUNES, Enedina Betânia Leite de Lucena Pires; DUARTE, Michelle Matilde Semigueem Lima Trombini; PEREIRA, Isabel Cristina Auler. Planejamento e avaliação institucional: um indicador do instrumento de avaliação do Sinaes. Avaliação, Campinas, v. 22, n. 2, p. 373-384, 2017.
• OECD. Organisation for Economic Co-operation and Development. About. Paris: OECD, 2011. Disponível em: https://www.oecd.org/about/ Acesso em: 10 ago. 2020.
  » https://www.oecd.org/about/
• OLIVEIRA, Lucilene Rebouças de. Estudo do processo de avaliação interna: ferramenta de tomada de decisão, de autocontrole, de autoconhecimento e de melhoria na gestão das universidades e institutos federais. 2013. Dissertação (Mestrado em Engenharia de Produção) – Faculdade de Tecnologia, Universidade Federal do Amazonas, Manaus, 2013.
• PEIXOTO, Maria do Carmo de Lacerda. A avaliação institucional nas universidades federais e as comissões próprias de avaliação. Avaliação, Campinas, v. 14, n. 1, p. 9-28, 2009.
• PINHEIRO, Any Kadidja de Melo Tavares. Avaliação institucional da UFRN: viabilidade e utilização. 2018. Tese (Doutorado em Ciências Sociais) – Centro de Ciências Humanas, Letras e Artes, Universidade Federal do Rio Grande do Norte, Natal, 2018.
• PRODANOV, Cleber Cristiano; FREITAS, Ernani Cesar de. Metodologia do trabalho científico: métodos e técnicas da pesquisa e do trabalho acadêmico. 2. ed. Novo Hamburgo: Feevale, 2013.
• RIBEIRO, Elisa Antonia. O processo de autoavaliação institucional proposto no Sistema Nacional de Avaliação do Ensino Superior (Sinaes) para as instituições públicas e privadas. 2010. Tese (Doutorado em Educação) – Universidade Federal de Uberlândia, Uberlândia, 2010.
• RIBEIRO, Jorge Luiz Lordêlo de Sales. Sinaes: o que aprendemos acerca do modelo adotado para avaliação do ensino superior no Brasil. Avaliação, Campinas, v. 20, n. 1, p. 143-161, 2015.
• RODRIGUES, Cláudia Medianeira Cruz. Proposta de avaliação integrada ao planejamento anual: um modelo para as UCGs. 2003. Tese (Doutorado em Engenharia de Produção) – Escola de Engenharia, Universidade Federal do Rio Grande do Sul, Porto Alegre, 2003.
• ROSA, Maria João et al. The EUA institutional evaluation programme: an account of institutional best practice. Quality in Higher Education, Abingdon, v. 17, n. 3, p. 369-386, 2011.
• ROSAS, Ana Karolina Ramalho de Araújo. Sinaes e os procedimentos e avaliação: a construção de uma nova cultura na universidade? 2014. Dissertação (Mestrado em Sociologia) – Centro de Ciências Humanas e Letras, Universidade Federal da Paraíba, João Pessoa, 2014.
• SAIS, Rafael Martins. Avaliação institucional numa universidade multicampi: diálogos, tensões e interfaces com o planejamento. 2017. Dissertação (Mestrado Profissional) – Universidade do Vale do Rio dos Sinos, São Leopoldo, 2017.
• SANTOS, Sérgio Machado dos. Comparative analysis of european processes for the assessment and certification of internal quality assurance systems. Lisboa: Agência de Avaliação e Acreditação do Ensino Superior (A3ES), 2011.
• SCRIVEN, Michael. The final synthesis. Evaluation Practice, Thousand Oaks, v. 15, n. 3, p. 367-382, 1994.
• SCRIVEN, Michael. The methodology of evaluation. In: TYLER, Ralph Winfred; GAGNÉ, Robert Mills; SCRIVEN, Michael. Perspectives of curriculum evaluation. Chicago: Rand Mcnally, 1967. p. 39-83.
• SILVA, Antonio Lopes Nogueira da et al. Critical factors in the participation of institutional evaluation in a Salesian higher education institution. Brazilian Journal of Business, Curitiba, v. 1, n. 3, p. 785-804, 2019.
• SILVA, Assis Leão da. Avaliação institucional no Sistema Nacional de Avaliação da Educação Superior (Sinaes). 2015. Tese (Doutorado em Educação) – Centro de Educação, Universidade Federal de Pernambuco, Recife, 2015.
• SILVA, Juliano Reginaldo Corrêa da et al. The incidence of isomorphic mechanisms in the processes of institutional self-evalution. Navus, Florianópolis, v. 9, n. 3, p. 173-186, 2019.
• SOARES, Antonio Augusto dos Santos. Gestão da qualidade no ensino superior: desafios para os modelos de regulação e avaliação no Brasil e em Portugal. 2016. Tese (Doutorado em Gestão) – Universidade de Trás-os-Montes e Alto Douro, Vila Real, 2016.
• SOUZA, Francisco Antônio de Araújo e. Avaliação e planejamento para o desenvolvimento institucional de universidades. 2010. Tese (Doutorado em Educação) – Faculdade de Educação, Universidade Federal do Ceará, Fortaleza, 2010.
• VENTURINI, Jonas Cardona et al. Percepção da avaliação: um retrato da gestão pública em uma instituição de ensino superior (IES). Revista de Administração Pública, Rio de Janeiro, v. 44, n. 1, p. 31-53, 2010.
• VIEIRA, Regina Lúcia Bastos; FREITAS, Kátia Siqueira de. Sinaes at the public state university: analysis of the internal evaluation process at Bahia State University (UNEB). Ensaio, Rio de Janeiro, v. 18, n. 68, p. 443-464, 2010.

Edited by

Editor: Márcia Aparecida Jacomini

Francisca Goedert Heiderscheidt is part of the public servants of the Federal University of Santa Catarina, within the Rectory and focuses on Institutional Assessment. In parallel, she is a PhD student in Production Engineering and develops research under the Lean approach applied to public services.

Fernando Antônio Forcellini holds a PhD in Production Engineering and is currently Full Professor at Federal University of Santa Catarina. He participated in the preparation of books and articles in the areas of development, improvement, and management of processes, products, and services.

Publication Dates

• Publication in this collection
  06 Mar 2023
• Date of issue
  2023

History

• Received
  19 Feb 2021
• Reviewed
  22 June 2021
• Accepted
  02 Aug 2021