Evaluation of virtual external higher education and the impacts of the Covid-19 pandemicAbstract
This art. addresses the external evaluation of Higher Education conducted virtually by the National Institute of Educational Studies and Research Anísio Teixeira (INEP) in undergraduate and technological courses offered by private and community educational institutions, as a basis for issuing authorizing acts and the impacts caused by the Covid-19 pandemic in the adoption of this evaluation modality. The functions of the regulatory State and its authorizing acts, which are subject to compliance with national standards and quality assessment, are highlighted. The National System for the Evaluation of Higher Education defines methods for assessing the quality of institutions and higher education courses. The pandemic brought changes to people's lives and work relationships, also affecting external evaluations, which shifted from in-person (in loco) to virtual modalities, and these changes persisted. This is bibliographic research that analyzes the legislative changes introduced in the regulation of virtual external evaluations and the theoretical framework used, identifying the positive and negative aspects of the new evaluation modality based on a survey conducted with Basis evaluators. The conclusions are the result of the interpretation of legislative changes, the theoretical framework, and evaluators' opinions, allowing, through deductive reasoning, the identification and demonstration of the positive and negative aspects of the new evaluation modality. The research outcome enabled the presentation of a practical proposal for adopting a new evaluation system: a hybrid in loco external evaluation system, which better addresses the necessary balance in applying national standards that aim to ensure education quality and authorizing acts.
Keywords:
evaluation; virtual; impacts
Resumo
O artigo trata da avaliação externa da Educação Superior, realizada de modo virtual pelo Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira nos cursos superiores de graduação e tecnológicos ofertados em instituições privadas e comunitárias de ensino, como insumo para a edição de atos autorizativos e dos impactos gerados pela pandemia da Covid-19 na adoção dessa modalidade de avaliação. São evidenciadas as funções do Estado regulador e seus atos autorizativos, que estão sujeitos à obediência às normas nacionais e à avaliação da qualidade. O Sistema Nacional de Avaliação da Educação Superior prevê as formas de aferição da qualidade das instituições e dos cursos superiores. A pandemia trouxe mudanças na vida das pessoas e nas relações de trabalho, alcançando também a avaliação externa, que passou de presencial (in loco) para o virtual, sendo que tais mudanças permaneceram. Trata-se de uma pesquisa bibliográfica no que diz respeito a análise das modificações introduzidas na legislação regulamentadora das avaliações externas virtuais e no aporte teórico utilizado, no que diz respeito a identificação dos aspectos positivos e negativos da nova modalidade de avaliação, a base advém de uma pesquisa de opinião obtida com avaliadores do Basis. As conclusões são resultado da interpretação das modificações legislativas, do aporte teórico e do resultado das opiniões dos avaliadores, possibilitando pelo modo dedutivo, identificar e demonstrar os aspectos positivos e negativos da nova modalidade avaliativa. O resultado da pesquisa viabilizou como contribuição, a apresentação de uma proposta prática de adoção de um novo sistema de avaliação. O sistema híbrido de avaliação externa in loco, o qual contempla de uma forma mais adequada as necessárias relações de equilíbrio na aplicação das normas nacionais que visam assegurar a qualidade da educação e dos atos autorizativos.
Palavras-chave:
avaliação; virtual; impactos
Resumen
El artículo aborda la evaluación externa de la Educación Superior realizada de manera virtual por el Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira en educación superior y carreras tecnológicas ofrecidas en instituciones educativas privadas y comunitarias, como insumo para la edición de actos de autorización y los impactos generados por la Pandemia Covid-19. Se destacan las funciones del Estado regulador y sus actos autorizantes, los cuales están sujetos al cumplimiento de las normas nacionales y a la evaluación de la calidad. El Sistema Nacional de Evaluación de la Educación Superior proporciona formas de medir la calidad de las instituciones y los cursos de educación superior. La pandemia provocó cambios en la vida de las personas y en las relaciones laborales, afectando también la evaluación externa, que pasó de presencial a virtual, y dichos cambios se mantuvieron. La investigación bibliográfica y de opinión explora los cambios legales, aspectos teóricos y prácticos del nuevo sistema de evaluación virtual, identificando y demostrando los aspectos positivos y negativos, culminando con la presentación de una propuesta para adoptar un sistema híbrido de evaluación externa presencial que alterne la modalidad virtual y presencial para los diferentes actos como una mejor manera de lograr una relación equilibrada en la verificación del cumplimiento de los estándares nacionales y la calidad de la educación y autorizando actos.
Palabras clave:
evaluación; virtual; impactos
1 Introductory consideration
The objective of this text is to analyze the external evaluation of Higher Education conducted virtually by evaluators from the National Institute of Educational Studies and Research Anísio Teixeira (INEP) and the impacts generated by the Covid-19 pandemic, stemming from the measures implemented to change the evaluation processes.
The Brazilian Constitution of 1988 established that education is not an exclusive public service provided by the State. In addressing it, art. 209 defines that teaching is free for private initiative, provided that the conditions established by law are met. There are three indispensable conditions for the provision of education by private entities: a) compliance with general national education regulations; b) authorization; and c) quality evaluation by the Public Power (Brasil, 1988).
To fulfill its constitutional and legal purposes, the National System for the Evaluation of Higher Education (Sinaes) defined the following evaluation processes: I - Internal evaluation of institutions; II - External in loco evaluation of institutions conducted by INEP; III - Evaluation of undergraduate courses; and IV - Evaluation of the academic performance of undergraduate students, the National Student Performance Exam (ENADE).
The Covid-19 pandemic had significant impacts on society as a whole. Many paradigms, such as public health, education, economy, personal interactions, state policies, beliefs, and dogmas, were affected by the consequences of the imposed social isolation. The shock was cruel in reality, highlighting all human and institutional vulnerabilities that were previously thought to be unshakable.
In this context, external in loco evaluations were also impacted by the pandemic, leading to legislative changes that established a new way of conducting them, shifting from face-to-face to virtual (remote) modalities. These changes have led to different interpretations, some positive and others negative. The text analyzes these changes to understand the impacts on in loco evaluation processes and proposes balanced alternative approaches to the system.
2 The regulatory role of the state in the provision of Higher Education
Education plays a crucial and fundamental role, being indispensable for preparing individuals for the workforce, especially in a society where knowledge and intellectual preparation are imperative elements, even for simpler tasks that are not predominantly intellectual.
For this reason, the State has the duty to provide minimum material conditions for individuals to acquire qualifications for a job and human development. Where and when the State does not provide these conditions, it must enable the necessary conditions for qualification to be offered. Furthermore, education is inherent to contemporary societies, marked by dynamism and innovation, which impose new challenges on workers daily. Therefore, education should serve as a permanent tool for every individual’s improvement.
Law no. 9.394, of dec. 20, 1996, the National Education Guidelines and Framework Law (LDB), in art. 7, regulates art. 209 of the Federal Constitution, maintaining the same conditions for the provision of education by private entities. Under art. 9, responsibility for issuing general norms, authorizing, and evaluating education quality is assigned to the Union, which must:
VI- Ensure a national evaluation process of school performance in elementary, secondary, and higher education, in collaboration with education systems, aiming to define priorities and improve the quality of teaching;
VII- Establish general standards for undergraduate and postgraduate courses; IX- Authorize, recognize, accredit, supervise, and evaluate, respectively, the courses of higher education institutions and establishments within their education system (Brasil, 1996, Art. 7).
When addressing Higher Education in chapter IV, specifically in art. 46, the LDB establishes that "the authorization and recognition of courses, as well as the accreditation of Higher Education institutions, shall have limited terms, being renewed periodically after a regular evaluation process" (Brasil, 1996, art. 46).
Decree no. 9.235/2017, of december 15 2017, which regulates "the exercise of regulation, supervision, and evaluation functions for higher education institutions and undergraduate and postgraduate courses in the federal education system" (Brasil, 2017b, art. 1), establishes in art. 3 that the competencies for the functions of regulation, supervision, and evaluation in the federal education system shall be exercised by the Ministry of Education (MEC), the National Education Council (CNE), INEP, and the National Higher Education Evaluation Commission (CONAES).
The competencies for regulation, supervision, and evaluation functions in the federal education system shall be carried out under the responsibility of the Ministry of Education (art. 4), the CNE (art. 6), INEP (art. 7), and CONAES (art. 8) of Decree no. 9.235/2017 (Brasil, 2017b), with amendments from Decree no. 11.691/2023 (Brasil, 2023).
The MEC is responsible for homologating CNE opinions on accreditation, reaccreditation, and accreditation requests, homologating opinions and proposals for normative acts approved by it, approving evaluation instruments produced by INEP, homologating CONAES deliberations, and issuing norms and instructions for implementing legal norms. Once homologated, these acts are final in the administrative sphere.
The Department of Regulation and Supervision of Higher Education (SERES) is responsible, among other activities, for planning and coordinating policy formulation processes for higher education regulation and supervision, in accordance with the National Education Plan (PNE); authorizing, recognizing, and renewing recognition of undergraduate and sequential courses, both in-person and distance learning; issuing opinions on accreditation and re-accreditation processes for higher education institutions in both in person and distance learning modalities; supervising higher education institutions and courses to ensure compliance with the law; establishing guidelines and instruments for regulation and supervision actions; establishing guidelines for developing institutional and course evaluation instruments; and coordinating the certification policy for charitable social assistance organizations in education.
The CNE is responsible for exercising normative, deliberative, and advisory functions to the Minister of Education on matters related to the regulation and supervision of Higher Education. Through the Higher Education Chamber, it deliberates on accreditation, re-accreditation, and de-accreditation requests for Higher Education Institutions (HEIs) and on authorizing the offering of courses linked to accreditations; proposes guidelines and deliberates on the development of evaluation instruments for accreditation and reaccreditation of institutions to be developed by INEP; and recommends, when the quality standard is not met, the accreditation and reaccreditation of institutions.
Meanwhile, INEP is responsible for designing, planning, coordinating, and operationalizing actions aimed at evaluating HEIs, undergraduate courses, and government schools; ENADE, exams, and evaluations of undergraduate students; designing, planning, coordinating, operationalizing, and evaluating indicators related to Higher Education from exams and data from official databases, in accordance with current legislation; establishing and maintaining a pool of evaluators and specialized collaborators, including appointing evaluation commissions; developing and submitting external in loco evaluation instruments for MEC approval, in accordance with guidelines proposed by the Department of Regulation and Supervision of Higher Education and other MEC bodies; designing, planning, evaluating, and updating indicators for these in loco instruments in line with the guidelines proposed by the Department of Regulation and Supervision of Higher Education of the Ministry of Education.
CONAES is responsible for proposing and evaluating the dynamics, procedures, and mechanisms for institutional evaluation, course evaluation, and student performance; establishing guidelines for organizing evaluation commissions, analyzing reports, drafting opinions, and forwarding recommendations to competent authorities; formulating proposals for the development of HEIs based on analyses and recommendations from the evaluation processes; collaborating with state education systems to establish common actions and criteria for higher education evaluation and supervision; and annually submitting to MEC for approval the list of courses whose students will take the ENADE exam.
Decree no. 9.057, of may 25, 2017, regulates art. 80 of Law no. 9.394/1996, which addresses distance education programs at different levels and teaching modalities. In its first art., it defines distance education as being:
The educational modality in which didactic-pedagogical mediation in teaching and learning processes occurs through the use of information and communication technologies, with qualified personnel, access policies, appropriate monitoring and evaluation, among others, and that carries out educational activities by students and education professionals who are in different places and times (Brasil, 2017a, art. 1).
Just like in-person courses, distance learning courses will have their "creation, organization, offering, and development in compliance with current legislation and the specific regulations issued by the Ministry of Education" (Brasil, 2017a, art. 3), under the responsibility of the Ministry of Education (MEC), in collaboration with the relevant bodies and entities:
I - the accreditation and re-accreditation of educational institutions within the federal, state, and district education systems for offering higher education in the distance learning modality; and
II - the authorization, recognition, and renewal of recognition of higher education courses in the distance learning modality offered by educational institutions within the federal education system, respecting the prerogatives of autonomy (Brasil, 2017a, art. 6).
According to art. 13 of the aforementioned Decree:
The processes of institutional accreditation and re-accreditation, authorization, recognition, and renewal of recognition of higher education courses in the distance learning modality shall undergo in loco evaluation at the institution’s headquarters, with the purpose of verifying the existence and adequacy of methodology, physical, technological, and human infrastructure, which enable the execution of the activities outlined in the Institutional Development Plan and the Course Pedagogical Project (Brasil, 2017a, art. 13).
Normative Ordinance no. 11, of june 20, 2017, "establishes rules for the accreditation of institutions and the offering of distance higher education courses, in accordance with Decree no. 9.057/2017" (Brasil, 2017c). In its art. 1, it states that "the operation of Higher Education Institutions, for the offering of distance higher education courses, depends on specific accreditation by the MEC, under the terms of art. 80 of Law no. 9.394, of 1996, and Decree no. 9.057, of 2017" (Brasil, 2017c, art. 1).
Regarding in loco evaluation, art. 5 defines that the processes will take place at the Institution’s headquarters, with the objective of “Verifying the existence and adequacy of methodologies, physical, technological, and human infrastructure that enable the execution of the activities outlined in the Institutional Development Plan (PDI) and the Course Pedagogical Project (PPC)” (Brasil, 2017c, art. 5, § 1º).
Verifications should be carried out through documentary means or with the use of technological resources provided by the HEIs, for the Distance Education Centers (EaD) outlined in the PDI and PPC, as well as the professional environments used for supervised internships and in-person activities (Brasil, 2017c, art. 5, § 2º).
Higher Education Institutions, as previously mentioned, depend on an authorizing act issued by the Union, through the Ministry of Education, for their operation. However, for the offering of undergraduate and technological higher education courses, when it comes to colleges, prior authorization from MEC will always be required (Brasil, 2017b, art. 39). In contrast, Universities and University Centers, under their autonomy, do not require prior authorizing acts. Nevertheless, institutions must inform the Department of Regulation for evaluation and subsequent recognition of courses (Brasil, 2017b, art. 40).
On the other hand:
The offering of undergraduate courses in Law, Medicine, Dentistry, Psychology, and Nursing, including in Universities and University Centers, depends on authorization from the MEC, following prior consultation with the Federal Council of the Brazilian Bar Association (OAB) and the National Health Councils (Brasil, 2017b, art. 41).
The State, in regulating private activities, aims to ensure social well-being, as noted by Mello (2015, p. 707):
[...] thus, the Administration is tasked with carrying out certain activities intended to ensure that private actions align with legal requirements, which entails the practice of preventive, supervisory, and sometimes repressive actions.
According to Art. 10 of Decree no. 9.235/2017, the operation of institutions and the offering of higher education courses depend on an authorizing act from the MEC, listing in § 1 the types of authorizing acts to be issued, and establishing that “the validity periods of the authorizing acts shall be included in the acts and counted from the date of publication” (Brasil, 2017b, art. 10, § 3). Furthermore, it adds that “authorizing acts shall be renewed periodically, in accordance with art. 46 of Law no. 9.394, of 1996, and the process may be simplified based on the evaluation results, as regulated by the MEC” (Brasil, 2017b, art. 10, § 4º).
Therefore, for the provision of Higher Education in the country by private and community institutions, the existence of an Authorizing Act for accreditation, re-accreditation, and renewal of re-accreditation of institutions is necessary. For the offering of undergraduate and technological courses, the issuance of this act of authorization, recognition, and renewal of recognition is essential.
These respective authorizing acts depend on the compliance with national general education standards and quality evaluation. To verify this, Sinaes was created by the Union, which provides the basic inputs that serve as evidence for the regulatory process. It contributes to institutions and society by generating important information.
2.2 National System for the Evaluation of Higher Education (Sinaes)
The National System for the Evaluation of Higher Education (Sinaes) was established by Federal Law no. 10.861, of April 14, 2004, with the objective of "ensuring a national evaluation process for higher education institutions, undergraduate courses, and the academic performance of their students" (Brasil, 2004a). MEC Ordinance no. 2.051, of July 9, 2004, regulates Sinaes.
The purposes of Sinaes are: improving the quality of Higher Education; guiding the expansion offer; continuously increasing institutional efficiency; promoting academic and social effectiveness; and deepening the commitments and social responsibilities of HEIs through the enhancement of their public mission, promotion of democratic values, respect for difference and diversity, affirmation of institutional autonomy and identity (Brasil, 2004a, art. 1, § 1). According to Cavalcanti and Guerra (2022, p. 515), as a public policy:
[...] it was created to organize and operationalize the evaluation through an articulated process of all higher education evaluations, namely: Institutional Evaluation of HEIs (AI), Evaluation of Undergraduate Courses, and Student Performance Evaluation (ENADE), integrating methodologies, spaces, and evaluation and information instruments, aimed at the goal of identifying merit and value - of institutions, areas, courses, and programs - in the dimensions of teaching, research, extension, management, and training, to successfully fulfill the goal of offering inclusive, high-quality higher education.
Verhine (2015, p. 616) affirms that Sinaes aims to provide relevant information to the Public Power, which is “Scientifically valid, reliable, and comparable over time, in processes conducted by specialists who operate free from external interference that may compromise the quality of the knowledge produced”.
Sinaes, by establishing a three-pronged evaluation approach, the Regulatory Decree (Brasil, 2004a, art. 80), composed of internal and external institutional evaluation and ENADE, seeks to conduct a comprehensive and integrated analysis of the various dimensions of higher education institutions and their courses, aiming to meet social purposes and responsibilities, contributing to the government's accountability to society for the significant public investments made in Higher Education.
During the occurrence of the Covid-19 pandemic, numerous changes occurred in people's daily lives, impacting not only personal relationships but also the ways work and activities were carried out. External in loco evaluations were also affected, and the methods of their execution were redefined, continuing even after the pandemic ended.
3 The Covid-19 pandemic
The text addresses the impacts of the Covid-19 pandemic on daily life, as it profoundly altered people's routines and work activities. These effects transcended health and psychological aspects, significantly affecting the work structure and organizational processes in various areas.
In this context, organizations had to adapt their practices to ensure the continuity of operations and the achievement of their objectives. Thus, remote work, supported by technology, gained relevance as an essential facilitator for the new daily reality of interpersonal and professional relationships. According to Bollela, Medeiros, and Telles (2021, p. 8), "technology emerges in this scenario as the main protagonist, that is, as a resource that enables communication between people, whether in the personal, professional, or academic spheres, regardless of distance”.
Regarding external in-person evaluations conducted by INEP, just as private organizations had to adapt to a new way of life and work imposed by the pandemic, to ensure the continuity of evaluative work and prevent greater losses for institutions and higher education programs, external evaluations were implemented in a virtual format, utilizing technology.
According to the World Health Organization (WHO, 2020):
A pandemic is the worldwide spread of a new disease, and the term is used when an epidemic, an outbreak affecting a specific region, spreads across different continents with sustained person-to-person transmission
For the Butantan Institute, it is defined as:
A disease becomes a pandemic when it reaches global levels, that is, when a certain agent spreads to various countries or continents, usually affecting a large number of people. It is the World Health Organization (WHO) that defines when a disease becomes this type of global threat. A pandemic can start as an outbreak or epidemic; in other words, outbreaks, pandemics, and epidemics have the same origin - the only difference is the scale of disease spread (2021).
The Ministry of Health (MS) defines Covid-19 in its official publications as:
An acute respiratory infection caused by the SARS-CoV-2 Coronavirus, which is potentially severe, highly transmissible, and globally widespread. SARS-CoV2 is a betacoronavirus discovered in bronchoalveolar lavage samples from patients with pneumonia of unknown origin in the city of Wuhan, Hubei province, China, in December 2019. It belongs to the sarbecovirus subgenus of the coronaviridae family and is the seventh coronavirus known to infect humans. Coronaviruses are a large family of viruses common to many different species of animals, including humans, camels, cattle, cats, and bats. Animal coronaviruses rarely infect humans and then spread between people, as has occurred with MERS-CoV and SARS-CoV-2. So far, the wild reservoir of SARS-CoV-2 has not been identified (Brasil, 2021a).
The first reports and news about the Covid-19 pandemic date back to the end of 2019, after the causative virus was identified in the city of Wuhan, China. By early 2020, more precisely in March, the Covid-19 virus had spread worldwide, leading WHO to declare a pandemic.
Despite the existence of advanced technologies and the availability of information stored in various systems, it is difficult to measure the impacts caused by the Covid-19 pandemic on people and organizations in general. However, it seems certain and agreed upon by public authorities, researchers, and the population that the impacts on various sectors of economic activity, professional performance, daily routines, and behavior were highly significant.
In educational institutions in general, in-person classes were replaced by remote (online) classes due to the prohibition of student transportation and gatherings, as the State sought to prevent the spread of the disease. In this sense, INEP initially suspended external in loco evaluations. Later, a few selective evaluations were allowed, following specific health guidelines, and eventually, virtual in loco evaluations were implemented to ensure the minimum continuity of activities, which also resulted in legislative changes.
However, from the perspective of Saviani (2023, p. 185), the post-pandemic period will bring pressures:
The post-pandemic period will bring pressures for the generalization of Distance Education as if it were equivalent to in person teaching. This deepens the trend of commodifying education in the wake of privatization, which always implies seeking cost reductions to increase profits.
According to Guerra and Cavalcanti (2021, p. 898), the pandemic has brought about changes that prevent a return to the pre-pandemic status quo:
The paradigms, beliefs, dogmas, education, public health, economy, labor relations, personal interactions and subjectivities, state policies, as well as the consequences and impacts imposed by social isolation, have caused a cruel reality check, exposing the fragility of human beings and institutions that were once believed to be unshakable.
In this understanding, or for other reasons, the tendency remains for INEP’s virtual external evaluations, which continued in the modality adopted during the pandemic period.
4 The virtual in loco evaluation implemented by INEP due to the COVID-19 pandemic
Among INEP's responsibilities are evaluations, educational exams, statistical research and educational indicators, knowledge management, and educational studies. Educational evaluations include on-site visits to higher education institutions and undergraduate and technological courses, contributing to the issuance of regulatory acts and ensuring the quality of this level of education.
The in loco evaluation at higher education institutions and undergraduate courses conducted by INEP aims to produce a basic informational reference that serves as a foundation for the decision making process regarding the regulation and supervision of Higher Education by the Ministry of Education (MEC). It is a necessary procedure to produce a basic reference that will support decisions within the regulatory process, essential for institutions to be accredited and/or reaccredited, and, in relation to higher and technological courses, it refers to the reference for them to be authorized, recognized, and have their recognition renewed.
The external in loco evaluation is guided by evaluation instruments (IAIE) or evaluation instruments for undergraduate and technological courses (IACG), with the aim of producing reports that reflect the reality of institutions and courses, constituting an important tool for supporting decision making regarding public policies, providing information to society, and contributing to the improvement of the quality of Brazilian Higher Education.
The on-site evaluation is carried out by committees of external evaluators appointed by INEP, from a pool of evaluators (BASIs), constituted under the terms of § 2º and art. 3º of Federal Law no. 10.870/2004 (Brasil, 2004b), taking into account the complexity and breadth of the course or institution to define the number of evaluators, which, depending on the number of qualifications in the course, can vary from two to five evaluators and, for an institution, from three to eight evaluators.
The reports produced by the in loco evaluation committees become essential evidence for MEC when issuing decision making acts and approving authorization acts for the authorization, recognition, renewal of recognition of higher courses, accreditation, reaccreditation, or transformation of higher institutions.
The COVID-19 pandemic brought numerous changes to people's way of life, forcing the implementation of actions, until then unimaginable, related to work and various human activities. Among the changes implemented at work, the external in loco evaluation conducted by INEP began to occur virtually, replacing in person evaluations.
The external in loco evaluation started in the virtual mode with the issuance of Ordinance no. 165, dated april 20, 2021 (Brasil, 2021b), issued by the president of INEP, Danilo Dupas Ribeiro, which “Institutes the virtual external in loco evaluation within the scope of visits by specialist committees for external evaluation of Higher Education Institutions and undergraduate courses, within the scope of Sinaes”.
Art. 1 of the aforementioned Ordinance establishes:
To institute the Virtual External in loco Evaluation to improve external evaluation visits to HEIs and undergraduate courses by a committee of specialists who are part of the National System of Higher Education Evaluation (BASis) or the Government Schools Evaluators' Pool.
§ 1º The Virtual External in loco Evaluation is defined as an evaluation environment in which new or innovative procedures may be implemented to improve and modernize external evaluation visits to HEIs and undergraduate courses. § 2º The Virtual External in loco Evaluation concerns the organization, monitoring, and supervision of evaluation visits conducted by evaluation committees in a technology-mediated format.
§ 3º The Virtual External in loco Evaluation is under the responsibility of the General Coordination of Evaluation of Undergraduate Courses and Higher Education Institutions (CGACGIES) of the Higher Education Evaluation Directorate (DAES) (Brasil, 2021b, art. 1).
As objectives to be achieved by the virtual external in loco evaluation, art. 3º of the Ordinance (Brasil, 2021b) listed:
The Virtual External in loco Evaluation will be implemented with the intensive use of Information and Communication Technologies (ICT), aiming to: I- Strengthen the organization of the evaluation, its monitoring and supervision, information security, the availability of evaluators, and the service to HEIs and undergraduate courses in the country;
II- Enable new forms of interaction between HEIs and evaluation committees synchronously, ensuring conditions for the faithful and detailed recording of evidence of educational provision, its inputs, and processes, by the committees;
III- Improve service to municipalities that are difficult to access or whose service is impaired due to air, road, waterway availability, geographic or weather conditions;
IV- Address logistical elements that affect the conduct of evaluations; V- Ensure the delivery of the evaluation report, educational data, and information to the stakeholders in the external evaluation results - Ministry of Education's Secretariat for Higher Education Regulation and Supervision (SERES/MEC) and HEIs;
VI- Maintain the process of external evaluation of HEIs and undergraduate courses even in local, regional, or national contingency scenarios, such as those caused by the spread of the new coronavirus;
VII- Incorporate new technologies for organizing external evaluation; VIII- Optimize the evaluators' dedication from the evaluator pools to interact with HEIs; e
IX- Increase the efficiency of visits conducted by committees (Brasil, 2021b, art. 3).
Ordinance no. 2.051/2004 (Brasil, 2004c), regulating the SINAES Law, in its art. 5, determines that "for external in loco evaluations, INEP shall designate: I- External Institutional Evaluation Committees; II- External Course Evaluation Committees". The external evaluation committees of institutions, according to art. 15, will assess the following information and documents:
I- The Institutional Development Plan (PDI); II- Partial and final reports of the self-evaluation process produced by the HEI according to the general guidelines provided by INEP; III- General and specific data of the HEI listed in the Higher Education Census and the Registry of Higher Education Institutions; IV- Data on the performance of HEI students in ENADE, available at the time of evaluation; V- Course evaluation reports from the HEI produced by External Course Evaluation Committees, available at the time of evaluation; VI- Data from the students' Socioeconomic Questionnaire collected during ENADE application; VII- Reports from the Commitment Protocol Monitoring Committee, where applicable; VIII- CAPES evaluation reports and scores for HEI postgraduate courses, where applicable; IX- Documents on the accreditation and most recent re-accreditation of the HEI; X- Other documents deemed pertinent (Brasil, 2004c, art. 15).
Regarding the evaluation of undergraduate and technological higher education courses, according to art. 20 of Ordinance no. 2.051/2004 (Brasil, 2004c), in addition to the data made available in electronic form by the institution, the following aspects will be considered:
I- The profile of the teaching staff;
II- The conditions of the physical facilities;
III- The didactic pedagogical organization;
IV- The performance of HEI students in ENADE;
V- Data from the students' socioeconomic questionnaire, available at the time of evaluation;
VI- Updated data from the Higher Education Census and the General Registry of Institutions and Courses; and VII - Other aspects considered pertinent by CONAES.
As a result of the work performed by the Evaluation Committee, according to art. 32 of the Ordinance, the report will result in the assignment of scores to each of the evaluated dimensions and to the set of dimensions as a whole, on a scale of five levels. Levels four and five are indicative of strengths, levels one and two indicate weaknesses, and level three represents the minimum acceptable for the processes of course authorization, recognition, and renewal of course recognition, as well as for the accreditation and re-accreditation of institutions.
In 2022, Federal Law no. 14.375, dated june 21 (Brasil, 2022a), was approved, amending provisions of the Student Financing Fund (FIES) legislation, and also amending Law no. 10.861, dated april 14, 2004 (Brasil, 2004), to establish the possibility of in loco evaluation in the virtual modality of higher education institutions and their undergraduate and technological courses. Art. 3º and 4º were updated to stat:
Art. 3º [...] § 2º For the evaluation of institutions, diversified procedures and instruments will be used, including self-evaluation and external in loco evaluation, either in person or virtually, with georeferencing.
[...] § 4º The provision in § 2 of this art. regarding the modalities of external in loco evaluations does not apply to medical, psychology, dentistry, and nursing courses, as well as other higher courses established in accordance with regulations, for which external in loco evaluations will only be conducted in person (NR) (Brasil, 2022a, art. 3).
Art. 4º [...] § 1º The evaluation of undergraduate courses will use diversified procedures and instruments, including, obligatorily, external evaluation by specialist committees in the respective areas of knowledge (Brasil, 2022a, art. 4).
Thus, with the issuance of Law no. 14.375/2022 (Brasil, 2022a), legislative authorization was given for the use of diversified procedures and instruments, including virtual in loco evaluation with georeferencing, while maintaining the possibility of inperson evaluations.
It is important to note that the authorization for conducting virtual in loco external evaluations did not apply to all courses. § 4 of art. 11 excluded certain courses, such as "Medicine, Psychology, Dentistry, and Nursing, as well as other higher courses established according to regulations, for which external in loco evaluations will only be conducted in person" (Brasil, 2022a, art. 4).
After the COVID-19 pandemic, the virtual modality of external evaluation continued, becoming INEP's preferred mode of conducting evaluations, although it remains just one of the methods for applying external in loco evaluation, as mandated by law.
The virtual external evaluation system for the regulatory process has shown some advantages but also disadvantages. In this regard, opinions gathered from a survey conducted among BASis evaluators resulted in the following positive and negative aspects:
To demonstrate the results described in the table above, as well as to understand the theoretical and legal reference base and its relationship with the conclusions proposing an alternative system for external evaluation, a methodological procedure of bibliographic and opinion research was used.
The theoretical foundation, based on bibliographic research, contributed to the understanding of public policies for external evaluation of higher education, the role of the regulatory State, and the context of the Covid-19 pandemic. The normative basis provided important support by enabling the presentation of regulatory evolution and the modalities of external evaluation implemented by MEC/INEP, as well as facilitating the analysis of legal modifications that occurred due to the impacts of the pandemic on the systematics of conducting external evaluations.
The opinion research resulted from a data collection plan that established a target audience directly related to the research problem, given their involvement in inperson external evaluation, and also because this audience began to operate within the new system of virtual external evaluation implemented by MEC/INEP as a direct consequence of the pandemic.
The opinion survey questioned the respondents, all of whom are evaluators trained to operate in the new virtual external evaluation modality by INEP, as follows: “As an INEP evaluator, which aspects do you highlight as positive and which do you point out as negative regarding the new virtual in-person external evaluation modality implemented as a consequence of the Covid-19 pandemic”?
It is noteworthy that the evaluators who participated in the opinion survey are organized into dialogue and mutual support groups via WhatsApp, a technological tool that facilitates the organization of virtual communities with common objectives and challenges.
The opinion survey was conducted voluntarily among evaluator professors who are part of BASis and participate in the mentioned groups. The evaluator groups are named as follows: “Avaliadores”, with 266 participating professors; the group “Avaliadores INEP,” with 291 participating educators; the group “Avaliadores do INEP”, with 272 participating professors; and the group “Apenas Avaliação”, with 256 participants. A total of 75 professors from all regions of the country participated in the opinion survey.
The findings obtained and demonstrated reflect the experiences of the survey participants, who evaluate courses in various fields of study and have conducted more than two dozen evaluations in the virtual modality, representing personal contributions. It is worth noting that these findings are in perfect alignment with the general opinion of the survey.
It is important to highlight that, as an opinion survey reflecting the results of a statistical inquiry drawn from a particular understanding sample, it is necessary to emphasize the limitations inherent in research conducted in this format. In this regard, inherent implications include the risk that the respondent may perceive potential identification and possible consequences in the hierarchical relationship with superiors, even though this relationship is not employment-based but exists for the purpose of designation to the evaluation team. There is also the risk of the respondent's commitment to the survey and the prevalence of their personal position.
Such research risks are not ignored. However, considering that there was, in the overwhelming majority of opinions, alignment regarding the main positive and negative aspects highlighted and that most respondents expressed similar opinions, the existing risks in an opinion survey would not have the impact of altering the conclusions in their overall aspect. This is the finding.
It is worth noting that, in the context of an opinion survey involving a target audience engaged in the same evaluative function, the data collection process within the respective groups was supported by evaluator professors who motivated other teachers to contribute to the survey. The opinions were sent to the researchers' personal WhatsApp accounts by the teachers who voluntarily provided their input and contributed to the research. In this regard, potential weaknesses pertain to the number of participants. However, the spontaneous nature and the consistency observed among the different participants ensure that these weaknesses do not significantly influence the results.
It is important to highlight that, given this is an opinion survey involving a target audience engaged in the same evaluative function, data collection within the respective groups was supported by evaluator professors who motivated other teachers to contribute to the survey. The opinions were sent to the researchers' personal WhatsApp accounts by teachers who voluntarily provided their input and contributed to the research. In this regard, potential weaknesses relate to the number of participants. However, the spontaneous nature and the consistency observed among the different participants ensure that these weaknesses do not significantly impact the results.
On the other hand, INEP (2022, p. 6) conducted an opinion survey with evaluators registered in its database, highlighting in a report the positive aspects of the change in the evaluation model. In its interpretation, it states that "it was possible to observe the advantages and benefits that this evaluation modality brought to the external evaluation procedures," listing the speed of designation, the ease of conducting visits to institutions located in hard-to-reach areas, and the significant savings of public resources generated by the new process.
INEP's understanding is that this is an advancement that has brought many benefits to society. With agile processes, the new model allowed for simultaneous visits by committees to different institutions and greater availability of evaluators to participate in processes remotely.
From the perspective of Saviani and Galvão (2021, p. 42), remote activities and teaching are impoverished by the coldness and the impossibility of different forms of approach, communication, interaction, and teaching. However, they end up being imposed due to the strong private interests involved. This same relationship can be applied to virtual in loco evaluation.
The pandemic generated and promoted:
A set of actions conceived, planned, and put into practice by the State, which, under the pretext of the precautions recommended to combat the pandemic, triggered institutional evaluation, from its regulatory perspective, through the virtual modality (Paiva; Otranto; Souza, 2021, p. 11).
According to (Paiva; Otranto; Souza, 2021, p. 11), in the new maze of "structural changes of the MEC/INEPE, there has been a relaxation in the quality of education in its historical, human, democratic, and participatory essence", they claim that "the conception adopted for regulation is based on the opposite direction of the essence of SINAES, which, in its origin, encouraged participation as a democratic principle", and they emphasize that "the simplification of the evaluation process, transferred from inperson to virtual, with the use of the same methodology, is done in the same way as industrial products produced in series".
Continuing their critical position regarding the shift to remote external evaluation, (Paiva; Otranto; Souza, 2021, p. 11) state that:
In this regulatory virtual journey, it will not be possible to demonstrate the result of the reinterpretation of knowledge, the atmosphere provided by interpersonal relationships, and the perception that emanates from dialogues, enriched by scientific findings over time.
Thus, it becomes evident, based on all the findings, that there is a critical stance regarding various aspects of the virtual in-person external evaluation modality implemented by INEP as a result of the Covid-19 pandemic, whether through the opinions of the survey respondents or the position of science.
In this regard, the research proves to be relevant due to its dual contribution: to science and to the definition and improvement of public policies for external evaluation of higher education quality. Regarding the scientific contribution, it highlights the demonstration of the theoretical stance on the concepts and themes addressed, which involve the external evaluation carried out by INEP in higher education institutions and courses, aiming to support the regulatory processes of higher education under the responsibility of the MEC.
On the other hand, regarding the contribution to the implementation and improvement of public policies in the field of regulation, significant contributions are evident. These contributions stem from an analysis and understanding of the systematic approach of in-person external evaluation and the virtual modality, implemented with technological support as a result of the Covid-19 pandemic.
In this sense, public policies need to be in constant analysis and evaluation to effectively meet their purposes. The theoretical review revealed criticisms regarding the abandonment of in-person visits and various negative aspects highlighted in this system. Similarly, the respondents' opinions pointed out several weaknesses in the virtual remote evaluation system. Understanding the critical aspects and weaknesses is an essential process in evaluating public policies related to assessment and regulation, which are the responsibility of the Brazilian State.
Finally, the identification of positive aspects and existing weaknesses in the virtual (remote) in-person external evaluation modality implemented by INEP, evidenced by the research and theoretical contributions, allowed for the presentation of a balanced proposal for INEP's external evaluations. This proposal consists of the adoption of a hybrid system of in-person external evaluation, presented as follows:
-
I Proposal for external evaluation for institutional accreditation, reaccreditation, and renewal of accreditation:
-
a) In-person external evaluation for Institutional Accreditation - Conducted in the virtual (remote) modality.
-
b) In-person external evaluation for Institutional Reaccreditation - Conducted in person by the evaluators.
-
c) Subsequent Renewals of Recognition - Conducted through alternating (hybrid) in-person external evaluations, with one in the virtual format and the following in the in-person modality.
-
-
II Proposal for external evaluation for course authorization, course recognition, and renewal of course recognition:
-
a) In-person external evaluation for Course Authorization - Conducted in the virtual (remote) modality.
-
b) In-person external evaluation for Course Recognition - Conducted in person by the evaluators.
-
c) Subsequent Renewals of Recognition - Conducted through alternating (hybrid) in-person external evaluations, with one in the virtual format and the following in the in-person modality.
-
This proposal for a Hybrid modality (remote/in-person) allows for the incorporation of the positive aspects of both the in-person and virtual (remote) modalities through alternating implementation. On the other hand, it represents an important contribution to the improvement of public evaluation policies, especially with regard to addressing the weaknesses (negative aspects) identified in the adoption of the virtual (remote) modality as the sole format for in-person external evaluation currently implemented by MEC/INEP.
This hybrid evaluation solution proposal, as a result of the conclusion of the present research, would more adequately and equitably encompass both forms of inperson external evaluation in terms of their positive and negative aspects. Moreover, it would lead to better utilization of the advantages of each modality, benefiting the regulatory State, the institutions, and their evaluated programs, ensuring that all parties are considered.
This proposal aligns with Saviani's (2023, p. 196) understanding, who, when reflecting on the end of the pandemic, posits that "the dominant sectors of society tend to use the health crisis to deepen forms of domination." He refers to the adoption of the systematic use of remote work to conclude that "the same is occurring in the field of education, with a tendency to widely replace in-person education with distance learning, attributing a permanent character to so-called 'remote teaching.'" This proposal also aligns with the results of the opinion survey conducted with INEP evaluators and demonstrated in this study.
5 Final considerations
From the constitutional definition that education, according to art. 209, is free for private initiative, subject to compliance with the general rules of national education, it becomes necessary for the Public Authority to authorize its operation and evaluate its quality. The regulatory process was interpreted to demonstrate the existing legal procedures.
In the regulation process, many entities, professionals, and procedures are involved. There are laws and regulatory acts that discipline the regulation of authorization and quality evaluation, as defined by the Constitution (Brasil, 1988), with emphasis on the LDB (Brasil, 1996), the SINAES Law (Brasil, 2004a), the in loco Evaluation Fee Law (Brasil, 2004b), Decrees no. 9.235/2017 (Brasil, 2017b) and no. 9.057/2017 (Brasil, 2017a), and Normative Ordinances no. 2051/2004 (Brasil, 2004c) and no. 11/2017 (Brasil, 2017c), among other legal norms.
The purpose of in loco external evaluations is to produce inputs for the issuance of authorization acts for institutional accreditation, reaccreditation renewal, and deaccreditation of institutions, for the authorization, recognition, and renewal of recognition of higher education courses, assessing compliance with legal provisions and contributing to quality control.
During the COVID-19 pandemic, in person in loco external evaluations began to be conducted virtually, which occurred with the issuance of Normative Ordinance no. 165/2021 (Brasil, 2021b) and the issuance of Law no. 14.375/2022 (Brasil, 2022a). This significant change in the way evaluations were conducted impacted the performance of regulatory bodies and those responsible for evaluations, the evaluators, the institutions, and all professionals involved in the evaluation processes, generating both positive and negative impacts, as demonstrated by the opinion survey conducted among evaluators and the authors' findings.
The theoretical contribution derived from scientific production and the results of the opinion survey made it possible to identify positive aspects in conducting inperson external evaluations, both in the in-person and virtual (remote) modalities. On the other hand, weaknesses (negative aspects) were identified concerning both modalities of in-person external evaluation (in-person and remote). These findings enabled the presentation of a proposal for a hybrid in-person external evaluation modality, demonstrated as a result of the research.
The hybrid in-person external evaluation modality, as presented, represents the implementation of an alternating system between virtual/remote and in-person formats. This modality contributes to the improvement of public policies for the evaluation of higher education quality and supports the regulatory activities of MEC/INEP.
The proposed hybrid in-person external evaluation, by contributing to the improvement of regulatory public policies, has the capacity to incorporate all the positive aspects of in-person and virtual (remote) evaluations in a balanced and productive manner, establishing itself as a more effective approach for achieving results in the assessment of compliance with norms and quality
References
-
BOLLELA, V. R.; MEDEIROS, l. S. TELLES, S. Educação remota em tempos de pandemia: reflexões no contexto acadêmico. Medicina, Ribeirão Preto, v. 54, n. 1, p. e-184771, 2021. Disponível em: https://www.revistas.usp.br/rmrp/article/view/184771/175264 Acesso em: 4 set. 2024.
» https://www.revistas.usp.br/rmrp/article/view/184771/175264 -
BRASIL. [Constituição (1988)]. Constituição da República Federativa do Brasil, de 05 de outubro de 1988 Brasília, DF: Presidência da República, 1988. Disponível em: https://www.planalto.gov.br/constituicao.htm Acesso em: 22 dez. 2023.
» https://www.planalto.gov.br/constituicao.htm -
BRASIL. Coronavírus Brasília, DF: Ministério da Saúde, 2021a. Disponível em: https://www.gov.br/saude/pt-br Acesso em: 14 out. 2023.
» https://www.gov.br/saude/pt-br -
BRASIL. Decreto nº 11.691, de 5 de setembro de 2023 Aprova a Estrutura Regimental e o Quadro Demonstrativo dos Cargos em Comissão e das Funções de Confiança do MEC. Brasília, DF: Presidência da República, 2023. Disponível em: https://www.planalto.gov.br/ccivil_03/_ato2023-2026/2023/decreto/D11691.htm Acesso em: 21 dez. 2023.
» https://www.planalto.gov.br/ccivil_03/_ato2023-2026/2023/decreto/D11691.htm -
BRASIL. Decreto nº 9.057, de 25 de maio de 2017 Regulamenta o art. 80 da Lei nº 9.394, de 20 de dezembro de 1996, que estabelece as diretrizes e bases da educação nacional. Brasília, DF: Presidência da República, 2017a. Disponível em: https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2017/decreto/d9057.htm Acesso em: 10 jan. 2024.
» https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2017/decreto/d9057.htm -
BRASIL. Decreto nº 9.235, de 15 de dezembro de 2017 Dispõe sobre o exercício das funções cursos superiores de graduação e de pós-graduação no sistema federal de ensino. Brasília, DF: Presidência da República, 2017b. Disponível em: https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2017/decreto/d9235.htm Acesso em: 04 jan. 2024.
» https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2017/decreto/d9235.htm -
BRASIL. Lei nº 10.861, de 14 de abril de 2004 Institui o Sistema de Avaliação da Educação Superior - SINAES e dá outras providências. Brasília, DF: Presidência da República, 2004a. Disponível em: https://www.planalto.gov.br/ccivil_03/_ato20042006/2004/lei/l10.861.htm Acesso em: 4 set. 2024.
» https://www.planalto.gov.br/ccivil_03/_ato20042006/2004/lei/l10.861.htm -
BRASIL. Lei nº 10.870, de 19 de maio de 2004 Institui a Taxa de Avaliação in loco das instituições de educação superior e dos cursos de graduação e dá outras providências. Brasília, DF: Presidência da República, 2004b. Disponível em: https://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.870.htm Acesso em: 16 nov. 2023.
» https://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.870.htm -
BRASIL. Lei nº 14.375, de 21 de junho de 2022 Altera as Leis nºs 10.260, de 12 de julho de 2001, 10.522, de 19 de julho de 2002, e 12.087, de 11 de novembro de 2009, para estabelecer os requisitos e as condições para realização das transações resolutivas de litígio relativas à cobrança de créditos do Fundo de Financiamento Estudantil (Fies), a Lei nº 10.861, de 14 de abril de 2004, para estabelecer a possibilidade de avaliação in loco na modalidade virtual das instituições de ensino superior e de seus cursos de graduação, a Lei nº 13.988, de 14 de abril de 2020, para aperfeiçoar os mecanismos de transação de dívidas, e a Lei nº 13.496, de 24 de outubro de 2017; e revoga dispositivos das Leis nºs 13.530, de 7 de dezembro de 2017, 13.682, de 19 de junho de 2018, 13.874, de 20 de setembro de 2019, e 14.024, de 9 de julho de 2020. Brasília, DF: Presidência da República, 2022a. Disponível em: https://www.planalto.gov.br/ccivil_03/_ato2019-2022/2022/lei/L14375.htm Acesso em: 05 jan. 2024.
» https://www.planalto.gov.br/ccivil_03/_ato2019-2022/2022/lei/L14375.htm -
BRASIL. Lei nº 9.394, de 20 de dezembro de 1996 Estabelece as diretrizes e bases da educação nacional. Brasília, DF: Presidência da República, 1996. Disponível em: https://www.planalto.gov.br/ccivil_03/leis.htm Acesso em: 22 set. 2023.
» https://www.planalto.gov.br/ccivil_03/leis.htm -
BRASIL. Ministério da Educação. Gabinete do Ministro. Portaria Normativa nº 11, de 20 de junho de 2017. Estabelece normas para o credenciamento de instituições e a oferta de cursos superiores a distância, em conformidade com o Decreto nº 9.057, de 25 de maio de 2017. Diário Oficial da União: seção 1, Brasília, DF, n. 118, p. 14-16, 22 jun. 2017c. Disponível em: http://portal.mec.gov.br/docman/junho-2017-pdf/66441pn-n11-2017-regulamentacao-ead-republicada-pdf/file Acesso em: 14 nov. 2023.
» http://portal.mec.gov.br/docman/junho-2017-pdf/66441pn-n11-2017-regulamentacao-ead-republicada-pdf/file -
BRASIL. Ministério da Educação. Gabinete do Ministro. Portaria nº 2.051, de 09 de julho de 2004 Regulamenta os procedimentos de avaliação do Sistema Nacional de Avaliação da Educação Superior (SINAES), instituído na Lei no 10.861, de 14 de abril de 2004. Brasília, DF, 14 abr. 2004c. Disponível em: http://portal.mec.gov.br/arquivos/pdf/PORTARIA_2051.pdf Acesso em: 18 set. 2023.
» http://portal.mec.gov.br/arquivos/pdf/PORTARIA_2051.pdf -
BRASIL. Ministério da Educação. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP). Avaliação externa virtual in loco: desafios da implementação e análise dos primeiros resultados. Brasília, DF: Inep, 2022b. Disponível em: https://www.gov.br/inep/pt-br/centrais-de-conteudo/acervo-linha-editorial/publicacoes-institucionais/avaliacoes-e-exames-da-educacao-superior/avaliacao-externa-virtual-in-loco-desafios-da-implementacao-e-analise-dos-primeiros-resultados Acesso em: 4 set. 2024.
» https://www.gov.br/inep/pt-br/centrais-de-conteudo/acervo-linha-editorial/publicacoes-institucionais/avaliacoes-e-exames-da-educacao-superior/avaliacao-externa-virtual-in-loco-desafios-da-implementacao-e-analise-dos-primeiros-resultados -
BRASIL. Ministério da Educação. Portaria nº 165, de 20 de abril de 2021. Institui a Avaliação Externa Virtual in loco no âmbito das visitas por comissões de especialistas para avaliação externa de Instituições de Educação Superior e cursos de graduação, no âmbito do SINAES, e da avaliação das Escolas de Governo. Diário Oficial da União: seção 1, Brasília, DF, n. 74, p. 181, 22 abr., 2021b. Disponível em: https://abmes.org.br/arquivos/legislacoes/PORTA-RIA%20INEP%20165%20DE%2022%20DE%20ABRIL%20DE%202021.pdf Acesso em: 17 set. 2023.
» https://abmes.org.br/arquivos/legislacoes/PORTA-RIA%20INEP%20165%20DE%2022%20DE%20ABRIL%20DE%202021.pdf -
CAVALCANTI, L. M. R.; GUERRA, M. das G. G. V. Uso de modelo de utilidade para avaliação de cursos de graduação. Avaliação: Revista da Avaliação da Educação Superior, Campinas; Sorocaba, v. 27, n. 3, p. 513-530, set. 2022. Disponível em: https://www.scielo.br/j/aval/a/SShN4hbnYKqf6VvmnqBzC6S/?lang=pt Acesso em: 13 nov. 2023.
» https://www.scielo.br/j/aval/a/SShN4hbnYKqf6VvmnqBzC6S/?lang=pt -
GUERRA, M. G. G. V.; CAVALCANTI, L. M. R. . A Universidade Pública Pós-Pandemia da COVID-19: uma nova instituição para um novo tempo. Currículo sem fronteiras, [S. l.], v. 21, n. 2, p. 884-900, maio/ago. 2021. Disponível em: http://curriculosemfronteiras.org/vol21iss2articles/guerra-cavalcanti.pdf Acesso em: 08 mai. 2024.
» http://curriculosemfronteiras.org/vol21iss2articles/guerra-cavalcanti.pdf -
INSTITUTO BUTANTAN. Portal do Butantan. Entenda o que é uma pandemia e as diferenças entre surto, epidemia e endemia São Paulo: Butantan, 2021. Disponível em: https://butantan.gov.br/covid/butantan-tira-duvida/tira-duvida-noticias/entenda-o-que-e-uma-pandemia-e-as-diferencas-entre-surto-epidemia-e-endemia#:~:text=Uma%20pandemia%20pode%20come%C3%A7ar%20como,de%20casos%20de%20uma%20doen%C3%A7a Acesso em: 4 set. 2024.
» https://butantan.gov.br/covid/butantan-tira-duvida/tira-duvida-noticias/entenda-o-que-e-uma-pandemia-e-as-diferencas-entre-surto-epidemia-e-endemia#:~:text=Uma%20pandemia%20pode%20come%C3%A7ar%20como,de%20casos%20de%20uma%20doen%C3%A7a - MELLO, C. A. B. de. Curso de direito administrativo São Paulo: Malheiros, 2015.
-
ORGANIZAÇÃO MUNDIAL DA SAÚDE (OPAS). Histórico da pandemia de COVID-19 2020. Disponível em: https://www.paho.org/pt/covid19/historico-da-pandemia-covid-19 Acesso em: 28 dez. 2023.
» https://www.paho.org/pt/covid19/historico-da-pandemia-covid-19 -
PAIVA, L. D. C; OTRANTO, C. R. SOUZA, N. M. P. Os (Des) caminhos da avaliação externa virtual in loco: os Institutos Federais de Educação, Ciência e Tecnologia em discussão. REVELLI - Revista de Educação, Linguagens e Literatura, Inhumas, v. 13, p. 1-23, 2021. Disponível em: https://www.revista.ueg.br/index.php/revelli/article/view/12212 Acesso em: 05 jun. 2024.
» https://www.revista.ueg.br/index.php/revelli/article/view/12212 -
SAVIANI, D. A importância da presencialidade no ensino superior. Revista Linhas, Florianópolis, v. 24, n. 56, p. 182-207, set./dez. 2023. Disponível em: https://revistas.udesc.br/index.php/linhas/article/view/24422 Acesso em: 05 jan. 2024.
» https://revistas.udesc.br/index.php/linhas/article/view/24422 -
SAVIANI, D.; GALVÃO, A. C. Educação na pandemia: a falácia do ensino remoto. Revista Universidade e Sociedade, Tirol, v. 31, p. 36-49, jan. 2021. Disponível em: https://docente.ifrn.edu.br/julianaschivani/disciplinas/midias-educacionais/educacaona-pandemia-a-falacia-do-201censino201d-remoto/view Acesso em: 4 set. 2024.
» https://docente.ifrn.edu.br/julianaschivani/disciplinas/midias-educacionais/educacaona-pandemia-a-falacia-do-201censino201d-remoto/view -
VERHINE, R. E. Avaliação e regulação da educação superior: uma análise a partir dos primeiros 10 anos do SINAES. Avaliação: Revista da Avaliação da Educação Superior, Campinas; Sorocaba, v. 20, n. 3, p. 603-619, nov. 2015. Disponível em: http://www.scielo.br/scielo.php?script=sci Acesso em: 23 nov. 2023.
» http://www.scielo.br/scielo.php?script=sci
