Resolute factors for the performance of Brazilian Federal Institutes in the National Examination of High School Education * * English version by Karima Bezerra de Almeida.

Rogério Severiano Dutra Giselle Bezerra Mesquita Dutra Paulo Henrique Nobre Parente Lya Oliveira da Silva Souza Parente About the authors

Resumo

O objetivo do estudo é investigar em que medida os contextos determinam o desempenho dos Institutos Federais do Brasil no ENEM. Para a consecução do trabalho, optou-se por utilizar o ENEM como fonte de avaliação educacional, por apresentar características como funções somativa e proativa, de periodicidade pontual, com orientação a tomada de decisões, do tipo formal e de larga escala. O levantamento e a coleta de dados relativos aos contextos e aos desempenhos educacionais dos Institutos Federais do Brasil divulgados pelo INEP constituíram o conjunto amostral de 865 observações relativas aos anos de 2011, 2012, 2013, 2014 e 2015. O delineamento metodológico dispôs da aplicação de métodos univariados de medidas de tendência central e de dispersão, assim como de testes de comparação de médias amostrais; Teste t de Student para comparação de duas amostras; e Anova para comparação acima de duas amostras. Como principais achados da pesquisa, tem-se que a hipótese de que os contextos determinam diferenças significantes entre os desempenhos dos Institutos Federais do Brasil foi confirmada para os contextos regionais, para o espaço geográfico e para o perfil socioeconômico discente. Quanto ao contexto relativo ao porte escolar, mesmo não apresentando diferenças significativas de desempenho evidenciadas pelos testes, as áreas do conhecimento em Ciências Humanas e Redação apresentaram indícios de superioridade nos Institutos Federias de menor porte; já em Matemática e suas Tecnologias, o teste apresentou diferenças significativas, com desempenho superior nos Institutos de maior porte escolar. Outro achado da pesquisa foi a identificação do perfil dos Institutos Federais participantes do ENEM, o que foi fortemente percebido na zona urbana das regiões Nordeste e Sudeste, que apresentaram elevado quantitativo de alunos de classe média matriculados.

Avaliação educacional; Contexto escolar; Desempenho educacional; Institutos Federais do Brasil

Abstract

This study aims to investigate to what extent the context of the Brazilian Federal Institutes determines their performance in ENEM (High School National Examination). For the development of this work, we chose to use ENEM as a source of educational evaluation, since it features summing and proactive functions, with prompt frequency, and guidance to decision-making, it is a large scale formal assessment. Data collection and survey of the learning performance and context of the Brazilian Federal Institutes disclosed by INEP (National Institute for Educational Studies and Research) included a sample of 865 reports concerning the years 2011, 2012, 2013, 2014, and 2015. The methodological outline comprised the application of univariate methods of measures of central tendency and dispersion, as well as of sample mean comparison tests; Student’s T-Test to compare two samples; and Anova for more than two samples. The main findings of this present research show that its hypothesis has been confirmed to the extent that regional contexts, geographical location, and the socio-economic profile of students have strongly affected student performance in the Brazilian Federal Institutes. As for the context of school facility size, even not presenting significant general performance difference in test results, the fields of Humanities and Writing showed evidence of higher performance in smaller facilities; and for the fields of Mathematics and Technologies, the tests showed significant differences, with better performance in larger facilities. This research has also found a greater number of upper-middle class students enrolled in Federal Institutes that participate in ENEM, strongly perceived in the Brazilian Northeast and Southeast urban units.

Educational evaluation; School context; Educational performance; Brazilian Federal Institutes

Introduction

The primary aspect of the educational evaluation process is: “conocer el papero que tiepeña, la función que cumple, saber quién se benefits from ella y, en definitiva, al servicio de quién se pone” ( RUIZ, 1996RUIZ, José Maria. Teoría del curriculum: diseño y desarrollo curricular. Madrid: Universiatas, 1996. , p. 17). According to Andriola (1999)ANDRIOLA, Wagner Bandeira. Evaluación: la vía para la calidad educativa. Ensaio , Rio de Janeiro, v. 7, n. 25, p. 355-368, 1999. determining the elements that improve the quality of education is the most relevant objective of educational evaluation.

Endogenous and exogenous factors capable of clarify the quality of education can be found in school environment ( GADOTTI, 2010GADOTTI, Moacir. Qualidade na educação: uma nova abordagem. Produção de terceiros sobre Paulo Freire. São Paulo: Série Livros, 2010. ). These factors highlight the context of the educational institutions and determine the setting of goals for a successful educational output and, consequently, educational performance.

Educational research has shown that diverging student performance is associated with differences in the contexts in which student groups are inserted ( BRYK; LEE, 1989BRYK, Antony; LEE, Valerie. A multilevel model of social distribution of high school. In: DARLING-HAMMOND, Linda (Ed.). Review of research in education . Washington, DC: AERA, 1989. p. 171-267. ; FRANCO et al., 2007FRANCO, Creso et al. Qualidade e equidade em educação: reconsiderando o significado de “fatores intra-escolares”. Ensaio , Rio de Janeiro, v. 15, n. 55, p. 277-298, 2007. ). To check for such differences, this study has selected the Brazilian Federal Institutes in their contexts, by means of information provided by the National Institute for Educational Studies and Research Anísio Teixeira (INEP) with respect to (i) regional location; (ii) geographical area; (iii) socioeconomic profile of students; and (iv) school facility size.

The existing diversity among school locations is a parameter that can show differences in educational opportunity that can affect educational performance ( BARROS et al., 2001BARROS, Ricardo Paes de et al. Determinantes do desempenho educacional no Brasil. Pesquisa e Planejamento Econômico , Rio de Janeiro, v. 31, n. 1, p. 1-42, 2001. ). Similarly, school results can be determined by the socio-economic and cultural origin of students enrolled in schools ( COLEMAN et al., 1966COLEMAN, James et al. Equality of educational opportunity. Washington, DC: U.S. Department of Health, Education, and Welfare, 1966. ; BOURDIEU; PASSERON, 2008BOURDIEU, Pierre; PASSERON, Jean-Claude. A reprodução: elementos para uma teoria do sistema de ensino. Petrópolis: Vozes, 2008. ). Also, the size of the facilities is a context parameter that entails students of different levels and courses, that need larger areas, with furniture and equipment adapted to students of different age groups and specific needs, as well as more members in the staff and other supporting jobs ( ALVES; SOARES, 2013ALVES, Maria Teresa Gonzaga; SOARES, José Francisco. Contexto escolar e indicadores educacionais: condições desiguais para a efetivação de uma política de avaliação educacional. Educação e Pesquisa , São Paulo, v. 39, n. 1, p. 177-194, jan./mar. 2013. ). This context parameter can highlight the educational performance of whether they can or not meet school needs.

The development of knowledge, skills and competences should be stimulated for the construction of critical and analytical thinking ( LIMBERGER, 2013LIMBERGER, Jane Beatriz. Metodologias ativas de ensino-aprendizagem para a educação farmacêutica: um relato de experiência. Interface , Botucatu, v. 17, n. 47, p. 969-975, 2013. ), enabling learning to be constructive, contextual, collaborative, and self-directed ( JONG et al., 2014JONG, Nynke et al. Blended learning in health education: three case studies. Perspectives Medical Education , Maastrich, v. 3, n. 4, p. 278-288, sep. 2014. ). The educational performance (proficiency) evaluations carried out inside and outside the school facilities are used to ascertain the results of the teaching-learning process ( HANUSHEK, 1979HANUSHEK, Eric A. Conceptual and empirical issues in the estimation of educational production functions. Journal of Human Resources , Madison, v. 14, n. 3, p. 351-388, 1979. ; PRIMI; HUTZ; SILVA, 2011), but not necessarily student learning level with accuracy.

External large-scale evaluations are used to assess the factors (resolute contexts) related to higher (or lower) performance, aimed at finding information that can help to promote actions that will improve educational results ( CAMARGO et al., 2016CAMARGO, Raphael Vinicius Weigert et al. Desempenho dos alunos de ciências contábeis na prova ENADE/2012: uma aplicação da Teoria da Resposta ao Item. Revista de Educação e Pesquisa em Contabilidade , Brasília, DF, v. 10, n. 3, p. 332-355, jul./set. 2016. ). Thus, the focus of this research is to assess student proficiency using their results by field of study and their mean score at the National Examination of High School Education (ENEM) as parameters of the educational performance of the institutions.

ENEM was established in 1998 and is considered a large-scale evaluation tool devised by the Ministry of Education and Culture (MEC) with the aim of assessing high school student learning and skills in the following areas of knowledge: Languages and Codes; Mathematics and Technologies; Humanities; Natural Sciences; and Writing ( COTTA, 2001COTTA, Tereza Cristina. Avaliação educacional e políticas públicas: a experiência do Sistema Nacional de Avaliação da Educação Básica (SAEB). Revista do Serviço Público , Brasília, DF, v. 52, n. 4, p. 89-111, 2001. ).

Thus, considering that the context of institutions is resolute for educational performance, the following research question comes up: in what measures do the contexts determine the performance of the Brazilian Federal Institutes in the National Examination of High School Education (ENEM)? To answer this question, based in the objective of this study, we intend to investigate to what extent school context might determine the performance of the Brazilian Federal Institutes in ENEM.

Taking into account that Brazil is a very diverse country, due to its great extension, we have expected to find significant differences in educational performance among the Brazilian Federal Institutes, according to each reality determined by their context. Therefore, our research has proposed the hypothesis that school contexts determine significant differences in the student performance of Brazilian Federal Institutes in ENEM.

This present study on the context of the Brazilian Federal Institutes is considered relevant because it: (i) presents statistical test results that are able to provide important supporting information on the learning proficiency levels of students in ENEM, which can be used in the decision-making of public policies referred to education; (ii) uses indicators from the surveys of INEP to represent school environments; (iii) uses student results from ENEM, which is a learning evaluation featuring summing and proactive functions, with prompt frequency, and guidance to decision-making, formal typology and in large scale; and (iv) monitors the evolution of student performance in education over the research period of time.

This work has been organized into four sections, including this introduction. Section 2 brings the theoretical reference, some possible contexts of school institutions that are resolute for educational performance, introduces the National Examination of High School Education (ENEM), and previous studies. Section 3 describes the methodology used. Section 4 shows the results found; and Section 5 presents the main conclusion of our research.

Theoretical reference

Context of school institutions

Teaching proposals should depend on school context. Knowing the regional, geographical, socio-economic contexts of students and the demand for educational services is paramount to formulate educational practices ( SOUZA et al., 2013SOUZA, Norma Valéria Dantas de Oliveira et al. Perfil socioeconômico e cultural do estudante ingressante no curso de graduação em enfermagem. Revista de Enfermagem UERJ , Rio de Janeiro, v. 21, n. 2, p. 718-722, 2013. ).

The characteristics of the location where educational projects are to be employed is a strategic context parameter in which we should observe, among other factors, the local economy, the shortage of qualified employees in the labor market, life expectancy, schooling, the cultural levels of the community for the proper employment of courses, whether the area is metropolitan or not, urban or rural ( BARROS et al., 2001BARROS, Ricardo Paes de et al. Determinantes do desempenho educacional no Brasil. Pesquisa e Planejamento Econômico , Rio de Janeiro, v. 31, n. 1, p. 1-42, 2001. ).

Another context parameter to be observed, according to Alves and Soares (2009)ALVES, Maria Teresa Gonzaga; SOARES, José Francisco. Medidas de nível socioeconômico em pesquisas sociais: uma aplicação aos dados de uma pesquisa educacional. Opinião Pública , Campinas, v. 15, n. 1, p. 1-30, 2009. , is student’s socio-economic level, measured through the schooling and job position of their parents, along with family income and goods, their potential to the consumption of goods and services, and their different sources of income.

The size of the school facility, measured by the number of students enrolled, is a context parameter that determines the number of rooms, the number of employees, the offer of teaching course modalities and the number of groups. The different levels and teaching modalities – regular education, from nursery to High School, and special education for the same grades – need larger facilities, more furniture and equipment adapted to students of different age groups and with specific needs, as well as a larger staff and varied support assistants ( ALVES; SOARES, 2013ALVES, Maria Teresa Gonzaga; SOARES, José Francisco. Contexto escolar e indicadores educacionais: condições desiguais para a efetivação de uma política de avaliação educacional. Educação e Pesquisa , São Paulo, v. 39, n. 1, p. 177-194, jan./mar. 2013. ).

Thus, the reality of each institution defines its context, which can be considered a resolute factor for the success of students and institution, which is often measured by educational performance by means of internal evaluation or large-scale evaluations, which highlight the quality of the education offered by such schools according to region, state, education network, administrative facilities, socio-economic profile, urban or rural locations, complexity of school management, etc.

Educational assessment

Educational assessment is a tool for verifying the scenario of the teaching environment, with the aim of guiding decision-making and improving educational processes ( CORRÊA, 2013CORRÊA, Denise Maria Moreira Chagas. Indicadores de gestão do TCU e o desempenho dos cursos no Enade: um estudo de cursos de graduação. Tese (Doutorado) – Faculdade de Educação, Universidade Federal do Ceará, Fortaleza, 2013. ). Therefore, evaluative functions vary according to conceptions, data collection methods, and interpretative techniques (WORTHEN; SANDERS; FITZPATRICK, 2004).

For Luckesi (2014)LUCKESI, Cipriano Carlos. Avaliação da aprendizagem escolar: estudos e proposições. São Paulo: Cortez, 2014. , the evaluation process has the function of diagnosing. Luckesi believes that it checks the initial level of learning and the path to be taken in order to reach a particular goal. For Nevo (1997)NEVO, Daniel. Evaluación basada en el centro: un diálogo para la mejora educativa. Bilbao: Mensajero, 1997. , evaluation has both the formative and the summing functions. The formative function, of a procedural nature, makes it possible to improve methods and techniques that help the development or formation of the project, curriculum, organization and student ( CORRÊA, 2013CORRÊA, Denise Maria Moreira Chagas. Indicadores de gestão do TCU e o desempenho dos cursos no Enade: um estudo de cursos de graduação. Tese (Doutorado) – Faculdade de Educação, Universidade Federal do Ceará, Fortaleza, 2013. ). On the other hand, the summing function emphasizes the resulting product of the process, measuring the achievement of the objectives and previously established goals to a program, to an institution, to a curriculum or to learning ( SCRIVEN, 2000SCRIVEN, Michael. The logic and methodology of checklists . 2000. Disponível em: < https://wmich.edu/sites/default/files/attachments/u350/2014/logic%26methodology_dec07.pdf> . Acesso em: 14 abr. 2018.
https://wmich.edu/sites/default/files/at...
).

One can also consider the concepts mentioned, in which the frequency of evaluation can be defined as procedural (daily basis), and be categorized as formative, or defined as prompt (conclusive), and is characterized as summative, making it possible in the formative function, to build a daily process of learning, and in the summative function, to check for the difficulties that have been pointed out ( LUCKESI, 2014LUCKESI, Cipriano Carlos. Avaliação da aprendizagem escolar: estudos e proposições. São Paulo: Cortez, 2014. ).

For educational assessment, Stufflebean and Shinkfield (1987) suggest the proactive function, used for decision-making based on the context of students; and the retroactive function, which is carried out through teacher interventions, identifying differences between previous and current learning processes, identifying learning deficiencies. There is also, in the literature, models of educational evaluation that refer to: (i) objectives ( TYLER, 1986TYLER, Ralph Winfred. Princípios básicos de currículo e ensino. Tradução de Leonel Vallandro. 9. ed. Rio de Janeiro: Globo, 1986. ); (ii) decision-making ( CRONBACH, 1963CRONBACH, Lee Joseph. Course Improvement through evaluation. Teachers College Record , New York, v. 64, n. 8, p. 672-683, 1963. ; SCRIVEN, 2000SCRIVEN, Michael. The logic and methodology of checklists . 2000. Disponível em: < https://wmich.edu/sites/default/files/attachments/u350/2014/logic%26methodology_dec07.pdf> . Acesso em: 14 abr. 2018.
https://wmich.edu/sites/default/files/at...
; STUFFLEBEAM, 2000); (iii) negotiating ( STAKE, 1978STAKE, Robert E. Case study methodology in social inquire. Educational Researcher , Washington, DC, v. 7, p. 5-8, 1978. ; PARLETT; HAMILTON, 1976PARLETT, Malcolm; HAMILTON, David. Evaluation as ilumination: a new aproach to the study of innovatory programs. In: GLASS, Gene V. (Ed.). Evaluation studies review annual . Beverly Hills: Sage, 1976. p. 141-157. ; GUBA; LINCOLN, 1988GUBA, Egon; LINCOLN, Yvonna. Effective evaluation. San Francisco: Jossey-Bass, 1988. ); and (iv) empowerment, ( SAUL, 1988SAUL, Ana Maria. Avaliação emancipatória: desafio à teoria e à prática de avaliação e reformulação de currículo. São Paulo: Cortez; Autores Associados, 1988. ; FETTERMAN, 2001FETTERMAN, David M. Foundations of empowerment evaluation. CA/USA: Sage, 2001. ).

In turn, there is the model of Tyler (1986)TYLER, Ralph Winfred. Princípios básicos de currículo e ensino. Tradução de Leonel Vallandro. 9. ed. Rio de Janeiro: Globo, 1986. , which pursuits the achievement of the outlined objectives, and it is the result of educational processes measured by the level of achievement of goals outlined in the program. And the result of the evaluation is used as support for the decision-making process to the educational structure offered.

The model developed by Cronbach (1963)CRONBACH, Lee Joseph. Course Improvement through evaluation. Teachers College Record , New York, v. 64, n. 8, p. 672-683, 1963. does not prioritize unique criteria for the diagnosis of a program, instead it characterizes them as multidimensional results oriented to the decision-making process. Similarly, Scriven (2000)SCRIVEN, Michael. The logic and methodology of checklists . 2000. Disponível em: < https://wmich.edu/sites/default/files/attachments/u350/2014/logic%26methodology_dec07.pdf> . Acesso em: 14 abr. 2018.
https://wmich.edu/sites/default/files/at...
considers that the results of the evaluations are used for various decision-making purposes. According to such orientation, Stufflebeam (2000) defines that evaluation is intended to improve processes and not to prove them, generating data that will assist the decision-making process.

The negotiation-oriented model developed by Stake (1978)STAKE, Robert E. Case study methodology in social inquire. Educational Researcher , Washington, DC, v. 7, p. 5-8, 1978. provides information for groups with different interests and needs to be reported to data managers. Parlett and Hamilton (1976)PARLETT, Malcolm; HAMILTON, David. Evaluation as ilumination: a new aproach to the study of innovatory programs. In: GLASS, Gene V. (Ed.). Evaluation studies review annual . Beverly Hills: Sage, 1976. p. 141-157. understand that the diverse interactions among the actors of the teaching process help to guide on the decisions made to the development of education. Guba and Lincoln (1988)GUBA, Egon; LINCOLN, Yvonna. Effective evaluation. San Francisco: Jossey-Bass, 1988. argue that evaluation is a negotiating process, within the scope of providing information required by their target audience.

Saul’s (1988)SAUL, Ana Maria. Avaliação emancipatória: desafio à teoria e à prática de avaliação e reformulação de currículo. São Paulo: Cortez; Autores Associados, 1988. empowerment-oriented self-evaluation is acknowledged as a collaborative activity of assessing subjects, entities, or communities, in order to improve consolidated programs. For Fetterman (2001FETTERMAN, David M. Foundations of empowerment evaluation. CA/USA: Sage, 2001. , p. 3), this assessment model aims to “assist people on helping themselves and improving their programs, using self-evaluation and reflection”.

In general, Ristoff (2002RISTOFF, Dilvo Ilvo. Apresentação In: DIAS SOBRINHO, José. Universidade e avaliação: entre a ética e o mercado. Florianópolis: Insular, 2002. p. 5-8. , p. 10) states that assessing is

[...] first of all a process of self-acknowledgment, of institutional learning, it is a continuous, permanent, global, democratic process, politically legitimized, technically consolidated, and able to point the strengths, weaknesses, and potentialities of the institution in time and space.

In short, assessment enables managers to streamline decision-making processes with a focus on improving educational programs.

On a different classification, Fernandes (2013)FERNANDES, Domingos. A avaliação em educação: uma discussão de algumas questões críticas e desafios a enfrentar nos próximos anos. Ensaio , Rio de Janeiro, v. 21, n. 78, p. 11-34, 2013. points out that educational assessment can be both formal and informal. For him, in both cases there is the generation of data on the learning of each student. Schmidt (1993)SCHMIDT, Mary R. Grout. Alternative kinds of knowledge and why they are ignored. Public Administration Review, New Jersey, v. 53, n. 6, p. 525-530, 1993. acknowledges that these types of evaluation are complementary in the acquisition of knowledge. Formal evaluation deals with the verification of content related to scientific logic, while informal evaluation focuses on the logic of daily experience reasoning ( HAMMERSLEY, 2003HAMMERSLEY, Martyn. Social research today: some dilemmas and distinctions. Qualitative Social Work , v. 2, n. 1, p. 25-44, 2003. ; POLKINGHOME, 2000).

Educational assessment can also be employed for the (I) learning; (ii) institution; and (iii) large-scale. The first, related to learning, refers to the evaluation of each student in the processes or situations, with the aim of acquiring new knowledge or skill ( BELLONI, 1999BELLONI, Isaura. Avaliação institucional: um instrumento de democratização da educação. Linhas Críticas , Brasília, DF, v. 5, n. 9, p. 31-58, 1999. ). This author stated that this level of evaluation is also intended for the analysis of the syllabus or course outline, of teaching modalities and professional qualification activities. As for the evaluation of institutions, it focuses on policies, projects, and educational institutions, and is considered a tool for improving the administrative and educational management of schools and teaching systems ( BELLONI, 1999BELLONI, Isaura. Avaliação institucional: um instrumento de democratização da educação. Linhas Críticas , Brasília, DF, v. 5, n. 9, p. 31-58, 1999. ). The author also confirms that factors involving the objectives or mission of the institution, the social, economic, political, and cultural contexts should be taken into account in the evaluation of processes and results. As for the large-scale evaluation, its elaboration aims at a wide scope, both of contents and of technical and methodological applicability, characterized by the standardization of data collection and data processing ( COTTA, 2001COTTA, Tereza Cristina. Avaliação educacional e políticas públicas: a experiência do Sistema Nacional de Avaliação da Educação Básica (SAEB). Revista do Serviço Público , Brasília, DF, v. 52, n. 4, p. 89-111, 2001. ).

Table 1 shows a summary of the classification raised for the construction of this section, without, however, exhausting the various modalities and classifications present in the literature on the subject of educational assessment.

Table 1
– Summary of the classification and typology of educational assessment

National Examination of High School Education (ENEM)

Evaluation is acknowledged as a complex process. Evaluations carried out in or outside the school seek somehow to assess the mastery and aptitude of students (levels of learning) in formal education. Thus, one way to objectively and formally measure educational performance is by assessing the student competence in the large-scale external educational evaluations at a national level ( OLIVEIRA; ARAÚJO, 2005OLIVEIRA, Romualdo Portela; ARAÚJO, Gilda Cardoso de. Qualidade do ensino: uma nova dimensão da luta pelo direito à educação. Revista Brasileira de Educação , Rio de Janeiro, n. 28, p. 5-23, 2005. ).

The large-scale evaluation can present formative and summing objectives, as shown in the previous section. In the formative aspect, this type of examination reports to students both the knowledge they acquired and their need to improve in further examinations. From the summing perspective, for instance, the result of the test determines the score required to access the courses available at Colleges or Universities (IES).

In addition, this model of evaluation can have a proactive decision-making function by relating the results found to the context parameters of students and institutions (STUFFLEBEAN; SHINKFIELD, 1987).

The Brazilian government proposes through extensive learning tests a way to evaluate and measure the educational performance (competence) of students and educational institutions. The National Examination of High School Education (ENEM) makes part of the Brazilian system of educational assessment, along with the System of Evaluation of Basic Education (SAEB), the Brazilian Student Performance Examination (ENADE) and the Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES). These examinations / systems can assess the levels of high school, elementary school, higher and postgraduate education, respectively, presenting parameters of the Brazilian educational system ( TRAVITZKI, 2013TRAVITZKI, Rodrigo. ENEM: limites e possibilidades do Exame Nacional do Ensino Médio enquanto indicador de qualidade escolar. 2013. Tese (Doutorado) – Faculdade de Educação, Universidade de São Paulo, São Paulo, 2013. ).

Founded in 1998, ENEM aims to evaluate the level of competence of the student enrolled in basic education, through cognitive axes that seek to identify acquired competences and skills – through interdisciplinary questions on everyday experiences in exams of different areas of study – as well as the ability to comprehend and express their thoughts through writing (ADEODATO; SANTOS FILHO; RODRIGUES, 2014).

Cotta (2001)COTTA, Tereza Cristina. Avaliação educacional e políticas públicas: a experiência do Sistema Nacional de Avaliação da Educação Básica (SAEB). Revista do Serviço Público , Brasília, DF, v. 52, n. 4, p. 89-111, 2001. states that ENEM, through standardized measurement tools, survey, and procedures categorized on a large scale, provides feedback on the development of knowledge, competences, and skills of student enrolled on basic education in pursuit of higher education or a first job.

ENEM uses an array of required skills and abilities, with cognitive axis that define the following fields of study: (i) Languages, Codes, and its Technologies, which assesses a range of 9 competences and 30 skills; (ii) Mathematics, and its Technologies, which assesses 7 competences and 30 skills; and (iii) Natural Sciences and its Technologies, which assesses 8 competencies and 30 skills; and (iv) Humanities and its Technologies, which assesses 6 competencies and 30 skills ( PRIMI et al., 2001PRIMI, Ricardo et al. Competências e habilidades cognitivas: diferentes definições dos mesmos construtos. Psicologia , Brasília, DF, v. 17, n. 2, p. 151-159, 2001. ).

Test results are measured by the evaluation system of Item Response Theory (IRT), which assesses the amount of right answers, assigning a score according to the different levels of difficulty ( QUARESMA, 2014QUARESMA, Edilan de Sant’Ana. Modelagem para construção de escalas avaliativas e classificatórias em exames seletivos utilizando teoria da resposta ao item uni e multidimensional. Tese (Doutorado) – Escola Superior de Agricultura “Luiz de Queiroz”, Universidade de São Paulo, Piracicaba, 2014. ).

In addition to the student’s learning performance, ENEM has been a school parameter that assesses the performance of High Schools in general ( ANDRIOLA, 2011ANDRIOLA, Wagner Bandeira. Doze motivos favoráveis à adoção do Exame Nacional do Ensino Médio (ENEM) pelas Instituições Federais de Ensino Superior (IFES). Ensaio , Rio de Janeiro, v. 19, n. 70, p. 107-25, 2011. ). Thus, educational institutions need to synchronize their syllabus with the principles of interdisciplinary study and contextualization, aimed at the development of knowledge, competence, and skills to integrate relevant areas of study and social inclusion, to provide for student entry in university and job market ( MACENO et al., 2011MACENO, Nicole Glock et al. A matriz de referência do ENEM 2009 e o desafio de recriar o currículo de química na educação básica. Química Nova na Escola , São Paulo, v. 33, n. 3, p. 153-159, 2011. ).

In summary, educational performance, measured by student effectiveness in ENEM, has been the purpose of evaluating basic education egress, meanwhile it guides educational institutions in learning and adapting to the needs of providing the students with the ability to reason, have abilities, and competences needed for the planning of their academic and professional development.

Previous studies that have referred to educational context and performance

Table 2 shows the main results of previous research that have referred to educational context and performance, with the aim to display the literature on the topic, and the introduction of our research to show how it differs, particularly because it focuses on students enrolled in the Brazilian Federal Institutes, and it is based on student effectiveness on the National Examination of High School education (ENEM).

Table 2
– Previous research

In general, the results show that, in fact, educational performance, when objectively measured, can differ depending on the school contexts the students experience. Thus, the results found on this research can be confirmed, or not, by results found in other research works that refer to existing income differences, given the school context of students and schools. Unlike these studies, we have chosen a research object that has not yet been explored (Brazilian Federal Institutes of Education, Science and Technology), besides analyzing data from a quantitative methodological perspective, seeking to provide statistical reliability to the results.

Methodology

This is a descriptive research, considering its objective; as for its procedures, it is documentary. According to Sampieri, Collado and Lucio (2013, p. 102), “descriptive research seeks to specify important properties and characteristics of any phenomenon that is analyzed”. The documentary procedure, according to Martins and Theóphilo (2009MARTINS, Gilberto de Andrade; THEÓPHILO, Carlos Renato. Metodologia da investigação científica para ciências sociais aplicadas. São Paulo: Atlas, 2009. , p. 85), “resembles bibliographic research, but does not raise edited material – books, periodicals etc. – it looks for material that has not been edited yet.” Thus, this present work uses data reported by INEP to describe how school contexts may affect the educational performance of Brazilian Federal Institutes.

The approach to the problem has a quantitative nature, as it uses the data collected to test hypotheses by means of statistical tests in order to establish behavior patterns (SAMPIERI; COLLADO; LUCIO, 2013). Through the statistical Student’s T-Test and variance analysis (Anova), specified at the end of this section, we intended to check for differences in performance affected by the contexts selected for this study.

The outlined sample for this present research can be seen on Table 1 , and it was defined according to the availability of data required to apply the statistical tests, on INEP’s portal. The choice of investigating Brazilian Federal Institutes (IFs) was based on the relevance of performance evaluation (proficiency) of students in a large-scale education network initiated in 2008, as well as by the diverse target audience of IFs students, justifying the comparative analysis of performance from different contexts (see Table 3 ).

Table 3
– Research sample

The following Table 4 shows the operationalization of the contexts selected for this research to categorize the Brazilian Federal Institutes. The contexts expressed in Table 2 are presented by INEP and are usually used as a comparative performance parameter: socioeconomic, region and complexity of the school (ADEODATO; SANTOS FILHO; RODRIGUES, 2014; ALVES; SOARES, 2013ALVES, Maria Teresa Gonzaga; SOARES, José Francisco. Contexto escolar e indicadores educacionais: condições desiguais para a efetivação de uma política de avaliação educacional. Educação e Pesquisa , São Paulo, v. 39, n. 1, p. 177-194, jan./mar. 2013. ; BARROS et al., 2001BARROS, Ricardo Paes de et al. Determinantes do desempenho educacional no Brasil. Pesquisa e Planejamento Econômico , Rio de Janeiro, v. 31, n. 1, p. 1-42, 2001. ).

Table 4
– Categorization of contexts

The educational performance collected from the INEP database is calculated according to Equation (1) below,

D E S E M P i j = Student Individual Mean Total of students ( 1 ) ,

where DESEMP refers to the means assigned at each Brazilian Federal Institute ( i ) for each year ( j ) in: Languages and Codes (LC); Mathematics and Technologies (MT); Humanities (CH); Natural Sciences (CN); Writing (RED); and National Examination of High School Education (ENEM). Thus, the educational performance was calculated by the ratio between the sum of the grades of all ENEM’s formative itineraries and the total of students of each Brazilian Federal Institute. This procedure was carried out for the five periods analyzed (2011, 2012, 2013, 2014 and 2015).

In accordance with the purpose of our research, we have adopted statistical procedures, which are univariate with measures of central tendency and dispersion to describe the sample, as well as tests of mean comparison (Student T-Test and Analysis of Variance) to check for differences in performance due to the context of the Brazilian Federal Institutes.

While the Student T-Test verifies the existence of differences in independent sample means, the Analysis of Variance (Anova) tests the equality of three or more population means, based on the analysis of sample variances. Thus, Student T-Test is applied to compare whether the mean educational performance in ENEM differs according to geographic location. In turn, Anova is applied to compare whether the mean educational performance in ENEM differs according to school size, socio-economic, and regional context.

Data found were grouped in spreadsheets of Office Excel ® and treatment was carried out in Stata® ( Data Analysis and Statistical Software ), version 13.

Analysis and discussion of results

Preliminary, the Kolmogorov-Smirnov normality test was carried out to determine the normality of the data set (865 observations), in order to verify if the sample set analyzed represents the population to which it is related (Brazilian Federal Institutes). Results are presented on Table 5 below.

Table 5
– Kolmogorov-Smirnov Normality Tests

It appears that the data presented normal distribution for the areas of Mathematics and Technologies, and Natural Sciences. Although the other fields of study did not show normal distribution, the normality test for the mean of ENEM indicated low amount or absence of extreme values to mean.

In addition to the normality test, Table 6 shows the changing of the means among the years.

Table 6
– Educational performance: years researched

Increasing differences considered significant for the competence levels assessed by the examination are found mainly in the data by means of comparison tests. The field of Humanities is highlighted, and that fact points out that, over the years, students enrolled in Brazilian Federal Institutes have greatly expanded (high coefficient: Test F = 139.02) the ability to solve questions related to the content required by the test. On the other hand, the field of Mathematics and Technologies has decreasing means; however, it is possible to noticed that this variation decreases (low coefficient: Test F = 31.01) among the years researched, indicating reduced performance throughout the years. The variations between the years in the researched fields of knowledge were enough to cause ENEM grade point averages to show significant and increasing differences. Results can also be checked in Chart 1 .

Chart 1
– Educational performance: researched years

The following analysis on Table 7 refers to the educational performance of the Brazilian Federal Institutes in relation to their regional location.

Table 7
– Educational performance: regional context

The Northeast (336 observations) and Southeast (199 observations) were the regions with the largest number of participants in ENEM. Results found confirm this research hypothesis that the performances show significant differences considering the regional context, agreeing with the results found by Adeodato, Santos Filho, and Rodrigues (2014). Superior performances of the Federal Institutes were predominantly found for the Southeast and South regions, corroborating the findings of Padilha et al. (2012)PADILHA, Frederica et al. As regularidades e exceções no desempenho do Ideb dos municípios. Estudos em Avaliação Educacional , São Paulo, v. 23, n. 51, p. 58-81, 2012. , and that, in general, the means showed low dispersion in each region. Another important point refers to the greatest performances in the fields of Humanities (CH) and Writing (RED), which can be better viewed in Chart 2 .

Chart 2
– Educational performance: regional context

Although the performances of the Brazilian Federal Institutes differ statistically in different regions of the country, this difference is irrelevant, when observed on Chart 2 . Such result may indicate the consequence of an educational policy aimed at guaranteeing education to all Brazilian regions.

In Table 8 , the performance is presented according to the location of the unit, represented by the geographical location context (rural and urban).

Table 8
– Educational performance: geographical location context

The observations (642) show that the participation of the Brazilian school network in ENEM includes a high concentration of students enrolled in the urban area. In this context, there is the low dispersion of the best-performing means for the Brazilian Federal Institutes located in the urban area. ENEM grade averages range in 25 points, the largest difference of which is on the Writing exam, with a difference of 40.6 points. It is also noted that the differences found are considered significant, as evidenced in Chart 3 , which confirms the hypothesis of our research for the geographic location context parameter and thus supporting the results found by Barros et al. (2001)BARROS, Ricardo Paes de et al. Determinantes do desempenho educacional no Brasil. Pesquisa e Planejamento Econômico , Rio de Janeiro, v. 31, n. 1, p. 1-42, 2001. .

Chart 3
– Educational performance: geographic location context

Next, we present the educational performance under the context of school facility size for each Brazilian Federal Institute ( Table 9 ). In this analysis, there was only a difference in the formative itinerary of Mathematics and Technologies, according to the size of the school facility.

Table 9
– Educational performance: school facility size

The number of findings (Facility size C and D), collected from the data, show that the Brazilian Federal Institutes with a great number of students provided stronger participation in ENEM. Unlike the findings of Soares and Sátyro (2008)SOARES, Sergei; SÁTYRO, Natália. O impacto de infra-estrutura escolar na taxa de distorção idade-série das escolas brasileiras de ensino fundamental: 1998 a 2005. Brasília, DF: IPEA, 2008. Texto para discussão, Instituto de Pesquisa Econômica Aplicada (IPEA). , Lee (2008)LEE, Valerie. Utilização e modelos hierárquicos lineares para estudar contextos sociais. In: BROOKE, Nigel; SOARES, José Francisco (Ed.). Pesquisa em eficácia escolar: origem e trajetórias. Belo Horizonte: UFMG, 2008. p. 273-298. , and Alves and Soares (2013)ALVES, Maria Teresa Gonzaga; SOARES, José Francisco. Contexto escolar e indicadores educacionais: condições desiguais para a efetivação de uma política de avaliação educacional. Educação e Pesquisa , São Paulo, v. 39, n. 1, p. 177-194, jan./mar. 2013. , student performances, when analyzed under the context of school facility size, predominantly did not show significant differences – which can be better viewed in Chart 4 –, therefore not confirming the hypothesis of our research. Although results do not converge with the literature, it can be noted that the performances in Humanities and Writing presented indications in favor of smaller facilities. For Mathematics and Technologies, data pointed out that largest facility size favored the learning process of the students, possibly due to the need for greater investments, the specificities, and the focus given in the courses offered by the units, approximating theory and practice.

Chart 4
– Educational performance: school facility size

Finally, educational performance is analyzed, in Table 10 , under the context of the student socioeconomic profile. In general, the analyses were significant, that is, there is a difference between the grades of the formative itineraries based on the socio-economic profile of the Brazilian Federal Institutes.

Table 10
– Educational performance: student socio-economic context

Results show on the many findings in the data that students enrolled in the Brazilian Federal Institutes participating in ENEM are predominantly middle class (socioeconomic profile – medium: 303; medium high: 239; and medium low: 184). It is inferred from the analysis that the performances were superior for the students categorized in the highest socio-economic profile (see also Chart 5 ), according to the works of Alves (2013)ALVES, Fátima. Políticas educacionais e desempenho escolar nas capitais brasileiras. Cadernos de Pesquisa , São Paulo, v. 38, n. 134, p. 413-440, 2013. and Alves and Soares (2013)ALVES, Maria Teresa Gonzaga; SOARES, José Francisco. Contexto escolar e indicadores educacionais: condições desiguais para a efetivação de uma política de avaliação educacional. Educação e Pesquisa , São Paulo, v. 39, n. 1, p. 177-194, jan./mar. 2013. . This can be justified by cultural differences and the level of care of basic needs, provided by the family nucleus. Thus, our research hypothesis is confirmed by the results of such differences.

Chart 5
– Educational performance: student socio-economic context

In general, the educational performances of the Brazilian Federal Institutes proved to be different and significant, determined by factors (contexts) such as regional and geographical location (urban and rural area) and the socioeconomic profile of the students, confirming the hypothesis of our research for these particular cases. As for the context of school facility size, we were not able to determine differences in educational performance. The research also provided a profile of the Brazilian Federal Institutes participating in ENEM with regard to the following contexts: theis participation in ENEM was strongly perceived in the institutions of urban areas in the Northeast and Southeast regions, as well as those with high numbers of middle class students enrolled.

Final remarks

This study aims to investigate to what extent the context of the Brazilian Federal Institutes determines their performance in ENEM. For this purpose, the ENEM exam was chosen as a parameter of educational evaluation, with its summing and proactive characteristics, prompt frequency, decision-making orientation, formal and large scale, aiming to provide managers and the community with data support on the results of educational public policies. It should be noted that the performances mentioned in our work refer to student competence measured in ENEM.

The path of statistical inferences to answer the question raised and test the research hypothesis began with the verification of the distribution of sample normality and with the application of the tests of comparison for the means. The different results did not affect the normality test of the average grade point of ENEM.

As for the regional context, the Brazilian Federal Institutes had better performance rates predominantly in the Southeast and South regions. This can be associated with differences in their structural scenario and student access to school in these regions. It was also clear that the assimilation of knowledge and development of skills and competencies were further enhanced in the field of Humanities, as well as the ability to express thought and comprehension in the Writing. In addition, the context of geographical location shows that these better performances were achieved by Federal Institutes located in the urban areas.

As for the assessment on Brazilian Federal Institute facility size, it did not show relevant rates of performance differentiation. However, student performance on the fields of Humanities and Writing tended to be higher for smaller institutes. On the other hand, the field of Mathematics and Technologies showed better performance for institutes of larger facilities. This fact can be associated with larger investments that favor the cognitive axis of the courses offered, providing, in many cases, approximation between theory and practice due to the specificity of the courses.

Finally, significant and superior differences were found in the performances of Federal Institutes with educators who presented better conditions of meeting basic needs, according to the classification of the socio-economic profile identified. Such results confirm that the social and cultural background of the family is a resolute parameter for the necessary means to improve educational performance.

We point out that, notably, our results show that educational performance is lower for students from rural areas, and low socio-economic class. Thus, it is paramount that new policies are created to tailor educational practices so that performance can be enhanced ( SOUZA et al., 2013SOUZA, Norma Valéria Dantas de Oliveira et al. Perfil socioeconômico e cultural do estudante ingressante no curso de graduação em enfermagem. Revista de Enfermagem UERJ , Rio de Janeiro, v. 21, n. 2, p. 718-722, 2013. ). Besides, welfare policies can also be taken into account, so that these clear performance differences can be assessed by managers and public politicians.

This research has also found a greater number of upper-middle class students enrolled in Federal Institutes that participate in ENEM, strongly perceived in the Brazilian Northeast and Southeast urban units.

In general, this present research contributes to the topic of educational performance and school context, and provides management information to public managers and society on the output of the existing public policies. In addition, it highlights key context parameters of educational performance, measured by the achievements of students in ENEM, which may indicate an urgent need for action on the part of the Brazilian government.

Further study should be carried out to extend this research, namely on: (i) comparative studies between federal, state, municipal and private institutions; (ii) the use of other performance metrics, such as SAEB and IDEB; and (iii) the use of contexts related to financial factors and services offered.

References

  • ADEODATO, Paulo Jorge Leitão; SANTOS FILHO, Maílson Melo dos; RODRIGUES, Rodrigo Lins. Predição de desempenho de escolas privadas usando o ENEM como indicador de qualidade escolar. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 25., 2014, Dourados. Anais... Porto Alegre: SBC, 2014. p. 891-895.
  • ALVES, Fátima. Políticas educacionais e desempenho escolar nas capitais brasileiras. Cadernos de Pesquisa , São Paulo, v. 38, n. 134, p. 413-440, 2013.
  • ALVES, Maria Teresa Gonzaga; SOARES, José Francisco. Contexto escolar e indicadores educacionais: condições desiguais para a efetivação de uma política de avaliação educacional. Educação e Pesquisa , São Paulo, v. 39, n. 1, p. 177-194, jan./mar. 2013.
  • ALVES, Maria Teresa Gonzaga; SOARES, José Francisco. Medidas de nível socioeconômico em pesquisas sociais: uma aplicação aos dados de uma pesquisa educacional. Opinião Pública , Campinas, v. 15, n. 1, p. 1-30, 2009.
  • ANDRIOLA, Wagner Bandeira. Doze motivos favoráveis à adoção do Exame Nacional do Ensino Médio (ENEM) pelas Instituições Federais de Ensino Superior (IFES). Ensaio , Rio de Janeiro, v. 19, n. 70, p. 107-25, 2011.
  • ANDRIOLA, Wagner Bandeira. Evaluación: la vía para la calidad educativa. Ensaio , Rio de Janeiro, v. 7, n. 25, p. 355-368, 1999.
  • BARROS, Ricardo Paes de et al. Determinantes do desempenho educacional no Brasil. Pesquisa e Planejamento Econômico , Rio de Janeiro, v. 31, n. 1, p. 1-42, 2001.
  • BELLONI, Isaura. Avaliação institucional: um instrumento de democratização da educação. Linhas Críticas , Brasília, DF, v. 5, n. 9, p. 31-58, 1999.
  • BOURDIEU, Pierre; PASSERON, Jean-Claude. A reprodução: elementos para uma teoria do sistema de ensino. Petrópolis: Vozes, 2008.
  • BRYK, Antony; LEE, Valerie. A multilevel model of social distribution of high school. In: DARLING-HAMMOND, Linda (Ed.). Review of research in education . Washington, DC: AERA, 1989. p. 171-267.
  • CAMARGO, Raphael Vinicius Weigert et al. Desempenho dos alunos de ciências contábeis na prova ENADE/2012: uma aplicação da Teoria da Resposta ao Item. Revista de Educação e Pesquisa em Contabilidade , Brasília, DF, v. 10, n. 3, p. 332-355, jul./set. 2016.
  • COLEMAN, James et al. Equality of educational opportunity. Washington, DC: U.S. Department of Health, Education, and Welfare, 1966.
  • CORRÊA, Denise Maria Moreira Chagas. Indicadores de gestão do TCU e o desempenho dos cursos no Enade: um estudo de cursos de graduação. Tese (Doutorado) – Faculdade de Educação, Universidade Federal do Ceará, Fortaleza, 2013.
  • COTTA, Tereza Cristina. Avaliação educacional e políticas públicas: a experiência do Sistema Nacional de Avaliação da Educação Básica (SAEB). Revista do Serviço Público , Brasília, DF, v. 52, n. 4, p. 89-111, 2001.
  • CRONBACH, Lee Joseph. Course Improvement through evaluation. Teachers College Record , New York, v. 64, n. 8, p. 672-683, 1963.
  • FERNANDES, Domingos. A avaliação em educação: uma discussão de algumas questões críticas e desafios a enfrentar nos próximos anos. Ensaio , Rio de Janeiro, v. 21, n. 78, p. 11-34, 2013.
  • FETTERMAN, David M. Foundations of empowerment evaluation. CA/USA: Sage, 2001.
  • FRANCO, Creso et al. Qualidade e equidade em educação: reconsiderando o significado de “fatores intra-escolares”. Ensaio , Rio de Janeiro, v. 15, n. 55, p. 277-298, 2007.
  • GADOTTI, Moacir. Qualidade na educação: uma nova abordagem. Produção de terceiros sobre Paulo Freire. São Paulo: Série Livros, 2010.
  • GUBA, Egon; LINCOLN, Yvonna. Effective evaluation. San Francisco: Jossey-Bass, 1988.
  • HAMMERSLEY, Martyn. Social research today: some dilemmas and distinctions. Qualitative Social Work , v. 2, n. 1, p. 25-44, 2003.
  • HANUSHEK, Eric A. Conceptual and empirical issues in the estimation of educational production functions. Journal of Human Resources , Madison, v. 14, n. 3, p. 351-388, 1979.
  • JONG, Nynke et al. Blended learning in health education: three case studies. Perspectives Medical Education , Maastrich, v. 3, n. 4, p. 278-288, sep. 2014.
  • LEE, Valerie. Utilização e modelos hierárquicos lineares para estudar contextos sociais. In: BROOKE, Nigel; SOARES, José Francisco (Ed.). Pesquisa em eficácia escolar: origem e trajetórias. Belo Horizonte: UFMG, 2008. p. 273-298.
  • LIMBERGER, Jane Beatriz. Metodologias ativas de ensino-aprendizagem para a educação farmacêutica: um relato de experiência. Interface , Botucatu, v. 17, n. 47, p. 969-975, 2013.
  • LUCKESI, Cipriano Carlos. Avaliação da aprendizagem escolar: estudos e proposições. São Paulo: Cortez, 2014.
  • MACENO, Nicole Glock et al. A matriz de referência do ENEM 2009 e o desafio de recriar o currículo de química na educação básica. Química Nova na Escola , São Paulo, v. 33, n. 3, p. 153-159, 2011.
  • MARTINS, Gilberto de Andrade; THEÓPHILO, Carlos Renato. Metodologia da investigação científica para ciências sociais aplicadas. São Paulo: Atlas, 2009.
  • NEVO, Daniel. Evaluación basada en el centro: un diálogo para la mejora educativa. Bilbao: Mensajero, 1997.
  • OLIVEIRA, Romualdo Portela; ARAÚJO, Gilda Cardoso de. Qualidade do ensino: uma nova dimensão da luta pelo direito à educação. Revista Brasileira de Educação , Rio de Janeiro, n. 28, p. 5-23, 2005.
  • PADILHA, Frederica et al. As regularidades e exceções no desempenho do Ideb dos municípios. Estudos em Avaliação Educacional , São Paulo, v. 23, n. 51, p. 58-81, 2012.
  • PARLETT, Malcolm; HAMILTON, David. Evaluation as ilumination: a new aproach to the study of innovatory programs. In: GLASS, Gene V. (Ed.). Evaluation studies review annual . Beverly Hills: Sage, 1976. p. 141-157.
  • POLKINGHORNE, Donald. Psychological inquiry and the pragmatic and the hermeneutical traditions. Theory and Psychology , Ontario, v. 10, n. 4, p. 453-479, 2000.
  • PRIMI, Ricardo; HUTZ, Claudio S.; SILVA, Marjorie C. Rocha da. A prova do ENADE de psicologia 2006: concepção, construção e análise psicométrica da prova. Avaliação Psicológica , Campinas, v. 10, n. 3, p. 271-294, 2011.
  • PRIMI, Ricardo et al. Competências e habilidades cognitivas: diferentes definições dos mesmos construtos. Psicologia , Brasília, DF, v. 17, n. 2, p. 151-159, 2001.
  • QUARESMA, Edilan de Sant’Ana. Modelagem para construção de escalas avaliativas e classificatórias em exames seletivos utilizando teoria da resposta ao item uni e multidimensional. Tese (Doutorado) – Escola Superior de Agricultura “Luiz de Queiroz”, Universidade de São Paulo, Piracicaba, 2014.
  • RISTOFF, Dilvo Ilvo. Apresentação In: DIAS SOBRINHO, José. Universidade e avaliação: entre a ética e o mercado. Florianópolis: Insular, 2002. p. 5-8.
  • RUIZ, José Maria. Teoría del curriculum: diseño y desarrollo curricular. Madrid: Universiatas, 1996.
  • SAMPIERI, Roberto Hernandez; COLLADO, Carlos Fernandez; LÚCIO, Maria del Pilar Baptista. Metodologia de pesquisa. 3. ed. São Paulo: McGraw Hill, 2013.
  • SAUL, Ana Maria. Avaliação emancipatória: desafio à teoria e à prática de avaliação e reformulação de currículo. São Paulo: Cortez; Autores Associados, 1988.
  • SCHMIDT, Mary R. Grout. Alternative kinds of knowledge and why they are ignored. Public Administration Review, New Jersey, v. 53, n. 6, p. 525-530, 1993.
  • SCRIVEN, Michael. The logic and methodology of checklists . 2000. Disponível em: < https://wmich.edu/sites/default/files/attachments/u350/2014/logic%26methodology_dec07.pdf> . Acesso em: 14 abr. 2018.
    » https://wmich.edu/sites/default/files/attachments/u350/2014/logic%26methodology_dec07.pdf>
  • SOARES, Sergei; SÁTYRO, Natália. O impacto de infra-estrutura escolar na taxa de distorção idade-série das escolas brasileiras de ensino fundamental: 1998 a 2005. Brasília, DF: IPEA, 2008. Texto para discussão, Instituto de Pesquisa Econômica Aplicada (IPEA).
  • SOUZA, Norma Valéria Dantas de Oliveira et al. Perfil socioeconômico e cultural do estudante ingressante no curso de graduação em enfermagem. Revista de Enfermagem UERJ , Rio de Janeiro, v. 21, n. 2, p. 718-722, 2013.
  • STAKE, Robert E. Case study methodology in social inquire. Educational Researcher , Washington, DC, v. 7, p. 5-8, 1978.
  • STUFFLEBEAM, Daniel L. The CIPP model for evaluation. In: STUFFLEBEAM, Daniel L.; MADAUS, George F.; KELLAGHAN, Thomas. Evaluation models: viewpoints on educational and human services evaluation. 2. ed. Massachusetts: Kluwer Academic Publishers, 2000. p. 279-317.
  • STUFFLEBEAM, Daniel L.; SHINKFIELD, Anthony J. Evaluación sistémica: guía teórica y práctica. Barcelona: Paidós, 1987.
  • TRAVITZKI, Rodrigo. ENEM: limites e possibilidades do Exame Nacional do Ensino Médio enquanto indicador de qualidade escolar. 2013. Tese (Doutorado) – Faculdade de Educação, Universidade de São Paulo, São Paulo, 2013.
  • TYLER, Ralph Winfred. Princípios básicos de currículo e ensino. Tradução de Leonel Vallandro. 9. ed. Rio de Janeiro: Globo, 1986.
  • WORTHEN, Blaine R.; SANDERS, James R.; FITZPATRICK, Jodi L. Avaliação de programas: concepções práticas. Tradução Dinah de Abreu Azevêdo. São Paulo: Gente, 2004.

Publication Dates

  • Publication in this collection
    26 Sept 2019
  • Date of issue
    2019

History

  • Received
    18 May 2018
  • Reviewed
    26 Sept 2018
  • Accepted
    26 Oct 2018
Faculdade de Educação da Universidade de São Paulo Av. da Universidade, 308 - Biblioteca, 1º andar 05508-040 - São Paulo SP Brasil, Tel./Fax.: (55 11) 30913520 - São Paulo - SP - Brazil
E-mail: revedu@usp.br