Effects of schools and municipalities in the quality of basic education

Abstracts

Este estudo, em linha com a literatura sobre os efeitos das escolas, tem como principal objetivo identificar escolas públicas e municípios que contribuem para elevar significativamente os resultados de seus alunos. Para isso, comparamos os efeitos das escolas e dos municípios avaliados com outros indicadores de qualidade educacional, como o Índice de Desenvolvimento da Educação Básica - Ideb - e os gastos municipais por aluno, e também a eficiência das redes de ensino públicas dentro dos municípios. Utilizamos as bases de dados da Prova Brasil de 2005, 2007, 2009 e 2011, e estimamos modelos de regressão hierárquicos, que possibilitam um melhor controle sobre os fatores contextuais que influenciam os resultados dos alunos. A conclusão é que os efeitos das escolas e dos municípios são melhores indicadores da qualidade educacional do que o Ideb. Identificamos inúmeras escolas e municípios que, consideradas as características sociodemográficas dos alunos e do contexto escolar, têm efeitos muito acima do esperado e com eficiência de gastos.

Sistemas de Ensino; Escola Pública; Ensino Fundamental; Efeitos Educacionais

Este estudio, en línea con la literatura sobre los efectos de las escuelas, tiene el objetivo de identificar escuelas públicas y municipios que contribuyen para mejorar significativamente los resultados de sus alumnos. Para ello, comparamos los efectos de las escuelas y los municipios evaluados con otros indicadores de calidad educacional, como el Índice de Desenvolvimento da Educação Básica [Índice de Desarrollo de la Educación Básica] y los gastos municipales por alumno, así como la eficiencia de las redes de enseñanza pública dentro de los municipios. Utilizamos las bases de datos de la Prova Brasil de 2005, 2007, 2009 y 2011, y estimamos modelos de regresión jerárquicos, que hacen posible un mejor control de los factores contextuales que influyen sobre los resultados de los alumnos. La conclusión es que los efectos de las escuelas y los municipios son mejores indicadores de la calidad educacional que el índice de desarrollo. Identificamos un sinnúmero de escuelas y municipios que, consideradas las características sociodemográficas de los alumnos y del contexto escolar, obtienen efectos muy superiores a lo que se espera, y con eficiencia de gastos.

Sistemas de Enseñanza; Escuela Pública; Educación Básica; Efectos Educativos

OTHER ISSUES

^IRetired full professor at the Faculdade de Educação - FaE, and Researcher of the Grupo de Avaliação e Medidas Educacionais - GAME, of the Universidade Federal de Minas Gerais - UFMG (Belo Horizonte). E-mail: [e-mail]

^IIProfessor at the Faculdade de Educação - FaE, and Leader of the School Inequalities Research Center, bound to the Grupo de Avaliação e Medidas Educacionais - GAME, of the Universidade Federal de Minas Gerais - UFMG (Belo Horizonte). E-mail: [e-mail]

ABSTRACT

The purpose of this study, which is based on schools' effects literature, is to identify public schools and municipalities that improve the learning of their students. The effects of schools and municipalities were compared with other educational quality indicators such as the Índice de Desenvolvimento da Educação Básica (Basic Education Development Index) and the per-student municipal expenditures, as well as the efficiency of the municipal public school systems. Data from the 2005, 2007, 2009 and 2011 editions of the Prova Brasil exam were used to estimate hierarchical regression models that allow better control of contextual factors that influence the results of the students. This study shows that the effects of schools and municipalities are better indicators of educational quality than the Basic Education Development Index and identify several schools and municipalities where, given the social-demographic characteristics of the students and of the school context, have effects that are well above the expected, and are cost effective.

Keywords: Educational Systems; Public School; Basic Education; Educational Purposes

EMPIRICAL EVIDENCE GATHERED IN SEVERAL COUNTRIES and in different time periods show that student achievement reflects, in a very direct manner, their social, demographic and cultural characteristics, which synthesize their previous educational experience (BOURDIEU; PASSERON, 2008; COLEMAN et al., 1966). Yet, the question remains about the ability of schools and educational systems to offset, or at least to minimize social determinism, mainly the one impacting students from disadvantaged social origins.

This is the focus of studies that investigate the school context - comprising the characteristics shared by the students enrolled in each school - as well as the instruction quality, infrastructure conditions, the school complexity, and educational system management capacity. These factors should be considered in the search of public educational policies intended to improve the teaching/learning process (HANUSHEK, 1997; LEE, 2008; REYNOLDS; TEDDLIE, 2008; SOARES, 2007).

In this article, it is implicitly assumed that improvement in the effectiveness of the schools can be achieved by the adoption of management policies and instructional methods appropriate to schools from different locations within the country, and that these policies may be found on adequately contextualized examples of success. The main purpose here is precisely to identify exemplary public schools and educational systems. With that in mind, three procedures were adopted.

The first procedure consists in calculating the effects of municipal and state public Basic Education schools on student achievement, as well as the effects of each Brazilian municipality, based on the data collected in the 2005, 2007, 2009 and 2011 editions of Prova Brasil.

The second procedure consists in relating the effects of schools and municipalities to two of the indicators used most commonly to assess educational effectiveness: the Índice de Desenvolvimento da Educação Básica (Basic Education Development Index) - IDEB - and the per-student municipal expenditures. The IDEB is calculated every two years based on data about student promotion rates collected by the School Census and the averages in Reading and Mathematics of the two evaluations carried out by the Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira - INEP (National Institute for Educational Studies and Research): the Prova Brasil, and the Sistema de Avaliação da Educação Básica - SAEB (Basic Education Evaluation System).

The third procedure consists of comparing, for each municipality, the mathematics and reading learning levels of Basic Education

This present paper is organized into four parts: the first part comprises a review of previous studies and a discussion of their results; the second part presents data used and the variables analyzed herein, as well as statistical analytical models; the third part presents the major results obtained; and, the fourth part discusses the issues analyzed.

PREVIOUS STUDIES

For several decades, researchers from various theoretical traditions have been interested in investigating the influence of schools, educational systems and financial resources on the learning of students. Such persistence is due not only to the central role of education in the development of nations in political, economic and social terms, but also to the accumulation of controversial conclusions (BROOKE; SOARES, 2008; VELOSO et al., 2009).

To a considerable extent, the exploration of this theme appeared as a reaction to educational studies carried out during the 1960s and 1970s which explained the differences in performance essentially by inequalities between different groups of students, and not by school inputs such as the availability of libraries, resources and money (COLEMAN et al., 1966; JENCKS et al., 1972). Although research conclusions about the influence of schools on learning indicate that educational systems alone are not able to change the strength of social determination, evidence has been found that some schools are able to provide their students with better learning than expected for their social conditions (LEE, 2008).

In Brazil, since the mid-1990s, data produced by the INEP and other agencies have fostered innumerable studies on the effects of schools and associated factors on school effectiveness.

Since 2007, the education offered by Brazilian municipalities has been monitored using the IDEB - the objective indicator of educational quality), as defined by the MEC (Brazilian Ministry of Education) - which aggregates into a single number the student promotion rate (educational flux) and learning rates, combining data from the School Census and the results of both Prova Brasil and SAEB. One of the goals of the Plano Nacional da Educação - PNE (National Education Plan) - for the current 10-year period is to achieve an IDEB value equal to 6 (in a scale that ranges from 0 to 10), by 2021. That value would be comparable to the current school performance of developed countries that are part of the Organization for Economic Cooperation and Development - OECD - if the Brazilian curriculum were equivalent to theirs. Brazil's current IDEB, for the year 2011, is 5.0 in the early grades of Basic Education, and 4.1 in the higher grades.

Passage of the PNE was preceded by broad public debate about the resources needed in order to reach the goal, expressed in IDEB values. The discussions were supported by the publication of the Custo Aluno - Qualidade Inicial - CAQi (Student Cost - Initial Quality), an indicator developed based on studies from the National Campaign for the Right to Education,

Regarding academic research, analyses of the correlation between quality and financial resources applied to education are quite controversial. In general, economic literature indicates that expenditures for education are not associated with school performance (HANUSHEK, 1997).

In Brazil, Menezes-Filho and Pazello (2007), based on data from the 1997 and 1999 SAEBs, concluded that there is no relationship between the proficiency of public school students and teacher salaries. Nevertheless, they observed that the relative increase in teacher salaries which took place in 1998, by means of the Fundo para Manutenção e Desenvolvimento do Ensino Fundamental - FUNDEF (Basic Education Development Maintenance Fund), had a positive impact on proficiency.

In another study, based on data from the 2005 Prova Brasil and on information about public expenditures in municipal education, Amaral and Menezes-Filho (2008) also observed a lack of relationship between expenditures and student achievement. However, as they investigated this relationship according to the levels of educational quality in the municipalities, they found diverging results. In the municipalities where the students achieved, on average, the highest performance percentages, the relationship between the quality and the expenditures was significant, although with very low values. For these authors, this result suggests that the municipalities that already have better educational quality are more capable of transforming additional resources into improvements in education.

The study on the effectiveness of intergovernmental transfers for Basic Education carried out by Diniz (2012) substantiated this conclusion. The author concluded that, in the relation between effectiveness in the application of educational funding and the educational quality measured by the IDEB, the most significant factors are the students' ascriptive characteristics, family background and qualification of the teachers in the municipalities.

Therefore, there is evidence that the investments made in education do not necessarily produce equitable results, since the effects of the expenditures for quality occur in a selective manner. The relationships found suggest that improvements in educational quality by means of increased expenditures may benefit more certain municipalities or schools. A possible explanation for that may be the unequal distribution of the quality and equality attributes within our educational system (SOARES; MAROTA, 2009).

In line with the discussion, the contribution of this paper lies in associating the education expenditures with other measures of educational efficiency in a manner that has not been tested yet - which will be detailed herein.

The last issue proposed herein - the difference between state and municipal schools - is related to the studies of Leme, Paredes and Souza (2009) and of Ceneviva (2012), which investigated the impact of the municipalization of Basic Education on educational performance. The former discusses the process of decentralizing educational management stimulated by legal mechanisms, introduced during the 1990s, such as the LDB and the FUNDEF. In the empirical analysis, the authors used data from the SEAB and Prova Brasil up to 2005; in order to build school panels at two different points in time, they compared schools that changed from state management to municipal management against schools that remained under the same management. The results showed that the effects of municipalization were negligible: that is, students who went to the schools that were municipalized did not show proficiency levels significantly different from those who went to the schools that remained under state management. For the authors, however, the fact that the study did not find any positive effects of decentralization on proficiency should not be considered a negative outcome. Municipalization may have an impact on other processes not analyzed in the study, such as the system's cost and the increased freedom of local management to set forth goals and incentives.

Ceneviva's study (2012) also questions if the level of management is important in education. Based on data from the SAEB and Prova Brasil up to 2007, the author analyzed the differential in achievement of students enrolled in municipal and state schools. The results did not indicate any difference in proficiency between students of the state system and those of the municipal system, after controlling for external and internal factors that impact achievement, nor between municipal schools and schools that changed from state to municipal management.

Even though the results were not conclusive regarding the advantages of decentralization for the quality of education on school averages, there are examples of local management units that stand out and may contribute to a better understanding of the effect of school on the achievement of students. In order to investigate this issue, the present study analyzes the residuals associated with each school and municipality, identified using statistical models that consider the characteristics of the students, their families and the context of the school.

Hence, instead of comparing groups of schools in a global fashion and the gross results of the proficiencies, the focus herein is on the schools and municipalities that stand out, after the controlling for their differences. The proportion of the variation in the results achieved by the students which may be attributed to the differences between the contexts of schools and municipalities, as well as their educational practices, is taken into consideration. The assumption is that these schools and municipalities present management policies, resource use modalities and pedagogical practices that contribute to student achievement and are, thus, worthy of being acknowledged.

ANALYTICAL APPROACH

DATA

This study uses data from the 2005, 2007, 2009 and 2011 Prova Brasil, that is, from all editions completed.

The four versions of the Prova Brasil gather, as a whole, data from 17,977,489 students. For this paper, students from schools whose identification codes defined by the INEP (INEP Code) were not located, students that showed no proficiency in the two disciplines evaluated, and students who left all items of the contextual questionnaire blank, were excluded. After these filters were applied, schools that did not have at least 20 students per grade, answering the tests, were also excluded, as well as schools not characterized as public institutions.

To highlight the richness of these data for educational research, the descriptive statistics of the variables of students per school year and per version of the Prova Brasil, as well as the progress of their proficiencies, are presented below. This description does not include students from the federal schools, as these have a profile that is closer to that of private schools (ALVES; SOARES; XAVIER, 2012), and the number of students in these schools is quite low (less than 0.5%).

DESCRIPTION OF STUDENT VARIABLES

The calculation of the effect of the school and the municipality should necessarily consider the socio-economic status of the students and also characteristics such as gender, color/race, and school retention. These demographic variables synthesize life experiences that impact on school performance. One must consider not only the individual value of these variables, but also their value on the school as a whole, that characterize the peer effect (WILLMS, 1992). Table 1 shows the descriptive statistics of each one of these variables. Lines with a "No Information" label refer to students who did not answer the item on the questionnaire.

The students assessed are almost equally divided into the two gender groups, as expected. The percentage of girls, a little smaller in the 5^th grade, becomes larger in the 9^th grade - evidence that girls remain in school longer than boys. The number of cases without information is more significant among 5^th grade students, and for all questionnaire variables - which is also predictable - because younger students usually fill-in the contextual questionnaire less accurately. The larger percentage of missing data among 5th graders in 2007, which repeats itself in the following item, may seem to indicate atypical and unexplained behavior in the data registry.

The variable "color" follows the pattern in the demographic studies of the Instituto Brasileiro de Geografia e Estatística (Brazilian Institute of Geography and Statistics) - IBGE. In the contextual questionnaire, each student should choose, from among the five categories, the one with which he/she identifies. In the Prova Brasil sample, the students who declare themselves mestizos or pardos are the majority; this is followed by a considerable percentage that declares themselves whites, and a smaller number that declares themselves blacks; yellows and Indians represent minorities. Compared with the 2010 Population Census, the distribution of this variable presents a slight difference: within the entire population, individuals who declare themselves whites constitute the majority, followed by those who declare themselves mestizos.

Grade gap is defined as the difference between the student's age and the expected age for a certain grade in the regular academic path. In order to compute the grade gap, different calculation algorithms were used, all based on the information available in the contextual questionnaires. The 5^th grade students must fill-in the age field with their full age as of the date of the Prova Brasil exam. For those who did not fill-in this field, the delay was calculated using the failure and dropout variables. Those who answered they were 11 years old or less, or whose computed age was 11 years, were considered regular students. Those who stated they were older than 11 years were considered as having grade gap. 9^th grade students had to provide the month and year of birth, and are also classified into three categories: delayed (one or more years), regular or lack of information. It was observed that the percentage of students with grade gap is significant, and increases from 5^th to 9^th grade, as expected, since there are more opportunities for grade repetition in the latter case, whether due to failing or to dropping out and later re-enrollment. The high number of students who did not fill-in that field in 2009 is due to a serious inadequacy issue of the contextual questionnaire, in the way it was applied, that does not allow verification of the status of each student.

The socio-economic status indicator - SES - results from the aggregation of several ordinal indicators of the contextual questionnaire into a single measure using the Item Response Theory model - IRT - as described by Alves and Soares (2009). The IRT model transforms the information about the parents' academic level, possession of durable assets and domestic services hired in a scale of standard deviations. In order to make its use simpler, the scale was converted to an interval ranging from 0 to 10.

The SES of the students was validated by verifying the association of that indicator with the per capita income of each municipality, obtained from the Demographic Census 2010

Between 2005 and 2009, an increase was verified in the average SES of the students, following the country's economic growth during that period. The same was reported by other indicators, such as household per capita income and per capita Gross National Product (NERI, 2011; SOUZA; LAMOUNIER, 2010). In 2011, however, a slight decrease is observed in the average SES.

PROFICIENCIES

Table 2 synthesizes the descriptive statistics of the students' proficiency levels, that is, minimum and maximum values, means and standard deviations. Between 2005 and 2011, a clear increase was observed in the average proficiencies in both 5^th and 9^th grades, both in mathematics and in reading. It is also evident that the maximum values are higher. The same happened in relation to the minimum values of the 5^th grade. However, in the 9^th grade there was a decrease in the minimum values, which suggests that more students are completing Basic Education without mastering the minimum skills expected at that level of education. In general, however, considering the growing participation each time the Prova Brasil was applied - except for a slight decrease in 2011 in the 5th grade that could have been caused by the smaller number of enrollments in the early years, identified by the educational demography - this result could mean there was an improvement in the quality of education. However, the values shown in Table 2 should be assessed with some type of reference in order to have pedagogical meaning.

The Movimento Todos pela Educação (All for Education Movement), a civic organization focused on improving education in Brazil,

Table 3 shows the percentages of 5^th and 9^th grade students in Basic Education who are above and below the indicated values for both reading and mathematics. The diagnosis is very clear. Although there has been substantial progress, the results are still very far from desired. In other words, Brazilian education, from the perspective of student achievement in reading and mathematics, still has a long way to go.

ASSESSMENT MODEL

The effect of schools on learning is defined in this article, as set forth by Raudenbush and Willms (1995), as the number of proficiency points that may be attributed to the fact of that student going to a specific school. By design, this number assumes positive and negative values and has a zero mean within the set of all schools considered. In other words, it assumes that there are schools which, given their policies and practices, lead the students beyond the expected, whereas other schools are not able to do so.

The empirical test of this idea requires the use of individual student data - after all, they are the ones who learn and it is their learning that is measured. At the same time, it requires that the focus of the analysis be the schools where these students are enrolled and the municipalities where they are located. Therefore, it is broadly accepted in the educational literature that hierarchical multiple linear regression models should be used as the analytical technique. In the present work, three-level hierarchical linear models were adjusted that employ, jointly, data from the four cycles of the Prova Brasil for the two segments of Basic Education (5^th and 9^th grades).

^{Panel 1} shows the variables included in the adjusted models. For the purposes of this statistical analysis, a few dummy variables were created relating to those students who did not answer some of the questionnaire items. These variables have no analytical interest, but they allow students for whom information was not obtained to be maintained in the sample.

In mathematical language, the hierarchical multiple linear regression model used herein is described by the following equations:

Level 1:

Mathematics ijk = π_0jk + π_1jk*YEAR2007_ijk + π_2jk*YEAR2009_ijk + π_3jk*YEAR2011_ijk + π_4jk*SCHOOLYEAR9_ijk + π_5jk*FEMALE_ijk + π_6jk*GENDERNOTINF_ijk + π_7jk*WHITE_ijk + π_8jk*BLACK_ijk + π_9jk*YELLOW_ijk + π_10jk*INDIAN_ijk + π_11jk*COLORNOTINF_ijk + π_12jk*SES_ijk + π_13jk*REGULAR_ijk + _π14jk*DELAYNOTINF_ijk + e_ijk

Level 2:

π_0jk=β_00k+β_01k*SCHOOL5E9_jk + β_02k*SCHOOL9_jk + β_03k*MUNICIPAL_jk + β_04k*PFEMALE_jk + β_05k*AVGSES_jk + β_06k*AVGGAP_jk + r_0jk

Level 3:

β_00k=γ₀₀₀+u_00k

This model translates into statistical language the fact that the student's level of proficiency is associated with the socio-demographic characteristics, both at individual and of the school and municipality levels. Hence, in order to estimate the effect of the school, it is necessary to control the impact of the students' characteristics. The inclusion of a third level - regarding municipalities where the schools are located - allows the calculation of the effects of the municipalities.

The Level 1 model, that is, the student proficiency model, includes, as control variables, personal characteristics of the students: gender, race/color, socio-economic status and school retention. The Level 2 model considers the average proficiency of the students in the school, represented by π_0jk, as the response-variable; and, as explanatory variables, the characteristics of the school's student body that favor learning - such as the average socio-economic status of the students - . Inclusion of this variable in the analytical model prevents the assignment to schools the peer effects. Finally, the third level encompasses the municipalities.

In all three models there is an error term in the equation, e_ijk, r_0jk and u_00k, associated with the students, schools and municipalities. This term may be interpreted as student, school and municipality effects.

RESULTS

EFFECTS OF STUDENTS' CHARACTERISTICS

The results of the estimation of the coefficients of the models for mathematics and reading are summarized in Table 4. The values of the coefficients in the reading and mathematics models are similar, and they are all statistically significant at the 0.001 level, as a result of the large number of students, schools and municipalities involved in this study. Therefore the p-value statistical significance index does not add relevant information to the present analysis.

In general, student proficiency levels are substantially higher in the latest editions of the Prova Brasil than in 2005, the year used as reference. The high value of the coefficient of the variable 9^th Grade only indicates that, as expected, the scores obtained by 9^th grade students are greater than those of the 5^th grade students, used as reference. The variable SES affects both individual and collective proficiency of the students. In the opposite direction, the negative sign of the Grade- gap coefficient indicates that it is not an advantage to be retained, and it is terrible to be in a school with a high proportion of retained students.

Regarding gender, girls show better achievement in reading and boys show better achievement in mathematics; but, in schools where the proportion of girls is larger, all students in general have better achievement in mathematics. In reading, an area where the girls stand out, the greater presence of girls in the school also shows a positive effect, yet the coefficient is not significant. This fact is still not adequately understood and that deserves further study. A possible explanatory hypothesis is that the presence of more girls creates an academic environment in the school that is more orderly and favorable to learning.

Regarding race/color, whites show better achievement than students who declare themselves members of other ethnic groups, but the difference in relation to mestizos is very small and is not pedagogically significant. These results simply reproduce what has already been found and show that socio-cultural characteristics are strongly associated with achievement (SOARES; ALVES, 2003; FRANCO et al., 2007).

However, this study is interested in the analysis of the residuals of the models in which the substantial interpretation is the effect of the school or municipality.

THE EFFECTS OF THE SCHOOL

Chart 1 illustrates the effects of schools that participated in any issue of the Prova Brasil in mathematics. The theoretical average of the effects of the schools is zero per construction, within each municipality and within the set of schools. This is why the chart, presenting the histogram of the effect of all schools on mathematics, has zero as the central point.

The schools on the left side have a negative effect; that is, they are schools whose results are less than expected for the profile of the students and the municipality to which they belong. In many situations, these schools have nominal results considered good, but this is explained more by the privileged characteristics of the students than by the pedagogical practices adopted therein. The schools on the right side of the chart are those with positive effects.

The present work is interested in highlighting schools with large positive effects, as these are the schools with results much greater than expected, considering the students enrolled therein.

Since this paper uses data from a very large number of schools, it was possible to identify Basic Education schools in all states with large effects, that is, greater than 20 points - equivalent to one grade level on the SAEB scale. In all four issues of the Prova Brasil, 704 schools stood out (510 municipal and 194 state schools) with surprising results, due to the excellence of their pedagogical and management practices and not to the characteristics of their students. These schools should be observed systematically by qualitative studies, as has already been done in Brazil (ABRÚCIO, 2010; GAME, 2002; BRASIL, 2010b).

EFFECTS OF THE MUNICIPALITIES

Analogous to the effect of the schools, the residuals of the Level 3 model provide a value for the effect of each municipality on the learning of reading and mathematics. Once again, it is necessary to remember that the average of the effects is zero and their assessment consists of understanding the characteristics of the municipalities with the largest and smallest values. It is important to clarify that the effects of the municipalities is not the same as the effects of the schools located in each municipality. They capture the extent to which the fact that a student living in a municipality and going to a public school in that municipality - and not another one - increases or diminishes his/her achievement. In other words, this effect captures the typical educational environment of the municipality.

The largest positive effects in mathematics occur in very small municipalities. At this point, it is worth highlighting that many such municipalities offer only the first stage of Basic Education, which, in general, have better results in the Prova Brasil. Nevertheless, the performance in the municipality of Cocal dos Alves, in the state of Piauí, deserves special attention as it showed the best effects and is already known for the excellence of its results among other educational evaluations, such as the Brazilian Public School Mathematics Olympics. Furthermore, other small municipalities in the state of Minas Gerais are highlighted. The following stand out among medium-size municipalities, with over 150 thousand inhabitants: Sobral, in the State of Ceará; Patos de Minas, Conselheiro Lafaiete, Ubá and Muriaé, in the state of Minas Gerais; Sertãozinho, in the state of São Paulo; Rio das Ostras and Nova Friburgo, in the state of Rio de Janeiro; and Toledo and Foz do Iguaçu, in the state of Paraná.

The results accrued from the state capitals deserve a specific assessment. The data show that, in the set of Brazilian municipalities, they have below average results. Yet, when the position of these cities is examined in Table 5 below, shown in decreasing order of performance in reading, it becomes clear that several of them have good nominal results due to the fact that they comprise a student body with better socio-economic status - which translates into greater facility for acquiring the reading and mathematics skills evaluated by the Prova Brasil.

The impact of the socio-economic differences on the indicators of educational results in the municipalities is analyzed in Table 6, which shows Pearson's Correlation among five variables: the 5^th grade IDEB, the 9^th grade IDEB, the mathematics effect, the reading effect, and the SEL of the municipality. This last indicator, as mentioned before, is very similar to per capita income.

All the correlations are positive and significant, but it is worth highlighting their magnitude. It is clear that, at the municipal level, the IDEB and the SES are highly correlated to each other - 0.648. This has already been noticed at the school level in other work (ALVES; SOARES, 2013).

The second observation is that, although at a lower value, there is also a substantial association between the IDEB and the effects of municipalities calculated using the model presented earlier. This shows that both types of indicators measure something in common which, naturally, is student learning in reading and mathematics. Furthermore, they both may be considered as indicators of the quality of Basic Education in the municipality.

The third observation is that the effects of the municipalities on reading and mathematics are very weakly associated with the respective average SES (boldface values in the table). In other words, there are many municipalities with low effects in reading and mathematics, but with a high IDEB. The interpretation of this result is that the IDEB is high in these municipalities simply because they have students with a better social-economic status. Therefore, the IDEB does not capture educational quality, but a better economic situation.

MUNICIPAL SCHOOLS: EFFECT, COST AND EFFICIENCY

It is possible to get the annual per-student costs for the municipal schools from data in the publication Finanças do Brasil - Finbra: dados contábeis dos municípios (Brazil Finances: Accounting Data of the Municipalities) (BRASIL, 2012). Although the Finbra basis is the main source of information about expenditures for education, it has several inaccuracies. To obtain a more complete analysis of the association between per-student cost and other educational indicators, the former INEP president, Reynaldo Fernandes (2013), recommends that local purchasing power be considered, and that the values be deflated.

In this paper, the per-student cost of municipal schools was considered to be the largest value observed in the three years of the study, 2007, 2009 and 2001, excluding those municipalities for which no data are available for any of the three years of the analysis, as well as those that reported suspiciously large values - that is, above 15 thousand Reals per year per student. Other analytical options produced similar results.

Table 7 shows the correlation among quality indicators for municipal Basic Education - that is, effects calculated based on the analysis model described earlier - and the 2011 IDEB, as well as the per-student cost in the municipal schools. Considering that, in this case, schools from different municipalities are being compared, the effects for these schools are taken to be the computed effects added to the effects of the respective municipality.

It can be observed that the student cost is simultaneously associated with both the IDEB and the SES. That is, wealthier municipalities have larger student costs and also larger IDEB values, as expected. This result is consistent with the correlation observed between the infrastructure of the schools and the IDEB value, already found in other work (ALVES; SOARES, 2013). However, the correlation between student cost and the mathematics and reading effects is very close to zero. This low correlation indicates that obtaining better learning in reading and mathematics involves other factors in addition to resources, even though they are obviously essential.

An additional result (not included in the table above) reveals that the partial correlation between student cost and IDEB, controlled by the SES of the municipality, is 0.099, a value that is similar to the correlation between student cost and effects. In other words, restricting the analysis only to municipal schools, the result observed is the same one obtained when all public schools of the municipalities are considered, i.e., the high IDEB value of the municipal schools is associated with the socio-economic status of the students enrolled in them.

Adopting the mathematics effects as a measurement of educational efficiency in the municipalities, and restraining the focus to the state capitals, Teresina and Campo Grande are, in this order, the most efficient.

COMPARING MUNICIPAL SCHOOLS AND STATE SCHOOLS

In the North, Northeast and Mid-West regions, the state schools have slightly better effects than the municipal schools. In the Southeast and South regions, except for the state of Rio Grande do Sul, the positions are inverted, but the differences are also slight. However, in a detailed, city-by-city analysis,

DISCUSSION

This article uses data from the four cycles of the Prova Brasil, which constitute the most reliable and most comprehensive empirical basis available, within the scope of Basic Education, in order to identify schools and educational systems whose pedagogical projects produce the best results. In this analysis, the choice was made to analyze Basic Education as a whole and not, as is more common, to consider the 5^th and 9^th grades separately. The concomitantly inclusion as control variables in the model the editions of Prova Brasil and the two grades, included in the tests, allowed the simultaneous consideration of all data, while producing evidence about both segments separately.

The results confirm previous studies on the differences between boys and girls and between students suffering discrimination due to race/color, the impacts of the socio-economic status of the families and of school retention on their achievement. It has been observed that the socio-economic status of a school, and not only that of the student, is very important. The effect is that educational systems which, using different strategies, divide their students into schools based on their socio-economic status, prevent many of them from developing in a more educationally balanced manner. These results once again show that the quality of education cannot be analyzed without considering the characteristics of the students and the context of the schools.

This study focuses on the residuals of the hierarchical models adjusted to the data. In line with the literature in this area, these residuals are interpreted as being the true effects of the schools or municipalities, as they were obtained by controlling for all demographic, social and environmental factors outside the control of the schools (RAUDENBUSH; WILLMS, 1995). Hence, the residuals of the assessment models indicate the impact of these schools and municipalities on the proficiency of their students.

By analyzing the residuals, it was possible to highlight schools and municipalities that fulfill their roles well and contribute to the knowledge acquisition of their students. It was also possible to identify schools that present good results only because they enroll well-prepared students who would perform the same way wherever they studied. Furthermore, it identifies those schools that are able to obtain adequate levels of proficiency from students that theoretically should not be able to achieve them.

The correlation between per-student cost at municipal schools and the respective effect is positive, yet very low. This confirms that improvement in education does not depend only on the amount of resources but also, and above all, on the effectiveness of their use. Since there are inaccuracies in the data about per-student costs, the results obtained should be considered more as a reference for new studies. The fact that there are municipalities where the availability of resources makes a difference is an ongoing issue without a proper response.

The strong association of the IDEB with the averages of the socio-economic status in the schools and municipalities shows that this indicator reflects non-educational conditions very clearly. The results call attention to the fact that using the IDEB as the sole measure of quality of the educational system favors the municipalities and schools with students from better socio-economic status. Implicitly, this signals towards selection and, therefore, exclusion. It also constitutes an alert, considering that this option was incorporated in the current National Education Plan. The measurement of the effects of schools and municipalities introduced in the present work avoids this perverse view and, therefore, should be considered when verifying the success of educational policies.

Finally, analysis of the residuals that indicate the true effects of schools and municipalities contains useful information that may guide later qualitative studies of the schools (or municipal systems) identified as cases of success, as they show effects greatly above the expected ones, with cost efficiency.

REFERENCES

• ABRÚCIO, Fernando Luiz. Gestão da escola e qualidade da educação: um estudo sobre dez escolas paulistas. In: FUNDAÇÃO VICTOR CIVITA. Estudos e pesquisas educacionais 1: estudos realizados em 2007, 2008, 2009. São Paulo: Fundação Vitor Civita, 2010. p. 211-240.
• ALVES, Maria Teresa Gonzaga; FRANCO, Cresco. A pesquisa em eficácia escolar no Brasil: evidências sobre o efeito das escolas e fatores associados à eficácia escolar. In: BROOKE, Nigel; SOARES, José Francisco (Ed.). Pesquisa em eficácia escolar: origem e trajetórias. Belo Horizonte: Editora UFMG, 2008. p. 482-500.
• ALVES, Maria Teresa Gonzaga; SOARES, José Francisco. Medidas de nível socioeconômico em pesquisas sociais: uma aplicação aos dados de uma pesquisa educacional. Opinião Pública, Campinas, v. 15, n. 1, p. 1-30. 2009.
• ______. Contexto escolar e indicadores educacionais: condições desiguais para a efetivação de uma política de avaliação educacional. Educação e Pesquisa, São Paulo, v. 39, n. 1, p. 177-194, 2013.
• ALVES, Maria Teresa Gonzaga; SOARES, José Francisco; XAVIER, Flavia Pereira. O nível socioeconômico das escolas de educação básica brasileiras: versão 2. Belo Horizonte: Grupo de Avaliação e Medidas Educacionais (Game)/UFMG; São Paulo: Instituto Unibanco, 2013.
• AMARAL, Luiz Felipe Estanislau do; MENEZES-FILHO, Naércio. A relação entre gastos educacionais e desempenho escolar. In: ENCONTRO NACIONAL DE ECONOMIA, 36., 2008, Salvador. Anais.. Salvador: Anpec, 2008. Disponível em: <http://www.anpec.org.br/encontro2008>. Acesso em: 1 ago. 2012.
• ANDRADE, Renato Júdice de; SOARES, José Francisco. O efeito da escola brasileira. Estudos em Avaliação Educacional, São Paulo, v. 19, n. 41, p. 379-406, 2008.
• BOURDIEU, Pierre; PASSERON, Jean-Claude. A reprodução: elementos para uma teoria do sistema de ensino. Petrópolis: Vozes, 2008.
• BRASIL. Ministério da Educação. Lei n. 9.394, de 20 de dezembro de 1996. Estabelece as diretrizes e bases da educação nacional. Brasília, 1996.
• ______. Decreto n. 6.094, de 24 de abril de 2007. Dispõe sobre a implementação do Plano de Metas Compromisso Todos pela Educação. Brasília, 2007.
• ______. Parecer CNE/CEB n. 8/2010 Estabelece normas para a aplicação do inciso IX do artigo 4º da Lei n. 9.394/96 (LDB), que trata dos padrões mínimos de qualidade de ensino para a Educação Básica pública. Brasília, 5 maio 2010a.
• ______. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Melhores práticas em escolas de ensino médio no Brasil Brasília, 2010b.
• ______. Secretaria do Tesouro Nacional. Finanças Brasileiras  - Finbra 2011. Planilha para eletrônica publicada em 2012. Disponível em: <http://www3.tesouro.fazenda.gov.br/estados_municipios/index.asp>. Acesso em: mar. 2013.
• BROOKE, Nigel; SOARES, José Francisco. Comentários: seção 3: O que faz a diferença? Os métodos e evidências da pesquisa sobre efeito escola. In: BROOKE, Nigel; SOARES, José Francisco (Ed.). Pesquisa em eficácia escolar: origem e trajetórias. Belo Horizonte: Editora UFMG, 2008. p. 218-224.
• CARREIRA, Denise; PINTO, José Marcelino Rezende. Custo aluno-qualidade inicial: rumo à educação pública de qualidade no Brasil. São Paulo: Global, 2007.
• CENEVIVA, Ricardo. O nível de governo importa para a qualidade da política pública? A municipalização da educação fundamental no Brasil. In. SEMINÁRIO DISCENTE DA PÓS-GRADUAÇÃO EM CIÊNCIA POLÍTICA DA USP, 2., 2010. Disponível em: <http://www.fflch.usp.br/dcp/assets/docs/SemDisc2012/07-3_Ricardo_Ceneviva.pdf>. Acesso em: 10 ago. 2012.
• COLEMAN, James S.; CAMPBELL, Ernest Q.; HOBSON, Carol. J. et al. Equality of educational opportunity Washington, DC: U.S. Government Printing Office, 1966.
• DINIZ, Josedilton Alves. Eficiência das transferências intergovernamentais para a educação fundamental de municípios brasileiros Tese (Doutorado)  - Universidade de São Paulo, São Paulo, 2012.
• FERNANDES, R. Indicador de eficiência dos gastos na educação pública. Relatório: movimento todos pela educação, 2013. Mimeo.
• FRANCO, Creso et al. Qualidade e equidade em educação: reconsiderando o significado de "fatores intraescolares". Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 15, n. 55, jun. 2007.
• GRUPO DE AVALIAÇÃO E MEDIDAS EDUCACIONAIS  - GAME. Escola eficaz: um estudo de caso em três escolas da rede pública do Estado de Minas Gerais. Coordenação: José Francisco Soares. Belo Horizonte: Game/Segrac Editora, 2002.
• HANUSHEK, Eric. A. Assessing the effects of school resources on student performance: an update. Educational Evaluation & Policy Analysis, Washington, DC, v. 19, n. 2, p. 141-164, 2007.
• JENCKS, Christopher et al. Inequality: a reassessment of the effect of the family and schooling in America. New York: Basic Books, 1972.
• LEE, Valerie. Utilização e modelos hierárquicos lineares para estudar contextos sociais. In: BROOKE, Nigel; SOARES, José Francisco (Ed.). Pesquisa em eficácia escolar: origem e trajetórias. Belo Horizonte: Editora UFMG, 2008. p. 273-298.
• LEME, Maria Carolina; PAREDES, Ricardo; SOUZA, André Portela. A municipalização do ensino fundamental e seu impacto sobre a proficiência no Brasil. In: VELOSO, Fernando et al. (Ed.). Educação básica no Brasil: construindo o país do futuro. Rio de Janeiro: Campus-Elsevier, 2009. p. 261-280.
• MENEZES-FILHO, Naércio; PAZELLO, Elaine. Do teachers'wages matter for profďciency? Evidence from a funding reform in Brazil. Economics of Education Review, New York City, v. 26, p. 660-672, 2007.
• NERI, Marcelo. A nova classe média: o lado brilhante da base da pirâmide. São Paulo: Saraiva, 2011.
• ORGANIZAÇÃO DAS NACÕES UNIDAS PARA A EDUCAÇÃO A CIÊNCIA E A CULTURA  - UNESCO. Educação para todos: o compromisso de Dakar. Brasília: Unesco, Consed, Ação Educativa, 2001. Disponível em: <http://unesdoc.unesco.org/images/0012/001275/127509porb.pdf>. Acesso em: ago. 2013.
• RAUDENBUSH, Stephen W.; WILLMS, J. Douglas. The estimation of school effects. Journal of Educational and Behavioral Statistics, Washington D.C./Boston, v. 21, p. 307-335, 1995.
• REYNOLDS, David; TEDDLIE, Charles. Os processos da eficácia escolar. In: BROOKE, Nigel; SOARES, José Francisco (Ed.). Pesquisa em eficácia escolar: origem e trajetórias. Belo Horizonte: Editora UFMG, 2008. p. 297-328.
• SOARES, José Francisco. Melhoria do desempenho cognitivo dos alunos do ensino fundamental. Cadernos de Pesquisa, São Paulo, v. 37, n. 130, p. 135-160, jan./abr.2007.
• SOARES, José Francisco; ALVES, Maria Teresa Gonzaga. Desigualdades raciais no sistema brasileiro de educação básica. Educação e Pesquisa, São Paulo, v. 29, n. 1, p. 147-165, jan./jun. 2003.
• SOARES, José Francisco; MAROTTA, Luana. Desigualdades no sistema de ensino fundamental brasileiro. In: VELOSO, Fernando et al. (Ed.). Educação básica no Brasil: construindo o país do futuro. Rio de Janeiro: Campus-Elsevier, 2009. p. 73-91.
• SOARES, Rosalina Maria. Classificação racial e desempenho escolar 2006. Tese (Doutorado)  - Universidade Federal de Minas Gerais, Belo Horizonte, 2006.
• SOUZA, Amaury de; LAMOUNIER, Bolívar. A classe média brasileira: ambições, valores e projetos de sociedade. Rio de Janeiro: Elsevier; Brasília: CNI, 2010.
• VELOSO, Fernando. 15 anos de avanços na educação no Brasil: onde estamos? In: VELOSO, Fernando et al. (Ed.). Educação básica no Brasil: construindo o país do futuro. Rio de Janeiro: Campus-Elsevier, 2009. p. 3-24.
• WILLMS, J. Douglas. Monitoring school performance: a guide for educators. Washington, DC, London: The Falmer, 1992.

Effects of schools and municipalities in the quality of basic education

José Francisco Soares^I; Maria Teresa Gonzaga Alves^II

¹

Several previous studies analyzed the effects of Brazilian schools and factors associated with the school effectiveness; however, none of them uses such broad and representative empirical evidence as used in this work

²

Municipal expenditures for Basic Education, published annually by the National Treasury Bureau,

³

allow verification and comparison of the amount of resources made available by the municipalities for the maintenance of their educational systems.

⁴

students attending the municipal and state systems. This procedure is justified because, although the

Lei de Diretrizes e Bases da Educação - LDB (Educational Guidelines and Basis Act) (BRASIL, 1996) requires that the municipalities shall prioritize early childhood education and Basic Education and that the states shall prioritize Secondary Education, in many of them this division has not been adopted completely. In such cases, the students are distributed among schools with quite different management and cost models.

⁵

The work of Andrade and Soares (2008) illustrates the potential use of these data to understand the influence of schools on student achievement. The authors analyzed five editions of the SAEB and their first observation is that, in general, the effect of Brazilian schools is very similar for all school years considered. This general observation should not, however, obscure the fact that there is a significant percentage of schools with a negative or positive effect that exceeds 20 points in the SAEB proficiency scale - which is equivalent to approximately one grade level. This indicates that there are certain establishments that deserve qualitative studies in order to learn their specifics, particularly their pedagogical projects. Data used by the authors, however, are sample-based and do not allow identification of these schools. This study continues this line of investigation, adding new issues to the research, as the data now available are far more comprehensive, and allow identification of every school included in the study.

⁶

which sets forth the basic inputs for a quality public education, considering the minimum values for personnel, construction and maintenance of school buildings and the respective infrastructure, the student/teacher ratio, and the school day (CARREIRA; PINTO, 2007). In 2010, the National Education Council established Basic Education quality guidelines in accordance with the CAQi (BRASIL, 2010a).

⁷

The

Prova Brasil comprises tests of the Portuguese language (with emphasis on reading), and mathematics, applied by INEP every two years, in all public schools that have more than 20 students enrolled in the grade level evaluated.

⁸

The results are presented on the same proficiency scale as the SAEB. The methodological resources used to calculate student proficiency allow a comparison between the two scales, as well as between the different versions of the

Prova Brasil. In addition to the tests, the

Prova Brasil includes a contextual questionnaire that requests information about the students (demographic data, school history, studying habits, educational expectations) and about their families (composition, socio-economic and cultural status indicators).

⁹

The final database corresponds to a sample of 15,859,560 students. Therefore, 11.8% of students initially included in the planning of Prova Brasil were excluded because of insufficient information. Hence, the data used in the study are not a random sample. However, considering the large number of cases in the final sample and the breadth of the coverage, it is assumed that the data used in this study adequately describe the situation of Basic Education in Brazil.

¹⁰

This inversion is explained by the fact that the

Prova Brasil does not include private schools, where white students predominate. This calls attention to the percentage of students who declare themselves yellow which, although very small, is greater than expected - which may reflect the difficulty of the students to understand the meaning of that term. This was observed by Rosalina Soares (2006) in her master's degree research on racial classification in Basic Education schools. For many students, as well as for some teachers interviewed by Soares, the classification "yellow" is not associated with Asian origin but to skin color.

¹¹

¹²

¹³

. The correlation between per capita income and the average SES of the municipalities - obtained by aggregating the average SES of the schools of each municipality - is 0.91 (Pearson's Correlation). This high value supports the claim that SES adequately captures the economic conditions of the municipalities, which justifies its use in the statistical analyses for characterizing the schools. Other validations are presented in the work of Alves, Soares and Xavier (2012).

¹⁴

set the goal that 70% of the 5

^th and 9

^th grade students in Basic Education and in the 3

^rd year of Secondary Education, within the system of public and private schools, will have proficiencies above 200, 275 and 300 points, respectively, in Portuguese; and, above 225, 300 and 350 points in mathematics in the SAEB (on a scale from 0 to 500). Values above these reference levels would indicate that the students mastered the expected content and skills for the relevant level of education. This is a widely accepted reference among specialists in educational evaluation, and it used as a criterion for the analysis of performance in several school systems.

¹⁵

in many cases substantial differences were observed (greater than 20 points in the SAEB scale) between the two systems. This fact needs to be acknowledged and considered in public policies, particularly regarding the municipalization of Basic Education, already recommended by the LDB.

Publication Dates

• Publication in this collection
  22 Nov 2013
• Date of issue
  Aug 2013

History

• Received
  Apr 2013
• Accepted
  July 2013