The quantification of quality: some considerations on the school failure rates in Brazil

Abstract

This article presents an analysis of the relations, observable in the Brazilian case, between the school failure rates and the measurement of quality of teaching. The focus is put on discussions about the possibility of quantifying the quality of teaching as of the 20th century. Initially, the polysemy of the term quality and its implications for the quantification processes is discussed. The analysis of documentation allows to see that, in Brazil, the naturalization of the students’ poor school performance coexisted with criticism of the high rates of grade repetition as a mechanism of selectivity and exclusion. The overlapping of school failure rates and the measurement of the quality of teaching is still perceptible today in the Brazilian educational context, as it was observed by the analysis of construction of the IDEB (Basic Educational Development Index) that, since 2007, seek to quantify the quality of the school in Brazil. This index comprises not only the measurement of the knowledge acquired by students, considered not to be a sufficient condition to assess teaching quality, but also the occurrence of a standardised school flow, with a low incidence of failure and dropout.

Keywords
education; statistics; IDEB; education quality; teaching evaluation

Resumo

Neste artigo, apresento análise acerca das relações, observáveis no caso brasileiro, entre os índices de reprovação escolar e a aferição da qualidade do ensino. A opção foi focalizar as discussões sobre a possibilidade de quantificação da qualidade do ensino desde o século XX. Inicialmente, foi preciso considerar a polissemia do termo qualidade e suas implicações para os processos de quantificação. O exame da documentação permitiu ver que coexistiram, no Brasil, por longo período a naturalização do fraco desempenho escolar dos alunos e críticas aos altos índices de reprovação repudiados como mecanismo de seletividade e exclusão. A articulação entre reprovação escolar e qualidade do ensino é, ainda hoje, perceptível no contexto educacional brasileiro, como se observou pela análise do modo de construção do Índice de Desenvolvimento da Educação Básica (IDEB) que, desde 2007, busca quantificar a qualidade da escola no Brasil. Nesse índice, a aferição dos saberes escolares que os alunos dominam não é condição suficiente para considerar o ensino de qualidade. Tem-se assumido também como necessário garantir um fluxo escolar normalizado, com baixa incidência de reprovação e abandono.

Palavras-chave
educação; estatísticas; IDEB; qualidade da educação; avaliação do ensino

Introduction

There is currently an evident concern about the quality of Brazilian education.¹ The media constantly disseminate indexes and rankings that seek to highlight which would be the good schools and rejoice or amaze, too, in reporting what the students of basic education have attained or (often) not regarding knowledge skills and competences.² Academic research and the work of private non-profit organizations have also devoted attention to measuring and/or debating the quality of teaching. If measured by the access to educational institutions, the quality of education would have improved, despite remaining problems. It is positive that 99.2% of young people aged 6 to 14 years have attended school in 2017,³ but it is worth remembering that this means that more than 211 thousand people in this age group were out of school (^{IBGE, 2018}). In 2017, the average years of study of people aged 25 years or more in Brazil was 9.1 years – 10.1 years among white people and 8.2 years for the black population. Considering the progression rates, that is, statistics indicating whether those who enter the system regularly progress the grades established by the legislation, what emerges, in general, is the following table: between the 1st and 5th year of Elementary School⁴ the failure, in 2017, was 5.2%; between 6th and 9th grade, it went to 10.1%; in High School, the failure rate reached 10.8%.⁵ As the failure is significant in Brazilian school, there is another index to which we can refer to: age-grade distortion. Thus, we know that 95.5% of children aged 6 to 10 years attended the initial years of Elementary School at the expected age, with no significant difference between the sexes. In the 11 to 14 age group, the rate was 83.3% of men and 88.0% of women at the age at which they should ideally attend the final years of Elementary School. For those who are more concerned about how much students learn, there are, for example, data from Prova Brasil.⁶ In this case, one possible interpretation, from the score obtained in the standardized tests is that, among the students of the 5th year, 56% present expected learning or beyond the expectations in reading and 44% in Mathematics; among those in the 9th grade, these percentages would be 34% and 15% respectively.⁷

Despite the profusion of numbers, it is not unreasonable to ask: is it possible to quantify quality? What is the limit of statistics in the task of allowing us to know the results of a country’s education? And, after all, what is quality? The issue is thorny, as we will see below, and those who produce and analyze educational statistics in general know this. However, these figures are formulated and presented in such a way as to suggest that quality can be quantified. The perspective is encouraging and, almost always, the animation is enough to close the discussion and move on to the analysis of the numbers. Hence, the difficulty in quantifying quality is considered a minor problem, at most someone makes ritually a mention of recognition of this methodological problem and follows to the fruition of the many advantages that quantitative analysis seems to have. From scientific research to wide dissemination, the problem deteriorates. Because, when it wins the headlines of the newspapers, the “quality” of education has already become a thing, so we have the reification of quality, and we do not come back to discuss about what we are talking about. The numbers gain fluidity and circulate happily, from news to news, allowing good insights, but also many kinds of interpretative abuse.

Reification is only one of the difficulties that need to be dealt with if you want to think about some subject using statistics. Such difficulties do not allow us to advocate the abandonment of quantitative analyses. They are useful – quite useful indeed – and contribute indisputably to the task of giving intelligibility to complex situations, such as education. Perhaps the problem is succumbing to the seduction of numbers and the necessary effort should be to distrust the impression of objectivity they give us. No, the statistics are not objective. They are a useful resource of objectification – which is different – and they carry risks, side effects, which we cannot ignore or disregard.

In the last three decades, a critical literature on quantification has emphasized that statistics are not a simple description of reality, but they are rather the result of objectifying processes defined by the used categories, the choice of what should be measured, the ways of observing (^{Hacking, 1990}; ^{Rose, 1991}; ¹⁹⁹⁹; ^{Porter, 1995}; ^{Desrosières, 2000}; ^{Popkewitz, 2011}, among others). Such processes give the numbers symbolic power and, being produced and used by and for people, also produce devices of subjectivation. ^{Bourdieu (1998, p. 117-118)} points out that

the social order owes in part its permanence to the imposition of classification schemes that, by adjusting to objective classifications, end up producing a form of recognition of this order that implies precisely the ignorance of the arbitrariness of its foundations: the correspondence between objective divisions and classificatory schemes, between objective structures and mental structures, is at the root of a type of original adherence to the established order.

Statistics produce what ^{Bourdieu (1998)} called the “effect of theory”, that is, to the extent that they seek to provide forms of intelligibility of the world, they contribute to the construction of the conditions of existence of what they intend to describe.

It is in the sense of contributing to a critical analysis of education statistics that, in this article, the focus is on ways to quantify the quality of teaching. More especially, what I intended to do was to scrutinize how, in the Brazilian case, school failure was taken as an indicator of the lack of quality of the Brazilian school.⁸ For this, I go back to the analysis at the beginning of the 20th century, when we have for the first time the systematic organization of education statistics in the country. This retreat is explained by the intention to know how the arguments were presented when the democratic school was established in Brazil.⁹ As ^{Bourdieu (2014, p. 103)} suggests, “the interest of the return to genesis is that it is very important because there are debates in the early days when things are said with all the letters that latter appear as provocative revelations to sociologists”.

The text is organized in three parts. In the first one, using the available literature on the subject, although not exhaustively, I analyze the notion of quality in education. The polysemy that characterizes the term makes the task difficult, but even so, such examination is unavoidable if the intention is to quantify the quality. Then, I locate, in the Brazilian educational debates, traces of what was considered as elements articulated around the quality of the school and its measurement. In this sense, it should be noted that the quantification was initially related to the characteristics and capabilities of the students; over time, it was also mobilized to denounce the school’s inability to fulfill its social role. Finally, I present the concepts that guide the construction of the Basic Education Development Index (IDEB), created in 2007, and I present some of its recurring limitations and criticisms. The intention was not to carry out a thorough historical analysis, but to pursue, with methodological rigour, the arguments that coexist, often in a contradictory way, in the debates on the quality of Brazilian education over time.

How to quantify quality in education?

After all, what is quality? This is a dangerous ground. Initially, it is important to state that quality, especially in education, is a polysemic concept. In this regard, the many authors who are dedicated to the examination of the question agree. There are several elements to consider. Without intending to establish a temporal sequence and, even less, any hierarchy, let us begin by noting that it is a historical concept, because it changes over time articulating itself to the values, knowledge and expectations shared socially in different periods. It is frequent that the mention of it suggests a linear and progressive temporal succession, which leads one to believe that the variation of meaning of the term quality of education would establish, in the historical transitions, total changes of meaning. Thus, as we intend to argue throughout this article, the analysis of historical processes leads us not to the understanding of a sequence of senses of quality that would replace their precedents, but to the perception of the coexistence of meanings formulated in different temporalities.

Another aspect to be observed is that quality and quantity are complementary when it refers to education. Several authors point out that the concern with the quality of education does not cancel out the continuous attention that must be given to ensuring access to schooling, embodied in the existence of places in schools in sufficient quantity to meet the entire demand. In this sense, it is worth noting that, as education has been proposed in most contemporary societies as a universal right, quality of education for only a few people is to be considered a privilege and, therefore, lack of quality from the social point of view.

Understanding quality and quantity as articulated aspects makes it possible to identify why, in Brazilian history, as in other countries, the discussion is strongly associated with enrollment totals. But it is not difficult to recall a fact already widely evidenced in several researches: “extending schooling is not, by itself, a guarantee of school justice. In fact, it all depends on how the school is organized and what is done in it” (^{Dubet; Duru-Bellat; Vérétout, 2012}, p. 35). The notion of quality of education associated with ensuring access for all to the same school - a quality school - presents some pitfalls. To argue that everyone has access to a quality school seems a noble and indisputable intention in democratic societies. But in that case, for who is that quality? In what perspective? In line with what project of society? As pointed out by ^{Esteban (2008, p. 6-7)},

quality is a polysemic word, plastic, which contains virtualities and positivities, expresses convergence of concerns, allowing the rapid construction of a consensus by creating the idea of aggregation around common commitments. These characteristics hide how their different meanings keep opposite and contradictory possibilities of school organization as a social project.

By obliterating the polysemy of the term, the sense constituted in certain contexts is assumed to be universal and timeless. Hence, we reinforce the “maintenance of the historical process of coloniality of power in which the relations of subalternization are woven” (^{Esteban, 2008}, p. 7). Regarding the discourses that refer to the rights of “all children”, ^{Popkewitz (2011)} notes that they give rise to practices of identification of those who do not share the characteristics of “all children”, that is, of those who would need to be rescued and saved from their original contexts because they were considered inadequate. When we establish a unique quality standard, we have the production of invisibility and subalternity of people that do not fit the standard (^{Esteban, 2008}).

Thus, the perhaps most crucial aspect for the discussion on polysemy is the notion of that quality is attached to the concept of education considered. That means, different educational paradigms – or, if you prefer, educational projects – will engender different quality criteria and different perceptions about what a quality education is. If it is true that there would be many conceptions of education that can be described, it occurs that two of them are the most recurrent in the debates around the theme. On the one hand, there is the understanding of the function of the school for human formation, on the other, there are arguments that advocate on the function of the school for economic development. In the first case, the school would be associated with the construction of a democratic society. ^{Virginio (2012, p. 179)} highlights the difficulty of ensuring the full accomplishment of this function in face of standardized evaluative criteria:

public policies and the efforts being made to ensure school success can guarantee better performance in terms of expectations of success in relation to certain curricular prescriptions. However, they are insufficient to account for the need for education in a democratic society. It is about thinking beyond the criterion of merit, or even the instrumental character of knowledge.

On the other hand, the function of the school, in the context of globalization and in face of neoliberal policies, strengthens the focus on the preparation of individuals with applicable knowledge and in accordance with economic demands (^{Stromquist, 2012}). Thus, the standardization of school contents and evaluation processes is defended as decisive to guarantee the quality of teaching. In this sense, it is necessary to remember that, in recent decades, the discussion on quality has appeared with special emphasis in business circles and this does not cease to have consequences in the area of education. According to ^{Oliveira and Araújo (2005, p. 7)},

this distinction is important in that it draws attention to the fact that the concept of quality, even in the business world, carries distinct meanings and procedures. Nor should we lose sight of the fact that a significant part of the debate on quality in education is imported from the business world and yet, in this restricted context, it incorporates different meanings.

According to these authors, in some circumstances, the quality of education has been referred to as the “final product”, the learner, or, may also be indicative of a low-cost process, especially low cost for the State.

If there is a lack of agreement on the meaning of quality of education, how could it be quantified? The answer to that question is political, but it is also technical. It is a political answer because it depends on the social group that has the most strength to impose its way of seeing. The elites have greater power to reify quality, according to their school and society conceptions (^{Bourdieu, 2014}). Made “thing”, quality can be quantified, leaving out of the account, inevitably, a wide set of aspects whose importance in the educational process is undeniable (such as the development of empathy and the learning of respect for others, the establishment of links between different generations, the construction of self-esteem, creative capacity, among others). This is an inescapable effect of quantification. As ^{Afonso (2009, p. 13)} argues, “not everything that counts in education is measurable or comparable”. This becomes a problem when one loses sight that the statistical representation of a phenomenon is only a way of seeing, that meet the criteria of power spaces; may be very useful in certain circumstances, but it does not correspond to the totality of what is accounted for. Thus, statistics cannot be a main element in measuring what is important – as if only what is expressed in figures was important.

The answer to the possibility of quantifying quality is also technical, because it implies excluding everything that, even if consensually considered relevant, cannot be expressed in figures: for example, how to objectively measure students’ satisfaction in learning a new knowledge? How to know the development of students regarding self-esteem, social skills and creativity? Or how to measure the importance of the bond between generations established throughout the educational process? The risk is that, because they cannot be expressed in objective indices, we will no longer give attention to them and devote time to those aspects which are essential in the processes of youth training. This shows that the choice of categories in statistical surveys, supposedly a purely technical question, ultimately has an indelible political dimension. As stated by ^{Rose (1991, p. 674)},

paradoxically, in the same process in which numbers reach a privileged status in political decisions, they promise both a “depoliticization” of politics, redrawing the borders between politics and objectivity, intending to act as automatic technical mechanisms to make judgments, prioritize problems and allocate scarce resources.

The lack of consensus and all the complexity that involves the definition of what is “quality of education” did not prevent, however, that the quantification has been done for a long time. Tacitly operating with this multiplicity of questions, there were overlapping conceptions and actions of quantification of the quality of education over time. In the Brazilian case, as we will see below, the measurement dates back to the 1930s and has been perfected amid the debate about what and how to quantify in quality. A brief historical foray allows examining the arguments and noting the origin of the meaning “layers” that exist in the discussion today.

Student quality or teaching quality?

School failure was not always a problem of the Brazilian school (^{Gil, 2018}). Although since the installation of the first colleges, still in the colonial period, there were school exams in which students could fail - practice that remained in the public classes of the imperial period –, such an occurrence is not mentioned in the documentation of both periods as frequent or as a problem. The apprehension of school failure as a problem arises basically from the association of two aspects: on the one hand, the progressive adhesion of Brazilian society to the notion of democratization of schooling as a right and, on the other, production, since the 1930s, of more reliable, comprehensive and regular education statistics.

However, even before this school phenomenon has gained prominence and could be measured statistically, the quantification of students’' performances took its first steps in Brazil. Like in France, some initiatives were organized to apply psychological tests to students in order to identify their abilities and classify them by levels of intelligence. It should be noted that the development of such measurements in France took place at the request of the Ministry of Education to address the problem of pupils lagging behind in school learning by selecting them to be taught in special classes, adapted to their needs. The difficulty in securing the democratic principle of the right to education for all resulted in the demand made by the French government to Alfred Binet, in 1904, to “developing techniques to identify children whose school failure suggested the need for some form of special education” (^{Gould, 2003}, p. 152).

The intelligence tests developed by Binet allowed the creation of scales for measuring IQ (intelligence quotient) and these scales traveled the world, being adapted to different national contexts.¹⁰ In Brazil, such tests were assumed with optimism by the educators at the head of the school’s democratization project, in the 1920s and 1930s, who had full conviction in meritocratic principles and saw school as an institution capable of carrying out the fair selection of innate talents.¹¹ The statistics, in this sense, fulfilled the role of measuring the distance of each individual in relation to the standard. Thus, a spectrum of abnormality was created (^{Lima, 2018}) which, from the performances in psychological tests, served to classify students and engendered subjectivation processes. As ^{Rose (1999)} points out, the statistics integrate the “technologies of the self” collaborating in the establishment of ways of self-government, considered adequate in a given historical moment. ^{Rose (1999)} also points out that statistics are also ways to produce knowledge about students in order to support government actions on specific populations, by proposing strategies for reform and prevention of behaviors considered undesirable in the exercise of the micropower.

The Brazilian educators who occupied prominent positions in the institutions of educational policies, in the period, showed strong adherence to liberal ideals. In the first decades of the 20th century, they vigorously defended the democratic school, comprising by this expression a school in which absolutely all children were guaranteed enrollment, whether they were boys or girls, poor or rich. Guaranteed to enroll everyone, the school would select, by its own processes, the best of each generation. They would be provided with a long education, so as not to waste their talents. To the others, there would be courses of preparation for the work, leading each person to the position befitting his or her abilities, so that each one would give his or her best for the progress of the nation. As an example, note what states Teixeira de Freitas, one of the founders of IBGE (Brazilian Institute of Geography and Statistic) and who, for many years, was at the forefront of the statistics produced by the Ministry of Education and Health: “To be fair, education must be extended to all citizens, without any privilege, except the natural gifts of personal receptivity” (^{Freitas, 1945}, p. 348). ^{Jardim (1946, p. 460, emphasis added)}, from the Statistical Service of Education and Health of the Ministry of Education and Health, in line with its contemporaries, considered the tests a fundamental resource for the organization of education:

the initial classification for the class groupings and application of educational programmes, etc., and periodic evaluations of the intimate process of educational work and its practical effects, considering the extent and composition of those programmes, in addition to other objective tests, mental tests and educational or educational tests are adopted, those for the verification and measurement of mental qualities, these for the verification of the students' performance in the various disciplines.

In this sense, it should be noted that, in the early debates, the notion of quality is associated with the students' innate abilities. In other words, the problem of the low performance of the Brazilian school, expressed in high rates of failure and dropout, would be, at least in part, due to the lack of quality of the children who arrived at the school banks. There was an evident naturalization of low school performance in Brazil, sometimes even expressed as inevitable. For example, ^{Cardoso’s statement (1949, p. 74, original emphasis)}:

the new student appears as an unknown; offers, always, field to the development of hopes: maybe it is strong worker, element that fits well to the group, interested in school activities, compensating for the efforts spent. The repeater is a failure – already been revealed, of him nothing is expected. Is it really worth keeping him in school, filling a vacancy and preventing, with his presence, the admission of another student?

But there is also, in the period, a specific debate on measuring instruments. The criteria considered by the teachers during the examinations differed from one school to the other. So, for example, to explain the approval rating in 1935, a little more favorable in some educational institutions, Almeida Junior, who held the position of manager of the state education system in São Paulo, mentioned the variation in test requirements as an explanation for differences:

reprobations are exact numerical data; but the criteria that determine them, as we well know, change from school to school. Will the 60% promoted in school A be equivalent, in level of education, to the 60% of school B? Maybe yes; maybe not. Who examines is the school authority – the inspector or the director - and as there are authorities undemanding, who settle for little, also there are very demanding, willing to tighten the screening of approvals

(^{São Paulo, 1936}, p. 3).

There is also mention of the recognition that, as school approval was based on exam results, it was possible for teachers to train students specifically for exams as a way of improving their scores. Almeida Junior stated that “the fact that we focus attention on the phenomenon of failure does not mean that the only thing we are asking schools to do is to prepare students for exams” (^{São Paulo, 1936}, p. 3).

Over the period, the analysis of the debates on Brazilian education allows to see the emergence of a discussion that displaces the cause of disapproval: from the evidence of the low capacity of the student it becomes possible also that it would be the indication of the inability of the school to fulfill its social role. Let us move on to that debate.

Already since the 1930s, Teixeira de Freitas indicated that the exclusive concern with the expansion of the school, verified by the increase in enrollment, was insufficient. According to him, the main problem of the Brazilian school was its low efficiency, since the school had difficulty in attracting the population at school age, as well as in maintaining attendance and ensuring students' approval from one grade to the next. He was against the uncritical expansion of a school that, in his opinion, needed to be improved. That’s why he thought

[...] the slogan that best expressed the claims of Brazilian childhood in primary education would have to be, in our view, this: “first of more schools, best school”. Understood as such, an inviting School that retains, protects, teaches and educates truly the infants entrusted to it

(^{Freitas, 1946}, p. 43).

In addition, ^{Anísio Teixeira (1935, p. 74)}, who was Director of Public Instruction in the Federal District, emphasized that the school failed to fulfill its social function by failing and driving away a significant amount of their student bodies:

it is not enough to have schools for the most capable; it is essential that there be schools for all. It is not enough to have schools for everyone, it is essential for everyone to learn.

It is not difficult to assess how much the change came to influence the concept of school performance. Before, given the selective character, failure was almost the index of the quality of teaching. If many failed, this meant that the criteria of judgment were really efficient and the fine flower of the population was being purified for the formation of the intellectual and professional elites.

If, however, the school has a duty to teach everyone, because everyone needs the fundamental elements of culture to live in modern society, the problem is reversed. Failing student means no longer success of the sorting apparatus, but failure of the institution of fundamental preparation of citizens, men and women, for common life.

But this debate will only echo in the area of education from the 1950s, when criticism of the high levels of school failure increases, sometimes associated with explanations for school failure that blamed students and their families (^{Patto, 1993}), in other times the discussion was about finding a solution to the problem. In this sense, it is interesting to note that the Revista Brasileira de Estudos Pedagógicos, official journal of the National Institute of Educational Studies Anísio Teixeira (INEP) will serve as a space for the analysis of the advantages and risks of automatic promotion in Brazil (^{Fernandes, 2000}). The arguments in favor pointed to the recognition that education as a right of all could not be anchored in selective pedagogical practices. Luiz Alves de Matos, for example, in an article published in 1956, pointed out “that the elementary school is a school guaranteed by law and that it should not become an agency that selects privileged talents, but should be a diffuser of education and culture at the service of youth and democracy” (^{Fernandes, 2000}, p. 82). The opposing arguments did not diverge from the principle of the democratic school but warned to the risks of a hasty action that could be inadequate to the Brazilian context. In this sense, Luiz Pereira, in a 1958 article, cited by ^{Fernandes (2000, p. 84)}, denounced

that the repetition is a consequence of a series of serious problems and that the automatic promotion, although "eliminating the high percentages of repetition would not directly and profoundly affect the factors of this phenomenon and that would lead to the loss of a valuable thermometer of the functioning of the primary school system – the repetition rates.

Such debates gave rise to policies that adopted alternative forms of evaluation and arranged student flow aiming to reduce school exclusion (^{Mainardes, 1998}). From the 1960s on, reaching its peak in the 1990s, the discussion on school flow problems stands out, with emphasis on denouncing the marginalization and exclusion of the poorest. Some important public-school systems began to adopt cycles or continuous progression in the organization of their schools, abandoning the annual class progression that provided for the possibility of failure at the end of the year if the student did not score enough in the evaluations performed by each teacher. The elimination of reprobation, however, raised questions about whether it would not be just a way to “formally solve school failure (failure rates), but not the real problem - that of learning of the students [...]” (^{Mainardes, 1998}, p. 25). In other words, these policies resulted in a significant reduction in the disapproval rates in the educational systems in which they were implemented, but they suffered several criticisms and generated mistrust about the ability to maintain the quality of teaching.

The end of the 20th century was also the period when neoliberal policies were widely disseminated. It was important to achieve maximum efficiency with the minimum public investment. Although states still want the school to ensure some level of social cohesion, in this new context, they tend to reduce their participation in the provision of public education (^{Stromquist, 2012}). Alternatively, they focused on privatization, decentralization, and accountability processes. Among the control strategies that characterize neoliberalism, great importance has been given to large-scale assessment, which aim to measure the quality of teaching as a way to induce its increase, for example, by the competitiveness raised between schools.

Quality measurement: the IDEB as a measuring instrument

Since the 1990s, guidelines on education produced by international organizations have accentuated the notion of equity as an essential aspect (^{Klein, 2017}). The World Conference on Education for All, held in 1990 in Jomtien (Thailand), reiterated education as a fundamental right of all and the World Declaration on Education for All: meeting basic learning needs, signed by the participating countries, established that “basic education should be provided to all children, young people and adults. To this end, it is necessary to universalize and improve quality, as well as take effective measures to reduce inequalities” (^{Unesco, 1990}, p. 4). The intention to guarantee a “minimum quality standard for learning” was linked to the recommendation to “implement performance assessment systems” (^{Unesco, 1990}, p. 4).

Brazil followed this movement and, since that period, began to build instruments to measure the quality of teaching that intended to measure what students learned in school. In 1991, the Ministry of Education established the System of Assessment of Basic Education (SAEB) whose objective was to measure the quality of teaching by sample evaluation, held every two years, of the performance of students in standardized tests (^{Bonamino; Sousa, 2012}). In 1998, the National High School Exam (ENEM) was created and, in 2005, the Prova Brasil was added to the system, establishing non-sample forms of evaluation that would allow individualizing the results per school. More than providing parameters for measuring the teaching performance, these policies aim to direct the curricular contents and pedagogical work. ^{Bonamino e Zákia Sousa (2012, p. 380, emphasis by the authors)} analyzed that

the strategy of the dissemination media, through rankings, although unofficial, along with the distribution in schools of the content and skills matrix used in the preparation of the tests of Portuguese language and mathematics, introduces concrete perspectives of more direct interference in what schools do and how they do it.

However, as in the Brazilian case the guarantee of access to basic school occurred only recently and the conclusion of studies is not assured for all students yet, the concerns about the measurement of learning could not be disarticulated from the measurements of failure and dropout,¹² phenomena still very recurrent in the country. In 2007, therefore, the INEP created an index that associated both aspects. As stated by ^{Fernandes (2007, p. 7)},

an educational system that systematically fails its students, causing a large part of them to drop out of school before completing basic education, is not desirable, even if those who complete this stage achieve high scores on standardized exams. On the other hand, a system where students complete high school in the correct period is not of interest if they learn very little. In short, an ideal system would be one in which all children and adolescents had access to school, did not waste time on repetitions, did not drop out of school early and, at the end of everything, learned.

In view of this, at that time, the Basic Education Development Index (IDEB) was proposed, whose formula combines information on the performance of students in standardized examinations of Portuguese Language and Mathematics and on output, by the approval rates. With regard to performance, the data comes from the Prova Brasil applied every two years for students enrolled in the 5th and 9th grade of Elementary School and in the 3rd year of High School. Regarding output, the approval rates allow to measure problems in school flow, since the calculation is made by the difference between those enrolled at the beginning of the year and those approved, so allowing to know the quantity of failure and dropout. Thus, a school whose students have good proficiency in the Prova Brasil, has its IDEB penalized if it has high rates of failure and dropout. It highlights, in the conformation of the instrument, the concern with the fulfillment of the legal precept of education as a right of all. It is also consistent with what the National Education Plans (2001-2010 and 2014-2024) have been proposing by pointing out, among the goals, the demand for regularization of school flow, aiming to progressively make failure and dropout rare events. In 2014, the PNE started to include quantitative goals, however, without specifying the basis from which the averages were established. Detached from the measurement of objective conditions in order to reach the indicated goals, the numbers become random and express desires disconnected from reality.

Table – Nacional Education Plan 2014-2024 – projected averages
Goal 7: Foster the quality of basic education at all stages and modalities, with improved school flow and learning to achieve the following national averages for Ideb

IDEB	2015	2017	2019	2021
Initial years of Elementary School	5,2	5,5	5,7	6,013
Final years of Elementary School	4,7	5,0	5,2	5,5
High School	4,3	4,7	5,0	5,2

• Source: PNE 2014-2024

The National Education Plan goal 7 states that it is intended to establish, for Brazil, tangible objectives in terms of quality of education and enable accountability, for which quantification is considered fundamental.

The IDEB was proposed as a way to establish a quality standard in the country’s education. It is an index intended to monitor the results per school, aiming to provide elements to the managers of educational systems, information supposedly accessible to families about the institutions in which their children are enrolled, as well as elements for evaluating Brazilian education as a whole. Therefore, the index is intended to serve as a beacon for educational policies as well as for social control over the quality of teaching. The specialized literature has, however, pointed out some of its risks and limits, many of which stem from the inadequate uses made of the numbers. A central aspect, in this sense, is that educational indices foster the illusion that it is possible to objectively know the quality of educational institutions. Questions are asked about the

use of the results of the large-scale assessment and the indexes created by them as the only source for the analysis of the work carried out by the schools, considering that the average performances obtained cannot be translated as a true portrait of the quality of the institutions

(^{Almeida; Dalben; Freitas, 2013}, p. 1155).

Furthermore, there has been unanimous criticism of the fact that the index disregards the socioeconomic level of students, leading to believe that schools with high IDEB would have a better education than the others, when it is known that the correlation between student performance and socioeconomic level is always strong.

Hence, by assigning to this indicator the status of the synthesis of the quality of education, it is assumed that the school can overcome all the exclusion promoted by society. There’s a lot of literature that shows that’s impossible. All students have the right to learn, and the knowledge and skills specified for basic education should be the same for all. However, obtaining this learning in schools that serve students who bring less from their families is much more difficult, a fact that should be considered when using the indicator of learning to compare schools and identify successes

(^{Soares, 2011}).

It is also necessary to bear in mind that the IDEB does not allow the assessment of aggregated value (i.e., the difference in proficiency between the beginning and the end of schooling) and that standardized proficiency tests do not evaluate important contents to students' education, because they are restricted to reading and mathematics. Some essential topics are left out, for example, components such as Geography, Art and Sciences or, still, skills such as valuing the diversity of knowledge, the care of oneself, others and the planet, the collaboration with the construction of a just, democratic and inclusive society (^{Brazil, 2018}, p. 9).

In spite of these weightings, widely known and considered by statisticians and researchers who focus on the subject, the circulation of numbers comes out of these reading keys and the figures gain autonomy. The effects of the indices extrapolate their initial propositions.¹⁴ Thus, for example, the reception of the index by schools can result in actions whose only objective is to improve the index and not necessarily to improve the quality of pedagogical work. ^{Soares e Xavier (2013, p. 915)} point out that

the use of an indicator as a single measure of the quality of the school and of the systems will, of course, make the schools seek to maximize it and, as this can be done in unsuitable ways pedagogically, can lead to a dysfunctional educational system.

It is therefore necessary to consider the effects of subjectivation that the IDEB causes in the school space (^{Klein, 2017}). Establishing a set of aspects from which quality is inferred, even if it is methodologically relevant, comes across the fact that the reception of the index ends up limiting efforts only in relation to this restricted set. That is, if the verified proficiency is limited to reading and mathematics, it is not necessarily because the formulators of the index consider that this is the unique task of the school, but because this would be a sufficient parameter for a standardized calculation. However, the actors of the school, the managers of educational systems and the large press tend to receive this methodological restriction as a pedagogical prescription about the curricular contents with which they must deal. Such effects escape the control of statisticians, causing the measuring instrument to act in the construction of what it was supposed to measure. Less than a distortion of operation, this is one of the characteristics of the quantification processes.

Final considerations

To assume education statistics as an object of analysis, seeking to scrutinize the way they are constructed and interpreted, corresponds here to consider them as participants in educational processes – and not mere measuring instruments. As far as the quantification of quality is concerned, the difficulty already begins in defining what quality education is. Far from being a simple and consensual issue, what I sought to emphasize was the fact that such conceptions are the result of socio-historical processes that, amid constant disputes, engender the coexistence of meanings that do not always agree. These meanings, in turn, are mobilized in the current debates, with varying weight and effectiveness.

The second element that I wanted to highlight in this article was the understanding, resulting from the analysis of historical sources, that, often, what is considered as an assessment of the quality of teaching is the examination of students’ capabilities. Here it is necessary to clarify an important sociological question, addressed by Bourdieu in several of his works. The analysis of the processes of quantification of school failure has been contributing to

describe the objective mechanisms determining the continued elimination of disadvantaged children. It seems, in fact, that the sociological explanation can completely clarify the differences of success that are most often attributed to differences of gifts. [...] The cultural heritage, which differs [...] according to social classes, is responsible for the initial difference of children in relation to school experience and, consequently, for success rates

(^{Bourdieu, 2012}, p. 41-42).

The tradition of evaluating the quality of teaching by measuring what students know is maintained in the IDEB. Although this index seeks to balance the verification of students' performance in standardized tests with quantitative data on school flow, the focus is still limited to the students. There is no mention of the infrastructure conditions of schools, no consideration of the training and remuneration of teachers and other education workers, no account is taken of public and/or private expenditure per student, that is, the socioeconomic level of families and the resources allocated to education are not included in the calculation. This way of quantifying quality suggests an excessively simple equation, leaving out the important part of the complexity of the social structure that permeates and constitutes the school institution.

Hence, one of the effects of quantifying quality that I wanted to emphasize in this article is the excessive simplification of the question of quality in education expressed by statistics. If, on the one hand, numbers are a useful resource of objectification that assists in the analysis of complex situations and in political and educational decision-making, on the other hand, they induce to assume that everything that matters would be expressed statistically. As highlighted by ^{Desrosières (2000, p. 9)}, it is necessary to understand how social facts are transformed into statistics, taking into account that “the history of its gestation allows to sketch, retracing the old controversies and debates, a space of articulations between the technical languages and their uses in the social debate”. From my point of view, we should not abandon the use of numbers, but we must be much more attentive to their risks and limits (^{Gil, 2019}).

Since quality education is a right guaranteed by Brazilian legislation, it is necessary to take into account another fundamental aspect. In an enthusiastic "objectivity" society, to count something is, as already mentioned, by supposedly describing it, to make it exist. Hence a dilemma: how to guarantee the right to education without measures, without parameters, without statistics? Therefore, some authors argue that it is essential to establish objective standards of quality in education (^{Oliveira; Araújo, 2005}). The issue is not of little importance, but its positive aspect does not cancel out the risks already pointed out. The space for clashes - of ideas and policies – is, of course, open. Understanding statistics as a mobilized element in this game is already a less restricted way of taking it into account.

• 1
  I thank Célia Caregnato for carefully reading the preliminary version and for the valuable suggestions. The problems and incompleteness of this study, of course, are my full responsibility.
• 2
  For example, see “Brasil é segundo país com pior nível de aprendizado, aponta estudo da OCDE”, published on 10/02/2016 and available at https://educacao.uol.com.br/noticias/agencia-estado/2016/02/10/brasil-e-segundo-pais-com-pior-nivel-de-aprendizado-aponta-estudo-da-ocde.htm and “São Paulo só tem uma escola no top 10 do Enem de 2019”, published on 27/06/2020 and available at https://noticias.uol.com.br/ultimas-noticias/agencia-estado/2020/06/27/sp-so-tem-uma-escola-no-top-10-do-enem-de-2019.htm.
• 3
  Data available at https://agenciadenoticias.ibge.gov.br/agencia-sala-de-imprensa/2013-agencia-de-noticias/releases/21253-pnad-continua-2017-numero-de-jovens-que-nao-estudam-nem-trabalham-ou-se-qualificam-cresce-5-9-em-um-ano. Accessed on 07/11/2020.
• 4
  In Brazil, Basic Education is divided into three levels: Early Childhood Education, for children from 0 to 5 years; Elementary School, with a duration of 9 years; and High School, lasting 3 years.
• 5
  Data available at http://portal.inep.gov.br/artigo/-/asset_publisher/B4AQV9zFY7Bv/content/inep-divulga-taxas-de-rendimento-escolar-numeros-mostram-tendencia-historica-de-melhora/21206.
• 6
  “The Prova Brasil and the National System of Assessment of Basic Education (Saeb) are evaluations for diagnosis, on a large scale, developed by the National Institute of Educational Studies Anísio Teixeira (INEP/MEC). They aim to evaluate the quality of education offered by the Brazilian educational system from standardized tests and socioeconomic questionnaires. In the tests applied in the fourth and eighth class (fifth and ninth grade) of elementary school, students respond to items (questions) of Portuguese language, with a focus on reading, and mathematics, with a focus on problem solving. In the socioeconomic questionnaire, students provide information on contextual factors that may be associated with performance”. Available at http://portal.mec.gov.br/prova-brasil.
• 7
  According to Qedu’s interpretation of the Lemann Foundation. Available at https://qedu.org.br/brasil/proficiencia?gclid=EAIaIQobChMIovKGq7rF6gIVVQmRCh1BygDbEAAYASAAEgLfCvD_BwE.
• 8
  In common sense, however, a high rate of disapproval is often considered as an expression of quality. Even many teachers continue to think in this way, despite the fact that the official documents and the specialized literature for more than a century indicate the opposite. On this, however, I will not deal with in this article, although it is a very important issue and is still little studied.
• 9
  In the 1930s, liberal educators advocated for the expansion of the democratic school, they understand it as an institution that would guarantee equality of opportunity for all and proceed to a selection considered fair because supported by the innate talents of students. Such defenses had a strong influence on the way the Brazilian school was established in the legislation.
• 10
  For more in-depth insights into the conceptions and criticisms of IQ testing, see ^{Gould (2003)}.
• 11
  It is important to note that the understanding of innate talents as a fair resource for the selection and classification of students will have a long life in educational debates. For a critique of the ideology of natural aptitudes, see ^{Bisseret (1974)}.
• 12
  The INEP conceptualizes school dropout as the situation of the student who stopped attending school before the end of the school year, without formalization of transfer request. Available at http://download.inep.gov.br/educacao_basica/censo_escolar/nota_tecnica/2015/nota_tecnica_indicadores_de_rendimento_2012.pdf.
• 13
  Although IDEB has a scale from 0 to 10, it does not behave as in the traditional school evaluation. Depending on how the calculation is done, 6,0 corresponds to a high IDEB and the difference between 4,0 and 6,0 is greater than it may appear. Notes below 2.0 and above 8.0 are extremely rare. For more information on technical aspects, see ^{Soares e Xavier (2013)}.
• 14
  It should be noted that there are differences between what the technical teams of production of statistics intend of the numbers and what the managers intend when ordering the creation of indexes. For them, it is often more explicit the objective of inducing the policies provided by the dissemination of statistics that, in the technical team speeches, appears more as distortion or abuse in the use of numbers. This aspect has not been systematically verified here, but it is an important issue that deserves more accurate studies.

Referências

• ¹
  AFONSO, Almerindo J. Nem tudo o que conta em educação é mensurável ou comparável. Crítica à accountability baseada em testes estandardizados e rankings escolares. Revista Lusófona de Educação, n. 13, p. 13-29, 2009.
• ²
  ALMEIDA, Luana C.; DALBEN, Adilson; FREITAS, Luiz Carlos de. O IDEB: limites e ilusões de uma política educacional. Educação & Sociedade, v. 34, n. 135, p. 1153-1174, 2013.
• ³
  BISSERET, Noële. A ideologia das aptidões naturais. In: DURAND, José Carlos. (org.). Educação e hegemonia de classe Rio de Janeiro: Zahar, 1974. p. 30-67.
• ⁴
  BONAMINO, Alicia; SOUSA, Sandra Z. Três gerações de avaliação da educação básica no Brasil: interfaces com o currículo da/na escola. Educação e Pesquisa, v. 38, n. 2, p. 373-388, abr.-jun. 2012.
• ⁵
  BOURDIEU, Pierre. A economia das trocas lingüísticas: o que falar quer dizer. São Paulo: EDUSP, 1998.
• ⁶
  BOURDIEU, Pierre. A escola conservadora: as desigualdades frente à escola e à cultura. In: NOGUEIRA, Maria Alice; CATANI, Afrânio (Orgs.). Escritos de educação 13. ed. Petrópolis: Vozes, 2012. p. 39-64.
• ⁷
  BOURDIEU, Pierre. Sobre o Estado: cursos no Collège de France (1989-92). São Paulo: Companhia das Letras, 2014.
• ⁸
  BRASIL. Ministério da Educação. Base nacional comum curricular Brasília: MEC, 2018.
• ⁹
  CARDOSO, Ofélia B. O problema da repetência na escola na escola primária. Revista Brasileira de Estudos Pedagógicos, v. 13, n. 35, p. 74-88, 1949.
• ¹⁰
  DESROSIÈRES, Alain. La politique des grands nombres: histoire de la raison statistique. Paris: La Découverte, 2000.
• ¹¹
  DUBET, François; DURU-BELLAT, Marie; VÉRÉTOUT, Antoine. As desigualdades escolares antes e depois da escola: organização escolar e influência dos diplomas. Sociologias, v. 14, n. 29, p. 22-70, 2012.
• ¹²
  ESTEBAN, Maria Teresa. Silenciar a polissemia e invisibilizar os sujeitos: indagações ao discurso sobre a qualidade da educação. Revista Portuguesa de Educação, v. 21, n. 1, p. 5-31, 2008.
• ¹³
  FERNANDES, Claudia de O. A promoção automática na década de 50: uma revisão bibliográfica na RBEP. Revista Brasileira de Estudos Pedagógicos, v. 81, n. 197, p.76-88, 2000.
• ¹⁴
  FERNANDES, Reynaldo. Índice de desenvolvimento da educação básica (Ideb) Brasília: Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, 2007.
• ¹⁵
  FREITAS, Mário A.T. de. Novos objetivos para a educação no Brasil. Revista Brasileira de Estudos Pedagógicos, v. 4, n. 12, p. 346-360, 1945.
• ¹⁶
  FREITAS, Mário A. T. de. O ensino primário brasileiro no decênio 1932-1941 Rio de Janeiro: IBGE, 1946.
• ¹⁷
  GIL, Natália de L. Reprovação escolar no Brasil: história da configuração de um problema político-educacional. Revista Brasileira de Educação, v. 23, p. 1-23, 2018.
• ¹⁸
  GIL, Natália de L. Estatísticas e educação: considerações sobre a necessidade de um olhar atento. Pensar a Educação em Revista, v. 5, n. 2, p. 1-29, jun.-ago. 2019.
• ¹⁹
  GOULD, Stephen J. A falsa medida do homem São Paulo: Martins Fontes, 2003.
• ²⁰
  HACKING, Ian. The taming of chance Cambridge: Cambridge University, 1990.
• ²¹
  IBGE – Instituto Brasileiro de Geografia e Estatística. Educação 2017 PNAD Contínua. Rio de Janeiro: IBGE - Diretoria de Pesquisas, Coordenação de Trabalho e Rendimento, 2018.
• ²²
  JARDIM, Germano. A coleta da estatística educacional IV. Revista Brasileira de Estudos Pedagógicos v. 7, n. 21, p. 452-463, 1946.
• ²³
  KLEIN, Delci H. IDEB e maquinarias: a produção, a quantificação e a expressão da qualidade da educação brasileira. 2017. Tese (Doutorado em Educação). Faculdade de Educação. Universidade Federal do Rio Grande do Sul, Porto Alegre, 2017.
• ²⁴
  LIMA, Ana Laura G. A “criança-problema” na escola brasileira Curitiba: Appris, 2018.
• ²⁵
  MAINARDES, J. A promoção automática em questão: argumentos, implicações e possibilidades. Revista Brasileira de Estudos Pedagógicos, v. 79, n. 192, p. 16-29, 1998.
• ²⁶
  MATOS, Luiz A. de. A aprovação e a reprovação escolar. Revista Brasileira de Estudos Pedagógicos, v. 26, n. 63, p. 254-257, 1956.
• ²⁷
  OLIVEIRA, Romualdo P. de; ARAUJO, Gilda C. de. Qualidade do ensino: uma nova dimensão da luta pelo direito à educação. Revista Brasileira de Educação, n. 28, p. 5-23, 2005.
• ²⁸
  PATTO, Maria Helena S. A produção do fracasso escolar São Paulo: T.A. Queiroz, 1993.
• ²⁹
  PEREIRA, Luiz. A promoção automática na escola primária. Revista Brasileira de Estudos Pedagógicos, v. 30, n. 2, p. 105-107, out./dez. 1958.
• ³⁰
  POPKEWITZ, Thomas. PISA: numbers, standardizing conduct, and the alchemy of school subjects. In: PEREYRA, Miguel A.; KOTTHOFF, Hans-Georg; COWEN, Robert (eds.). PISA under examination: changing knowledge, changing tests, and changing schools. Rotterdam: Sense Publishers, 2011. p. 31-46.
• ³¹
  PORTER, Theodore M. Trust in numbers: The pursuit of objectivity in science and public life. Princeton: University of Princeton Press, 1995.
• ³²
  ROSE, Nikolas. Governing by numbers: figuring out democracy. Accounting Organizations and Society, v. 16, n. 7, p. 673-692, 1991.
• ³³
  ROSE, Nikolas. Powers of freedom: reframing political thought. Cambridge: Cambridge University Press, 1999.
• ³⁴
  SÃO PAULO. Secretaria da Educação e da Saúde Pública. Diretoria do Ensino. As reprovações na escola primária (O fenômeno das reprovações. Análise das causas. Medidas contra o mal. Dados estatísticos). Boletim n. 7, 1936.
• ³⁵
  SOARES, José Francisco. José Francisco Soares: IDEB na lei? Simon’s Site, 13 de julho de 2011. Disponível em https://www.schwartzman.org.br/sitesimon/?p=2352
  » https://www.schwartzman.org.br/sitesimon/?p=2352
• ³⁶
  SOARES, José Francisco; XAVIER, Flávia P. Pressupostos educacionais e estatísticos do IDEB. Educação & Sociedade, v. 34, n. 135, p. 903-923, 2013.
• ³⁷
  STROMQUIST, Nelly P. Educação latino-americana em tempos globalizados. Sociologias, v. 14, n. 29, p. 72-99, jan./abr. 2012.
• ³⁸
  TEIXEIRA, Anísio. Educação pública: administração e desenvolvimento. Rio de Janeiro: Officina Graphica do Departamento de Educação, 1935.
• ³⁹
  UNESCO. Declaração Mundial sobre Educação para Todos: satisfação das necessidades básicas de aprendizagem. Jumtien: UNESCO, 1990.
• ⁴⁰
  VIRGINIO, Alexandre S. Educação e sociedade democrática: interpretações sociológicas e desafios à formação política do educador. Sociologias, v. 14, n. 29, p. 176-212, 2012.

Publication Dates

• Publication in this collection
  24 May 2021
• Date of issue
  Jan-Apr 2021

History

• Received
  09 Dec 2020
• Accepted
  15 Mar 2021