Is Brazilian education improving? Evidence from Pisa and Saeb

Abstracts

This article assesses the changes in mathematics and reading scores of socially advantaged or disadvantaged Brazilian students, in the International Student Assessment Program - Pisa - between 2000 and 2012, and the National Basic Education Assessment System - Saeb - between 1995 to 2013, in order to extract some provisional conclusions regarding the variations in the effectiveness of basic education in Brazil (1^st to 8^th/9^th grades). Our findings show that the gains in the Pisa math test are much larger than in the reading test. The gains (or their absence), in the Saeb test, for the 1999-2013 period, are also higher in mathematics than in Portuguese. Part of the gains in the Pisa math test and most of those in the reading test result from the gradual increase in the number of years fifteen year-old students spend in school. The gains in the Pisa test for more advantaged Brazilian students are lower than for those coming from families with lower educational resources; this also applies for the Saeb test.

Evaluation; Basic Education; Pisa; Saeb

Neste texto medimos as mudanças das pontuações em matemática e leitura dos alunos brasileiros favorecidos e desfavorecidos no Programa Internacional de Avaliação de Alunos - Pisa - entre 2000 e 2012 e no Sistema de Avaliação da Educação Básica - Saeb - no período de 1995 a 2013, a fim de extrair algumas conclusões provisórias em relação à variação da efetividade do ensino básico brasileiro (1ª a 8ª/9ª séries). Nossos achados mostram que os ganhos no teste de matemática do Pisa são muito maiores do que no teste de leitura. Os ganhos (ou sua ausência) entre 1999 e 2013 no teste do Saeb são também maiores em matemática que em português. Parte do ganho no teste de matemática do Pisa e a maior parte do ganho no teste de leitura resultam do aumento gradual no tempo que os alunos com a idade de quinze anos passam na escola. Os ganhos no Pisa para os estudantes brasileiros mais favorecidos são menores do que entre aqueles com níveis baixos de recursos acadêmicos familiares, o que também se verifica no teste do Saeb.

Avaliação; Educação Básica; Pisa; Saeb

En este texto medimos los cambios del puntaje en matemáticas y lectura de los alumnos brasileiros favorecidos y desfavorecidos en el Programa Internacional de Evaluación de Alumnos -Pisa- entre 2000 y 2012, y en el Sistema de Evaluación de la Educación Básica -Saeb- en el periodo de 1995 a 2013, con el propósito de extraer algunas conclusiones provisorias en lo que concierne a la variación de la efectividad de la educación básica brasileña (1º a 8º /9º años). Nuestros hallazgos muestran que las ganancias en la prueba de matemáticas del Pisa son mucho mayores que en la prueba de lectura. Las ganancias (o su ausencia) en 1999-2013 en la prueba de Saeb también son mayores en matemáticas que en portugués. Una parte de la ganancia en la prueba de matemáticas de Pisa y la mayor parte de la ganancia en la prueba de lectura resultan del aumento gradual en el número de años que los alumnos de quince años pasan en la escuela. Las ganancias en el Pisa para los estudiantes brasileños más favorecidos son menores que entre aquellos con niveles bajos de recursos académicos familiares, lo que también se verifica en la prueba de Saeb.

Evaluación; Educación Básica; Pisa; Saeb

As developing countries have expanded their educational systems to provide an increasing proportion of youth with secondary education, there has been a shift in focus from policies concerned with access to schooling to policies that improve the quality of schooling (UNESCO, 2005). Two other factors have contributed to this shift. The first is research claiming that quality of education, as measured by international test scores, is a better predictor of economic growth than the number of years of schooling in the labor force (HANUSHEK; KIMKO, 2000; HANUSHEK; PETERSON; WOESSMANN, 2013). The second is the increase in testing itself, both at the national and international levels. Student test results are being used increasingly to pressure national and local educational systems, schools, and individual teachers to have their students do better on the tests (OECD, 2013). League tables comparing schools, local school districts, regions, and nations against others are now a regular feature of educational politics in many countries of the world. To some extent, test scores are becoming important enough to affect government legitimacy.

It is therefore not unusual that a country such as Brazil, which considers itself an up-and-comer in the world economy, should be concerned with how well its students are doing compared to students in other countries, and particularly whether Brazilian student performance on tests is improving over time. Brazil has had its own national evaluation system for a number of years the Sistema de Avaliação da Educação Básica - Saeb (1995-2013) -, and since 2005, the Prova Brasil, are national tests given to 4^th (this changed to 5^th graders in the new Brazilian system¹ after 2007) and 8^th graders (this changed to 9^th graders after 2007) every two years. The Saeb is based on a large sample of Brazilian students in both public and private schools, and the Prova Brasil is a censal test (all schools with 20 students or more in the tested grade - 4^th/5^th or 8^th/9^th) only applied to public school students.

In addition, Brazil has participated in the Program of International Student Assessment - Pisa -, administered by the Organization for Economic Co-operation and Development - OECD -, in all five rounds - 2000, 2003, 2006, 2009, and 2012 (see OECD, 2013, v 1, ch. 2). Unlike the Saeb, the Pisa is a test administered to a sample of 15 year-olds based on the distribution of that age student in various grades. The Saeb is a classroom sample, which tests all students in selected classrooms,² and the Pisa is a school sample, testing a sample of about twenty-five 15 year-olds in each selected school.

These two assessments can be used to draw some inferences as to whether Brazilian students are improving the amount of mathematics and language skills they have learned by the end of basic education (Saeb 8^th/9^th grade) and by 15 years-old (Pisa 7^th-11^th grades, then 8^th-12^th grades). Our methodological approach is to approximate improvements in school system quality is to "net out" one major part of out-of-school influences by comparing students with similar family academic resources across countries. We argue that changes in test scores over time of students controlling for student characteristics (gender, ethnicity) and family academic resources - FAR - provide a better assessment of whether a country's educational system is improving than simply tracking average national scores. There are additional complexities concerning the Pisa test because students sampled are in a given age group, not in a single grade, and, in the Brazilian case, the test was applied at different dates in 2000, 2003/2006, 2009, and 2012, further biasing estimates of gains over time (KLEIN, 2011). We attempt to control for grade and date of test in estimating the test score changes in the Pisa for advantaged and disadvantaged students.

Our empirical strategy is descriptive and comparative. We first estimate the level of and changes in Brazilian disadvantaged and advantaged students' Pisa scores in mathematics and reading and compare them with their counterparts in other countries, focusing on Portugal, Argentina, Chile, and Mexico in 2000-2012. Secondly, we describe the test scores gains for Brazilian students in reading/Portuguese and mathematics by family resource groups using Pisa (2000-2012) and Saeb (1995-2011) data to draw some tentative conclusions from these estimates concerning the changing effectiveness of Brazil's basic schooling (grades 1-8/9). We use the Saeb test rather than Prova Brasil in 2005-2011 because Prova Brasil only includes public school students; as a result, the scores of students from higher academic resource families are severely downward biased.

We find that Brazilian students have made test score gains in the first decade of the 2000s on the Pisa, but much less so, if at all, on the Saeb. Brazilian gains on the Pisa mathematics test are much larger than on the Pisa reading test, and part of the gain on the Pisa mathematics test and most of the gain on the reading test result from 15 year-old students' gradually increasing the number of years they stay in school. The gains (or lack of gains) in 1999-2011 on the Saeb are about the same in mathematics and Portuguese. Gains on the Pisa test for advantaged Brazilian students are smaller than among students with low levels of FAR. This is also the case for the Saeb. Advantaged students' gains on the Pisa mathematics test are also smaller than those for advantaged students in Chile, but larger than advantaged students' gains in Argentina and Mexico. By the Pisa 2012 test, Brazilian students at all levels of FAR performed as well or better than students in Argentina, Colombia, and Peru, but not as well as students in Chile, Mexico, Uruguay, Portugal, or Spain.

The argument for comparing test scores by students' family academic resources

Studies have shown that various proxy measures of students' FAR, such as mother's education, parents' education, articles in the home, or books in the home, are correlated with students' academic achievement - for example, for the United States, Coleman et al. (1966), Jencks and Phillips (1998); for the United Kingdom, Peaker (1971); for Pisa, see Schulz (2005), Buchmann (2002), Adamson (2010); for TIMSS, see Raudenbush, Cheong and Fotiu (1996), Woessmann (2004) and Chudgar, Luschei and Fagioli (2012); for a meta-analysis across different types of tests, see Sirin (2005).

There many reasons why a student's family academic environment could be important factor in his or her cognitive (and non-cognitive) achievement. Students raised in a family in which reading materials are readily available and where a parent or parents have attained higher levels of schooling are more likely to be exposed to more complex verbal interaction (HEATH, 1983), to have been read to as a young child, to have had access to better health care and a more nutritious diet (regardless of income), to be subject to higher academic expectations once in school, and to interact with peers from similarly reading-oriented, verbal, higher academic expectation families. Whether one calls such family "investments" during early childhood and after the child enters school cultural capital (BOURDIEU; PASSERON, 1979), human capital (SCHULTZ, 1961), or social capital (COLEMAN, 1988), the concept is the same: outside of school factors associated with family environment are influential in how well a student achieves in school. Beyond this direct influence, families with more academic resources at home are generally more motivated to gain access for their children to schools with more academically motivated students and to pay for tutoring outside school (BRAY, 2006).

If students tested in various countries live, on average, in family environments (in terms of human, cultural, and social capital) that differ considerably, and family environments have an important influence on school achievement, comparisons of average student performance could incorrectly attribute higher or lower outcomes to educational policies when they may be the result of differing outside of school influences. Furthermore, educational policies may affect students from different environments differently. By comparing the academic performance of students in particular family and social environments over time, we can better understand the nuances of educational policies in various countries. Such comparisons are the core of our analysis in this study.

Which proxies should be used for measuring FAR? There is no precise way to make such comparisons between countries. Pisa collects data on many characteristics that are arguably related to family resources. Pisa also assembles them into an overall index called the "index of economic, social, and cultural status" - ESCS -, which has the important disadvantages of combining various factors that may not be international comparable and not revealing which factors are contributing to explaining test score differences (HAUSER, 2013). Although none of the possible indicators of family resource differences is entirely satisfactory, we use two in this paper: the number of books in the home (BH) and mother's education (ME) for our analysis, since they can be divided into specific categories that can be compared between Latin American countries, Spain, and Portugal. A very high fraction of students in the Pisa survey answers the BH question, something less true for other important family academic resource indicator questions asked on the student questionnaires, including ME. The advantage of using ME and BH definitions of FAR is that the Saeb also asks the ME question from 1999-2011 and the BH question from 1999-2005. We note that Pisa samples of students in Latin American countries are more equally distributed across categories of ME than BH categories. Most students in the Latin American Pisa samples, including Brazil's, fall into the bottom two BH categories, 0-10 and 11-25 BH. Yet, even with the more equitable distribution across ME categories, a high percentage of students fall into the bottom three categories, mothers with no education, primary, and lower secondary. It is likely that the proportion of students in these three lower ME categories is biased downward.

Comparing Pisa 2012 performance

As noted, we disaggregate mathematics and reading scores in Brazil and in eight comparison countries by two different measures of FAR: (a) BH, and (b) ME. In (a), we divide students into the six BH categories from the Pisa student questionnaire representing six family academic resource groups, from the least to the most advantaged. We refer to the group of students reporting 0-10 books in the home as very disadvantaged students, those with 11-25 books in the home as disadvantaged, those with 26-100 books in the home as middle advantaged students, and to those with more than 100 books in the home as advantaged students. Table 1 shows that in 2012, only 7.4 percent of Brazilian students fell into the advantaged group as defined by BH. In (b), we can also create six family resource levels from the seven categories of ME in the Pisa student questionnaire. These six categories are mothers reported by students to have had "no education," "primary education" (ISCED 1), "lower secondary education" (ISCED 2), "upper secondary education" (ISCED 3A, 3B, 4), "non-university tertiary education" (ISCED 5B), and "university or graduate education" (ISCED 5A, 6).

Table 1
Pisa 2012: Proportion of Sample by Family Academic Resource Group (BH and ME), Brazil and Comparison Countries

We have to make some adjustments to compare scores by family resource group in 2000-2012 because the BH and ME categories used in the 2000 survey differ from the categories in subsequent surveys (2003, 2006, 2009, and 2012). In the 2000 survey, the BH categories on the student questionnaire were 0, 1-10, 11-50, 51-100, 101-251, 251-500 and more than 500 books in the home. In subsequent years the student questionnaire changed the categories to 0-10, 11-25, 26-100, 101-200, 201-500, and more than 500 books in the home. Because of these changes in categories we need to interpolate scores for groups 2-5.³

For ME, Pisa 2000 only provides the option of reporting mothers as having non-university tertiary and university as one category (ISCED 5B, 5A, and 6), but in later years, they are disaggregated into ISCED 5B and a second category, ISCED 5A, 6. We have to approximate test scores for students in the non-university tertiary and the university groups in 2000 by interpolating the scores for students with ME university education in 2000 based on 2006 (Argentina and Chile did not take the test in 2003) proportions of reported ME and test score ratios for the two subgroups - those with mothers who had non-university tertiary education and those with mothers who had university education.

We want to create categories that are more comparable (in sample proportions) between our two definitions of students' FAR (BH and ME). Thus, we refer to students who reported that their mothers have no education or only primary education as very disadvantaged students, those who reported mothers with lower secondary education (ISCED 2) as disadvantaged, those who reported mothers with upper secondary education and non-university tertiary education (ISCED 3A, 3B, 4, and 5B) as middle advantaged, and those who reported mothers with university education (ISCED 5A, and 6) as advantaged.

Even so, the two definitions of FAR show somewhat different proportions of the Pisa sample in the very disadvantaged, the disadvantaged, the middle advantaged and the advantaged groups, especially in the Latin American countries. For example, based on BH, the very disadvantaged Brazilian Pisa 2012 group was 45 percent of the sample and the advantaged group, only 7.5 percent of the sample. Based on ME, the very disadvantaged group was 32 percent of the sample, and the advantaged group was 16 percent of the sample. The comparison Latin American countries have similar differences in the way the two variables we use to measure FAR categorize students in very disadvantaged, disadvantaged, and advantaged groups. However, one reason that this may occur is that students overestimate their ME levels. It is unlikely that such high percentages of mothers educated in the late 1980s and early 1990s in Latin American countries completed university when the proportion of the population attending university was considerably lower. In Brazil, only about 10-12 percent of the age group reached higher education before 1995.⁴ The percentages of students reporting mothers with higher education in the other countries in our comparison group are also suspiciously high. Thus, our estimated test scores, for the advantaged group as defined by ME, may underestimate the "true" average scores of advantaged students. Yet, since we are focused on changes over time, the level of scores is of less interest than the trend, and these may be similar in both definitions of advantaged and disadvantaged groups.

Brazil has one of the highest proportions of students with low FAR among our comparison countries, whether we measure family resources by BH or ME. Seventy-two percent of the Brazilian sample reported less than or equal to 25 BH, more than the other three countries with high proportions of low family academic resources - Colombia, Mexico, and Peru. In the Brazilian sample, 34 percent of students reported ME of primary school completed or less, comparable with other low FAR student samples in Colombia, Mexico, and Peru.

At the other end of the spectrum, the proportion of the Brazilian sample of students with more than 100 BH (8 percent) was the lowest in our comparison groups, although similar to the proportion in the three other low family resource country samples. In the case of ME, however, the proportion of Brazilian students reporting that their mothers had some higher education (22 percent) was about the same or higher than all our comparison countries except Argentina, Colombia, or Spain.

Table 2 shows that Brazilian students with similar FAR to students in Argentina, Colombia, and Peru generally scored about the same or higher in both reading and mathematics on the Pisa 2012. However, Brazilian students of similar FAR scored significantly lower than students in Chile, Mexico, Uruguay,⁵ Portugal, and Spain.⁶ Advantaged students in Brazil tended to have a much larger gap with their family resource counterparts in the higher scoring countries. In some cases (for example, advantaged students' mathematics scores in Brazil and Portugal) the gap is more than a standard deviation. Using ME as the measure of FAR produces the same comparative conclusions.

Table 2
Pisa 2012: Mean Student Mathematics and Reading Scale Scores by Family Academic Resource Group (BH and ME), Brazil and Comparison Countries

To compare average scores across countries corrected for the differences in the family academic resource composition of the samples, we weighted each of the family resource groups' scores in the comparison countries by the Brazilian sample proportions. Table 3 shows the average reported and Brazilian sample weighted scores for Pisa mathematics and reading using the BH category and the ME category proportions. The results are similar: when we adjust the scores for family academic resource differences in the samples, Brazil's students score higher in mathematics than students in Argentina, Colombia, and Peru, and substantially lower than students in the other comparison countries. In reading, when we use BH weights, Brazilian students score substantially higher than students in Argentina and Peru, about the same as students in Colombia and Uruguay, and lower than students in the other comparison countries. When we use the ME weights, the results are the same except that students in Brazil score higher than students in Colombia and lower than students in Uruguay.

Table 3
Pisa 2012: Mean Student Mathematics and Reading Scores Adjusted for Sample Family Academic Resource Differences, Brazil and Comparison Countries

Trends in Pisa scores, 2000-2012

Pisa has been administered every three years since 2000. This gives us the opportunity to estimate changes in Pisa mathematics scores over time. By observing such changes, we can assess how Brazil's academically disadvantaged and advantaged students increased their performance on the Pisa test.⁷ We can also compare Brazilian students' performance over time to students' performance in several other Latin American countries and in Portugal and Spain. We make our estimates by family academic resource group, because changes over time in the composition of a country's test takers by BH or ME can affect a country's average score while masking real changes (or lack of change) in the performance of that country's students.

Tables 4a and 4b shows that Brazilian students made large gains in the Pisa mathematics test across family academic resource groups in the period 2000-2012, and much smaller gains in the Pisa reading test. The gains in Brazil were larger in both math and reading for lower and middle family academic resource students than for higher resource students, although when FAR is measured by ME, the differences are less apparent, in part because the BH definition of FAR is a much more "exclusive" definition of "advantaged" (only 10 percent of students in 2000 and 8 percent in 2012). Table 4a and 4b also compare gains by Brazilian students in this period with gains by students in other Latin American countries and Spain and Portugal. Colombia and Uruguay are not shown because they did not take the Pisa test in 2000. Lower (<26 BH; mother has zero or primary education) and higher (> 100 BH; mother has university education) FAR Brazilian students' gains in mathematics are as high or higher than any comparison country students' gains except Peru's. Yet in reading, lower and higher family academic resource students in Brazil only made larger gains than students in Argentina, Mexico (particularly among higher resourced students) and in the ME definition of FAR, Spain.

Table 4a
PISA Gains in Mathematics and Reading, 2000-2012, by Books in the Home Categories and Country

Table 4b
PISA Gains in Mathematics and Reading, 2000-20012, by Mother's Education Categories and Country

When we chart the progression of gains by Brazilian students across the five applications of Pisa tests (2000, 2003, 2006, 2009, and 2012) they vary for mathematics and reading and somewhat by whether we measure FAR by BH or ME. In Figures 1a and 1b, Brazilian students' performance is divided into "very disadvantaged," "disadvantaged," and "advantaged," as defined earlier, and compared with students' test scores in Argentina and Chile, two neighboring countries, for which we have results for at least four of the five Pisa rounds.

The Figures 1a and 1b results confirm those in Tables 4a and 4b. Brazilian "very disadvantaged" and "disadvantaged" students made large and steady gains in the Pisa mathematics test in 2000-2012. Brazilian advantaged students also made substantial gains in mathematics in this period, mainly in 2000-2003 when we define advantaged by BH and mostly after 2003 when we measure advantaged by the broader definition of ME. The proportion of students reporting that their mother had university education seems very high - it is possible that this category represents a group of students, who, in reality, have mothers with lower average levels of schooling. The Brazilian gains in reading were smaller, as shown in Tables 4a and 4b, and the pattern of gains from test to test varies considerably across family resource group and the measure of family resource group.

Figure 1a
Achievement trends in Pisa mathematics scores, 2000-2012, by family academic resources (books in the home), Brazil, Argentina, and Chile

Figure 1b
Achievement trends in Pisa mathematics scores, 2000-2012, by family academic resources (mother's education), Brazil, Argentina, and Chile

When comparing Brazilian students' gains with student gains in Argentina and Chile, it is notable that very disadvantaged, disadvantaged, and advantaged Chilean students have higher test scores than comparable Brazilian students. Chilean students make substantial gains in mathematics (and reading) no matter how we define FAR. Only advantaged Argentine students did not make gains in mathematics, and also disadvantaged students in Argentina defined by ME among comparison countries did not make substantial gains in mathematics, even when all the countries shown in Tables 4a and 4b are included in the comparison

Correcting Brazilian Pisa gains for changes in the test date

Pisa differs from other tests such as the Trends in International Mathematics and Science Survey - TIMSS - and national surveys such the Saeb because the target student population is defined by age (15 year-olds) rather than grade. Thus, students surveyed in the Pisa in countries such as Brazil and other Latin American countries are distributed over a number of grades. Even in most developed countries, 15 year-olds in the Pisa survey are concentrated in two grades, and not always the same ones (OECD 2010a; 2013). Ruben Klein (2011) has argued that the date of the Pisa test in Brazil changed twice in the 2000-2009 period, but apparently stayed the same in 2012 as in 2009. According to Klein, the changes in 2000-2009 resulted in increases in the grade level of the students surveyed and therefore biased upward changes in the results over time. In 2003 and 2006, the average student surveyed was two months older than in the 2000 survey, and in 2009, the average student was six months older than in the 2000 survey. In 2012, according to the OECD, the test date was moved forward again and the average student surveyed was again only two months older than in 2000, reducing the bias in comparing with that base year.

In this section, we will apply a modified version of Klein's methods to estimate how much the Pisa mathematics and reading scores of Brazilian students in disadvantaged and advantaged FAR groups (BH) changed in 2000-2012. We then assess how the Brazilian test-date adjusted gains compare with gains for students in other countries.

Klein corrects the reported Pisa test scores for Brazil in reading, mathematics, and science for these test date changes in two ways: first, he weights the reported scores for each test year by grade with the sample proportions by grade for a constant test year - for example, the mean test scores of students in each grade in 2000, 2003, 2006, and 2009 are weighted by the proportions in each grade for the test year 2000 to get an average test score in each subject as if the sample proportions were always as in the year 2000. Second, he estimates an average test score in each test year only for students born in the months common to three of the four test dates and all four test dates; i.e., from May 1 to December 31 or from July 1 to December 31.

Klein (2011) concludes that Brazilian students had lower actual increases in test scores than reported by the OECD (2010b). Using the first method (weighting by proportions in a constant year), he estimated that scores in 2000-2009 declined in reading (rather than the 16 point increase reported), increased by 22-27 points in math, depending on the year used to weight the gains, compared to the 52 point gain reported, and increased 4-7 points in science rather than the 30 point increase reported. Using the second method (constant age group), Klein estimated that the corrected gains were 11 points in reading, 42 points in math, and 29 points in science. Thus, it appears that the lower bound (first method) may not take into effect the increase in average grade attended by 15 year-olds in Brazil, whereas the upper bound (second method) may.

Klein (2011) also makes correction for changes in the test date in other countries of interest to use namely Argentina, Chile, and Mexico. Mexico has the largest changes in test dates, but in different directions: between the 2000 and 2003 surveys, the sample shifted dates so that the those sampled were younger in 2003; between 2003 and 2006/2009, the sample date shifted so that those sampled were older than in 2003 but still one month younger than in 2000. In each case, Klein uses only the first method to adjust the scores, thus assuming that the average grade attended by 15 year-olds remained the same (proportion in each grade constant) in the years affected by the change in test date.

Applying Klein's corrections to family academic resource groupings

We estimated the proportion of students sampled in each year by grade for two FAR groups, 0-10 books in the home and more than 100 books in the home. We call the first "very disadvantaged" and the second, "advantaged." Both groups' definitions are the same as we used earlier in our analysis. Table 5 shows that Brazilian very disadvantaged students were much more likely to be in 7^th and 8^th grade than in 9^th, or, in later years, in 10^th grade, than advantaged students, who are mostly in 9^th and 10^th grade.

Klein's argues that there was a particularly large shift of students sampled from lower to higher grades from 2006 to 2009, when the test was applied later in the year and the birthdate used to define who would be sampled rose by several months (KLEIN, 2011). We can observe this for both FAR groups in Table 5.

Table 5
Brazil Pisa: proportion of students in sample, very disadvantaged and advantaged students, by grade enrolled, 2000-2012

In 2012, Pisa shifted its sample up one grade and did not sample any 15 year olds in 7^th grade, limiting the sample to the 8^th-12^th grades. A law passed in 2006 dropped the age of initial enrollment, and in 2007, Brazilian children in 60 percent of schools began entering the first grade of ensino fundamental (EF) a year younger. In turn, EF schools have extended the first cycle (first to fourth grades) from four to five years. The second cycle is still four years long and now ends in 9^th grade. Secondary school (ensino médio - EM) remained a three-year program, but moved from 9^th-11^th grade to 10^th-12^th grade.

At the same time, the Brazilian school system renumbered all its grades to account for the one-year increase in EF. A 15 year-old 8^th grader in 2009 is a 9^th grader in 2012, a 9^th grader in 2009 is a 10^th grader in 2012, and so forth. However, the 15 year-olds sampled in 2012 have not attended schools a year longer - the first 15 year-olds who entered Brazilian schools a year earlier will begin showing up in the 2015 Pisa, and the full impact on PISA results will be felt in 2018. In 2012, we can consider that the grade number change is simply a renaming of the grade without implications for curriculum exposure or opportunity to learn.

Table 6a shows that in 2000-2009 the estimated Pisa mathematics scores increased for Brazilian students within each grade for both very disadvantaged and advantaged students in 2000-2009 (only exception, 9^th grade for advantaged students), suggesting that mathematics performance on the Pisa really did increase in Brazil during this period. Many of the increases within grade are considerably smaller than that reported by the OECD for the Brazilian sample as a whole. The gains for advantaged students also tend to be somewhat smaller than very disadvantaged students.

The opposite is true for reading scores - in almost every grade for Brazilian disadvantaged and advantaged students scored lower in 2009 than in 2000 on the Pisa reading test (Table 6b). If we shift the 2012 scores over by a grade, the same lack of increase in reading scores holds for 2000-2012 in every grade. The declines for advantaged students tend to be larger than for disadvantaged students. The only exception is for the small percentage of advantaged students in the 11^th/12t^h grade in 2000-2012. They had a large increase in reading scores.

In both mathematics and reading, the scores for the very disadvantaged students are lower than for advantaged students, as we would expect. The differences get larger in the 9^th/10^th and 10^th/11^th grades, where most of the advantaged 15 year-old students are found. We cannot say whether the lower test scores of disadvantaged students are partly "caused" by less exposure to subject matter (being in a lower grade because of a late school start) or whether lower test scores reflect lower academic ability or spending their school years in worse schools, both of which could cause disadvantaged students to repeat and be in a lower grade.

Table 6a
Brazil Pisa: mean mathematics scores of students in sample, very disadvantaged and advantaged students, by grade enrolled, 2000-2012

Table 6b
Brazil Pisa: mean reading scores of students in sample, very disadvantaged and advantaged students, by grade enrolled, 2000-2012

Using Klein's first method of weighting these scores by grade for each of the two FAR groups (0-10 BH and > 100 BH), we show in Table 7 how average scores behaved from 2000 to 2012 for very disadvantaged and advantaged students when we apply the Pisa 2000 grade distribution to subsequent test years and when we apply the Pisa 2012 grade distribution to previous test years.⁸ We find that very disadvantaged students have substantially large mathematics gains in 2000-2012 of 31-41 points depending on the set of weights used, and that the gains are fairly steady over the nine years of testing. Advantaged students show much smaller gains in mathematics, but notably, they made large gains in 2000-2003, and declined subsequently. We also find that reading test scores declined or remained not significantly lower for disadvantaged students (depending on the grade weights used) and declined more for advantaged students. The decline is particularly notable using the 2000 weights, which suggests that the decline in reading scores was much greater among higher FAR students in lower grades.

Table 7
Brazil Pisa: mean mathematics and reading scores adjusted for changed sample proportions in various grades, very disadvantaged and advantaged students, 2000-2012

Correcting for higher grade attainment, 2000-2012

Klein's method assumes that all of the increase in grade level occurred because of the changes in the date of the test, which, relative to the 2000 test year, increased the average age of students, hence the likelihood that they would be in a higher grade on the test date. However, the average grade of attendance did increase among Brazilian students in 2000-2012 independent of the test date, and even without accounting for the year earlier entry that began in 2008. We should therefore consider the estimates of Pisa gain controlling for grade drift a "lower bound" estimate, and, as Klein (2011) interpreted it, an estimate of the increase in the "pure quality" of Brazilian education. This interpretation of increases in educational quality is also implicit in OECD publications, as well as the studies pushing "quality of education" as a main driver of economic growth (HANUSHEK; PETERSON; WOESSMANN, 2013). If we were to take that interpretation seriously, the quality of mathematics education in Brazil in 2000-2012 increased considerably for low FAR students but hardly at all for higher FAR students, and reading instruction became slightly worse for low FAR students but considerably worse for high FAR students.

However, assuming that students learn more by staying longer in school, attaining higher grades should, in and of itself, raise a student's Pisa score. In the Brazilian case, there is strong evidence that with the reform of 2006, those pupils who entered school a year earlier in 2007 did score about 0.25 standard deviations higher because of that extra year on the 2011 Prova Brazil fifth grade test than students who did not enter early (MARTINS, 2014). Therefore, when Brazilian students spend more years in school, it is likely that they learn more.

We can approximate the increase in average grade attended by students in Pisa test years using Klein's data on the grade distribution of 15 year-old students at the June 30^th cutoff in the 2000 Pisa test application and the December 31^st cutoff in the 2009 Pisa application, which was also the cutoff data in the 2012 Pisa application.⁹ Our estimates of the average grade are shown in Figure 2. From the curve in Figure 2, the change in grade attained in 2000-2012 assuming the Pisa 2000 definition of 15 year-olds is 8.80 minus 8.45 years, or 0.35 years. Assuming the 2009/2012 definition, the gain in grade is 0.47 years.

Figure 2
Brazil Pisa: increase in average grade attended by sampled students, 2000-2009, assuming 2000 and 2009 age definitions of students

When we apply the estimated change in grade level to the estimates of test score by grade in the various test years for the very disadvantaged and advantaged groups we employ in our estimates, we find that very disadvantaged Brazilian students added about 15-17 points in Pisa math score and 15-21 points in Pisa reading score because in 2012 they were likely to be in a higher grade than in 2000 regardless of the date of the test. As estimates we made for Table 7, we assume that in 2012, grade 8 is equivalent to grade 7 in earlier years, grade 9 to grade 8, and so forth. Advantaged students added even more points from the likelihood of being in a higher grade because the average test scores of advantaged students tended to have a higher grade-achievement gradient in both the test years 2000 and 2012. Advantaged Brazilian students added 25-37 points in math and 26-34 points in reading scores from the likelihood of being in a higher grade when they took the 2009 Pisa test than when they took the 2000 test, regardless of the date of the test.¹⁰

The summaries of these calculations in Table 8 show that Brazilian student gains in Pisa mathematics tend to be substantial, particularly for very disadvantaged students, and may also be large for advantaged students, depending on the assumptions of increased grade attainment in 2000-2012. The results also show that gains in Pisa reading have been much smaller, and may have been negative, depending on the assumptions made about gains from increased grade attainment, especially for advantaged students. The lower bound estimates conform to Klein's estimates using constant grade distributions across Pisa test years, and the upper bound estimates represents the assumption that students learn more as they stay in school longer (the average grade level attained by the 15 year-old population increases over time), regardless of how 15 year-olds are defined by the data of the test.

The upper bound estimates in Table 8 are generally the same or smaller than the gains in 2000-2012 reported directly from the Pisa scores disaggregated by FAR in Table 4a. Very disadvantaged and advantaged Brazilian students' gains in math of 63 points and 43 points are as large or larger than the 48/56 (<10 BH) and 28/48 points (>100 BH) upper bound gains in Table 8. In reading, the gain of 19 points in 2000-2012 reported for very disadvantaged students in Table 4a is as large or larger than the 6-16 points upper bound gain shown in Table 8, but the gain for advantaged students estimated directly from "uncorrected" Pisa scores is the same or smaller than the upper bound "corrected" -1/14 point gain for advantaged students shown in Table 8.

As mentioned, Klein (20111) also attempts to estimate the effects of changing dates of Pisa tests in Argentina, Chile, and Mexico - countries of interest to us for comparison purposes. In Argentina, he adjusts the reported Pisa test score gains upward because of test date changes, but in Chile and Mexico, the gains are adjusted downward - only slightly in Mexico but substantially in Chile. Nevertheless, these conform to his lower bound estimates for Brazil (first method). Chilean secondary school enrollment and completion expanded quite rapidly in the first decade of this century, suggesting that using constant grade weights to estimate test score gains probably underestimates the "true" increase in Chilean PISA test scores.

If we compare Argentine, Chilean, and Mexican student Pisa performance in math and reading with Klein's adjustments, we would still conclude (as above) that Brazilian students made as large or larger gains than students in those three countries in mathematics in 2000-2012, made about the same low gains in reading as Mexico but did relatively better than Argentina, and made considerably lower gains in reading than Chile (see Table 8).

Table 8
Brazil Pisa: Student Test Score Gains in Mathematics and Reading Under Different Adjustment Assumptions, 2000-2012

Brazilian student gains on the Saeb test

Brazil has tested its students nationally in the 4^th/5^th and 8^th/9^th grades for almost 20 years. This is defined as the National Evaluation of Basic Education - Aneb - (the) and the test is called the Saeb, based on a large sample of schools nationwide. After 2005, the Prova Brasil, the National Evaluation of School Achievement - Anresc - was also given to all students in schools with 20 or more students in each of these same two tested grades. Although Saeb is applied to students in both private and public schools, the Prova Brasil is only given to public school students. The results of the Saeb test were not published in 2007 and 2009, but are available for 2011 and 2013. We can therefore chart scores for 8^th/9^th graders over the same period as the Pisa test for students, estimating scores for groups of students with different levels of FAR (BH and ME). Since many advantaged students attend private schools and only the Saeb test applies the test in private schools, we will focus on the Saeb test in the period 1995-2013.

Several limitations exist in charting mean mathematics and Portuguese test results over the seven test applications in 1995-2011 because BH are not included in the student questionnaire in 1995, 1997, 2011, and 2013, and the categories for mother's (and father's) education change from the earlier years in 2003 and 2005, and then again in 2011 (same categories in 2013 as in 2011). We present results with the data we have on BH (Table 9a), and show two different estimates for mean scores for students grouped by ME (Tables 9b and 9c). The first (Table 9a and Figure 3a) defines ME groups in one way in 1995-2001, conforming to the group definitions in those years, and a second way in 2003-2013, conforming to the group definition in 2011. The second (Table 9c and Figure 3b) is able to unify definitions for 1995-2005, but in that table, the group definitions are different in 2011 and 2013. As a result of that different definition in 2011 and 2013, mean scores are biased downward in 2011 relative to earlier years' scores.

The results suggest that there may have been significant increases in mean mathematics and Portuguese Saeb scores for 8^th/9^th graders in 2005-2013. For students with relatively low ME, these gains did not increase scores to earlier levels. We also observe the differences in gains between mathematics and reading (Portuguese) in 1999-2013 reported in the Pisa results during this same period. Mathematics and Portuguese scores declined similarly in 1995-1999, leveled off (except for students with university educated mothers) in 1999-2005, and apparently rose for all groups but the most advantaged in both subjects in 2005-2013, but much less for reading (Portuguese). In the second version of our ME estimates (Table 9c, Figure 3b), note that the definition of ME in 2011 and 2013 is such that the mean score is biased downward except for the university complete category. The first version of the estimates (Table 9b, Figure 3a) more accurately reflects the "true" gains in 2005-2013.

In short, there seems to be a consensus that there were significant gains in mathematics on the both Pisa and Saeb in the period 2005-2013, and that the gains in 2005-2013 were much larger for very disadvantaged and disadvantaged students on both tests. There also appears to be a consensus that the gains in reading/Portuguese on the Saeb were small, similar to small gains in reading on the Pisa.

Table 9a
Brazil: 8^th/9^th grade Saeb, mean student scale scores by books in the home, 1999-2005

Table 9b
Brazil: 8^th/9^th grade Saeb, mean student scale scores by mother's education, 1995-2013

Table 9c
Brazil: 8^th/9^th grade Saeb, mean student scale scores by mother's education, 1995-2013

The results of the Saeb test therefore conflict with the results of the Pisa mathematics test, which has Brazilian students making substantial gains in 2000-2012 in Pisa but not Saeb in the first half of this period. Another important difference between the Saeb and Pisa results is the decline in achievement in Saeb for the advantaged group in 2005-2013 in both subjects. In the Pisa, the advantaged group as defined by ME makes gains after 2005.

There are many possible explanations for the difference in the gains on the two tests. The most important is that the Saeb is applied to students in a specific grade - in this case the 8^th/9^th grade, instead of a sample of 15 year-old students in a sample of schools. As shown by Klein (2011), a large part of the reported gains on the Pisa in 2000-2012 can be explained by the increase in the average grade attended by the 15 year-olds in the Pisa sample. It is probable that the increase was greater in the first part of this period. A second explanation is that the Saeb tests to an assumed version of a Brazilian curriculum (states and municipalities very in their curricula and use a wide variety of textbooks) rather than an assumed version of what 15 year-olds worldwide should know in the tested subjects, so may better reflect the gains that students are making on what they are being taught. It is possible that until 2005, students may not have been making much progress in how much mathematics they were learning in terms of what they were seeing in their classrooms. The Saeb is also a much larger sample than the Pisa. Could there be some problems with the Pisa sample in the early years as it expanded in size in 2000-2012? Finally, as far as the difference for the advantaged group in the positive gains in Pisa scores but decline in Saeb scores, it is possible that the students in this group in Pisa had overestimated their ME on the Pisa student questionnaire but not on the Saeb questionnaire.

Figure 3a
Brazil: Saeb mean mathematics scores by students' reported mother's education, 1995-2013, definitions of mother's education, version I

Figure 3b
Brazil: Saeb mean mathematics scores by students' reported mother's education, 1995-2013, definitions of mother's education, version II

Conclusions

Brazilian 15 year-old students have definitely made substantial gains on the Pisa mathematics test in 2000-2012, and may have made more modest gains on the Pisa reading test. This has taken place while Brazil, like other Latin American countries, has greatly increased the enrollment rate in 9^th and 10^th grades and reduced age-grade distortion. Brazilian students' gains in mathematics are as high or higher than achieved by students in most other Latin American countries, and those in Spain and Portugal. This is not the case for the Pisa reading test, where gains in Brazil have been lower than in most other countries. Brazilian 8^th/9^th grade students also have shown significant gains on the Saeb mathematics test (and modest gains on the Portuguese test), but only since 2005. In both Pisa and Saeb, gains have been generally somewhat larger for students from families with fewer academic resources than for students from families that could be called "advantaged" in term of FAR.

The results on the two tests give conflicting signals. The Saeb suggests that Brazilian students are making big gains in mathematics after 2005 and small gains in reading (Portuguese), but only after 2011; the Pisa results agree with these small gains in reading, but suggest that Brazilian middle and secondary school students were making major gains in mathematics already in 2000-2006. This is an unusual combination of results. There are a number of countries, such as the United States, Russia, and England/United Kingdom, that do relatively well and have made gains in mathematics on the 8^th grade TIMSS but not on the Pisa test (CARNOY; ROTHSTEIN, 2013). Yet, the examples are rare of countries that make large gains on an international test such as the Pisa, which is not linked to the national curriculum, but only have gains more recently on their national test, which is linked to the curriculum.

Like the Saeb, the TIMSS is designed to measure the math taught in school whereas the Pisa measures more general math problem solving skills (SCOTT, 2004; GRONMO; OLSEN, 2006). Further, in Pisa 2000 and 2003 - the first two Pisa tests - Brazilian students scored among the lowest in Latin America (along with Peru) in mathematics; they did relatively much better in reading. Therefore, the large gains in math may have been mainly a "catching up" to where they should have been in terms of their math knowledge, particularly because the Pisa was a new type of test, very unlike the Saeb or the tests they take in school. However, as we show, the most likely reason for the gains on the Pisa at the beginning of the decade is the significant increase in years of schooling taken by the average 15 year-old student, and not the greater efficiency of mathematics teaching in each grade.

The results of both tests suggest that students with fewer resources at home are definitely not falling farther behind their more advantaged counterparts; to the contrary, they may be gaining on advantaged students. The gains in Pisa math and reading by Brazilian students have been greater in the past decade among lower FAR students. Despite these gains, by Pisa 2012, disadvantaged Brazilian students were still not scoring as high in math and reading as their counterparts in the higher scoring Latin American countries, except in the case of Argentina. Brazilian students' achievement levels went up considerably more in 2000-2012 than those in Argentina.

Advantaged Brazilian students were in a somewhat worse position compared to their counterparts in other countries, except again in the Argentine case. The significant gains in mathematics and the smaller gains in Portuguese on the Saeb were concentrated among students in low and middle FAR groups, as measured by ME (data on BH are not available for 2001 or 2013, when the gains were made). From the standpoint of greater educational equity, this is a positive result for the Brazilian educational system; however, in terms of producing excellence at the higher end of the distribution, the results suggest important difficulties.

What should policy makers take away from these results? Are Brazilian students learning more in school today than 10 years ago? The answer is probably yes, but also probably not as much as the reported Pisa test results would have us believe. Klein's work (2011) has already made the point that the published Pisa results are overestimates of gains. We suggest that it is more likely that disadvantaged students made larger gains on the Pisa than advantaged students. We also suggest that especially for advantaged students, the mathematics gains that Brazilian students made on the Pisa since 2000 conflict with the Saeb results that show a decline in gains in mathematics for the advantaged group even after 2005. On the other hand, the Saeb and Pisa results for reading/language arts agree that gains there have been small even for disadvantaged groups

• ADAMSON, F. How does context matter? Comparing achievement scores, opportunities to learn, and teacher preparation across socio-economic quintiles in TIMSS and PISA. Dissertation (PhD Philosophy) - Stanford University, 2010.
• BOURDIEU, P.; PASSERON, J. C. La reproduction. Paris: Minuit, 1970.
• BRAY, M. Private supplementary tutoring: comparative perspectives on patterns and implications. Compare: A Journal of Comparative Education, v. 36, n. 4, p. 515-530, 2006.
• BUCHMANN, C. Measuring family background in international studies of education: conceptual issues and methodological challenges. In: PORTER, A.; GAMORAN, A. (Ed.). Methodological advances in cross-national surveys of educational achievement. Washington D.C.: National Academy Press, 2002. p. 150-197.
• CARNOY, M. E.; ROTHSTEIN, R. What do international tests really show about american student performance. Washington, DC: Economic Policy Institute, 2013.
• CHUDGAR, A.; LUSCHEI, T. F.; FAGIOLI, L. P. Constructing socio-economic status measures using the trends in international mathematics and science study data. East Lansing: Michigan State University, 2012.
• COLEMAN, J. S. Social capital in the creation of human capital. American Journal of Sociology, n. 94, (Special Supplement), p. 95-120, 1988.
• COLEMAN, J. S. et al. Equality of educational opportunity. Washington, DC: National Government Printing Office, 1966.
• GRONMO, L. S.; OLSEN, R. V. TIMSS versus PISA: the case of pure and applied mathematics. In: IEA INTERNATIONAL RESEARCH CONFERENCE. 2. Washington, D.C., November 8-11, 2006.
• HANUSHEK, E.; KIMKO, D. Schooling, labor force quality, and the growth of nations. American Economic Review, v. 90, n. 5, p. 1184-1208, 2000.
• HANUSHEK, E.; PETERSON, P.; WOESSMANN, L. Endangering prosperity. Washington, D.C.: The Brookings Institution, 2013.
• HAUSER, R. Some methodological issues in cross national educational research: quality and equity in student achievement. EurAmerica, v. 43, n. 4, p. 709-752, 2013.
• HEATH, S. Brice. Ways with words: language, life, and work in communities and classrooms. Cambridge, UK: Cambridge University Press, 1983.
• JENCKS, C.; PHILLIPS, M. The black-white test score gap. Washington, DC: The Brookings Institution, 1998.
• KLEIN, R. Uma re-análise dos resultados do Pisa: problemas de comparabilidade. Ensaio: Avaliação e Políticas Públicas em Educação, v. 19, n. 73, p. 1-20, out./dez. 2011.
• MARTINS, M. A Difference in Differences Analysis of an Extra Year of Compulsory Education in Brazilian Primary Schools. MA paper - Stanford University, 2014.
• ORGANIZATION OF ECONOMIC COOPERATION AND DEVELOPMENT - OECD. Pisa 2009 results: what students know and can do - student performance in reading, mathematics, and science. Paris: OECD, 2010a. v. I.
• ______. Pisa 2009 results: overcoming social background Paris: OECD, 2010b. v. II.
• ______. Pisa 2012 results: what students know and can do - student performance in reading, mathematics, and science. Paris: OECD, 2013. v. I.
• PEAKER G. F. The Plowden children four years later. Slough: National Foundation for Educational Research in England and Wales, 1971.
• RAUDENBUSH, S. W.; CHEONG, Y. F.; FOTIU, R. P. Social inequality, social segregation, and their relationship to reading literacy in 22 countries. In: BINKLEY, M.; RUST, K.; WILLIAMS, T. (Ed.). Reading literacy in an international perspective. Washington, D.C.: NCES, 1996. p. 3-62.
• SCHULTZ, T. Investment in human capital. American Economic Review v. 51, n. 1, p. 1-17, 1961.
• SCHULZ, W. Measuring the Socio-economic background of students and its effect on achievement in PISA 2000 and PISA 2003. In: ANNUAL MEETINGS OF THE AMERICAN EDUCATIONAL RESEARCH ASSOCIATION. San Francisco, California, April 7-11, 2005. Disponível em: <http://works.bepress.com/wolfram_schulz/10/>. Acesso em: 14 fev. 2013.
  » http://works.bepress.com/wolfram_schulz/10/
• SCOTT, E. Comparing NAEP, TIMSS and PISA in Mathematics and Science. Washington, D.C.: National Center of Educational Statistics, 2004. Disponível em: <http://nces.ed.gov/timss/pdf/naep_timss_pisa_comp.pdf>. Acesso em: 4 jul. 2013.
  » http://nces.ed.gov/timss/pdf/naep_timss_pisa_comp.pdf
• SIRIN, S. Socioeconomic status and academic achievement: a meta-analytical review of research. Review of Educational Research, v. 75, n. 3, p. 417-453, 2005.
• UNESCO. World education report 2005. Paris: Unesco, 2005.
• WOESSMANN, L. How equal are educational opportunities? Family background and student achievement in Europe and the United States. CESifo Working Paper, Munich, n. 1162, 2004.

• 11
  A partir de 2007, os alunos passaram a ser admitidos no ensino fundamental aos seis anos em vez de aos sete. Essa mudança acrescentou um ano de escolaridade para esses novos ingressantes, de modo que o primeiro ciclo do ensino básico passou a compor-se de cinco anos, em vez dos quatro anteriores, e o último ano do ensino médio é o 9º ano em vez da 8ª série. Muitas escolas (cerca de um terço) escolheram esperar um ano (2008) para implementar a alteração estabelecida pela nova lei, e um pequeno percentual esperou até 2009 ou 2010.
• 12
  Até 2005, metade dos alunos avaliados realizou o teste de leitura, e a outra metade, o de matemática; desde 2005, todos os alunos presentes na classe realizaram ambos os testes.
• 13
  Estimamos as pontuações interpoladas supondo que as pontuações médias dos estudantes aumentam linearmente de categoria para categoria. Supomos que aquela pontuação média correspondeu aos estudantes com o número médio de livros na categoria - 30 livros. A classe social similar nas amostras do Pisa de 2003, 2006 e 2009 foi 11-25 livros no domicílio - uma média de 17,5 livros. A categoria seguinte mais baixa de classe social em 2000 foi 1-10 livros no domicílio, uma média de 5 livros. Supomos que os estudantes com 17,5 livros pontuariam menos que aqueles com 30 livros pela proporção (17,5-5) / (30-5) da diferença em pontuação do teste entre categorias. Esta é a pontuação média que atribuímos à categoria interpolada de 11-25 livros no domicílio (grupo 2) em 2000. Fazemos estimativas semelhantes para as categorias interpoladas 26-100 livros (grupo 3), 101-200 livros (grupo 4) e 201-500 livros (grupo 5) para o teste de matemática do Pisa 2000 em cada país usado na comparação. Essas são as estimativas que usamos para calcular diferenças em pontuações de teste nos grupos de domicílio em 2000-2009.
• 14
  Em 2003, 30% dos estudantes brasileiros que se submeteram ao Pisa relataram que suas mães tinham instrução universitária. Como mostraremos, as pontuações médias de leitura e matemática do Pisa para os estudantes brasileiros "favorecidos" são significativamente mais baixas em 2003 quando comparadas a 2000 e anos subsequentes. É provável que isso resulte do fato de um grande grupo de estudantes "superestimarem" o nível de instrução de suas mães, provocando, assim, uma distorção para baixo na pontuação média do grupo favorecido.
• 15
  Uma exceção interessante é que os estudantes brasileiros com recursos acadêmicos familiares mais baixos, definidos quer pelo LD ou pela IM, pontuaram melhor que estudantes uruguaios com baixos recursos acadêmicos familiares; verificou-se o oposto para os estudantes moderadamente favorecidos e favorecidos no Brasil e no Uruguai.
• 16
  Os erros padrão das pontuações médias variam de acordo com o grupo de recurso familiar e o país, porque os tamanhos da amostra diferem para cada grupo/país. Empregamos uma "regra prática" convencional para definir a diferença estatisticamente significante entre as pontuações médias, a saber, dois erros padrão.
• 17
  De acordo com a OCDE, a pontuação no teste de matemática do Pisa 2000 pode não ser totalmente comparável às pontuações dos anos posteriores. No entanto, as tendências nas pontuações de matemática em 2000-2003 por grupo de recurso acadêmico familiar não são substancialmente diferentes das tendências nas pontuações de leitura em todos os oito países que estudamos, de modo que nos sentimos seguros sobre a comparabilidade de nossas tendências estimadas.
• 18
  Quando usamos os pesos de 2012, supomos que a proporção da 8ª série é equivalente à da 7ª série, a da 9ª série equivalente à da 8ª e assim por diante.
• 19
  Correspondência com Ruben Klein (2011).
• 20
  Isso pressupõe que os brasileiros desfavorecidos e favorecidos nas amostras do Pisa tinham os mesmos aumentos em conquista de série no período dos testes de 2000-2012. Não temos nenhuma evidência que confirme essa hipótese, embora pareça provável que, com a grande expansão do ensino secundário inferior e superior nesse período, os alunos muito desfavorecidos estariam mais propensos a aumentar sua conquista de série mais rapidamente do que as crianças favorecidas. Se isso for verdade, estamos subestimando os ganhos para os alunos muito desfavorecidos e superestimando os ganhos para os alunos favorecidos.
• 1
  Beginning in 2007, students were admitted into schools at 6 years-old rather then at 7 years-old. This has added a year of schooling for these new earlier entrants, so that the first cycle of basic education is 5 years rather than 4 years, and the last year of middle school is the 9^th grade rather than the 8^th grade. Many schools (about one-third) elected to wait a year (2008) to implement the new law, and a small percentage waited until 2009 or 2010.
• 2
  Up to 2005, half the students tested did the reading test, and half the math; since 2005, all students in the class did both the reading and the math tests.
• 3
  We estimate the interpolated scores by assuming that students' average scores increase linearly from category to category. We assume that that average score corresponded to students with the average number of books in the category - 30 books. The similar social class in the 2003, 2006, and 2009 Pisa samples was 11-25 books in the home -an average of 17.5 books. The next lowest social class category in 2000 was 1-10 books in the home, an average of 5 books. We assume that students with 17.5 books would score lower than those with 30 books by the proportion (17.5-5)/(30-5) of the difference in test score between categories. This is the average score we assign to the interpolated category of 11-25 books in the home (group 2) in 2000. We make similar estimates for the interpolated categories, 26-100 books (group 3), 101-200 books (group 4), and 201-500 books (group 5) for the 2000 Pisa math test in each comparison country. These are the estimates we use in calculating test score differences by books in the home groups in 2000-2009.
• 4
  In 2003, 30 percent of Brazilian students taking the Pisa reported that their mothers had university education. As we shall show, the average Pisa math and reading scores for "advantaged" Brazilian students are significantly lower in 2003 compared to 2000 and subsequent years. It is probable that this is the result of a large group of students "overestimating" their ME and therefore downwardly biasing the average score of the advantaged group.
• 5
  An interesting exception is that Brazilian students with lower family academic resources, defined either by BH or ME, scored higher than low FAR Uruguayan students; the opposite was true for middle advantaged and advantaged students in Brazil and Uruguay.
• 6
  The standard errors of the mean scores vary according to family resource group and country because sample sizes differ for each group/country. We employ a conventional "rule of thumb" to define a statistically significant different between mean scores; namely, two standard errors.
• 7
  According to the OECD, the 2000 Pisa mathematics test score may not be completely comparable to later year scores. However, the trends in 2000-2003 math scores by family academic resource group are not substantially different from trends in reading scores across the eight countries we study so feel confident in the comparability of our estimated trends.
• 8
  When we use the 2012 weights, we assume that the 8^th grade proportion is equivalent to the 7^th grade proportion, the 9^th grade to the 8^th grade proportion, and so forth.
• 9
  Correspondence with Ruben Klein.
• 10
  This assumes that disadvantaged and advantaged Brazilians in the Pisa samples had the same increases in grade level attained in the 2000-2012 period of the Pisa tests. We have no evidence that confirms this assumption, although it seems likely that with the large expansion of lower and upper secondary schooling in this period, very disadvantaged students would be more likely to increase attainment more rapidly than advantaged children. If this were true, we are underestimating the gains for very disadvantaged pupils and overestimating the gains for advantaged pupils.

Publication Dates

• Publication in this collection
  Jul-Sep 2015

History

• Received
  May 2015
• Accepted
  June 2015