A re-analysis of PISA results: comparability problems

Abstracts

In this paper it is shown a comparability problem in Pisa results of some countries among years and among countries. This problem occurs because the school age of each country is not considered and because there is no rule for the application date of the test in relation to the school year of each country. The case of Brazil is analyzed with detail. There are analysis of some other countries too, among them Luxembourg and Chile, that together with Brazil, were pointed out as having the biggest evolution in performance since 2000. It is shown that this evolution of performance may not be real.

Pisa; Comparability; School age; Composition of students by grade; Proficiency

Neste artigo apresenta-se um problema de comparabilidade de resultados entre aplicações do Pisa, em alguns países, em diversos anos e entre países. Esse problema ocorre porque a idade escolar de cada país não é considerada e porque não há uma regra para a data de aplicação da prova em relação aos anos letivos de cada país. A mudança de data da aplicação em um país modifica a composição do alunado entre as séries, tornando os universos não comparáveis. Analisa-se com detalhes o caso do Brasil e mostra-se que os universos de 2000 e 2009 não são comparáveis. Analisa-se também alguns outros países, entre eles Luxemburgo e Chile, que, juntos com o Brasil, foram destacados como tendo a maior evolução de desempenho desde 2000. Em Luxemburgo, não houve melhoria de desempenho de 2003 para 2009 e é preciso investigar porque os resultados de 2000 e 2003 são tão diferentes. O Chile apresenta melhoria de desempenho, mas provavelmente não tão grande quanto o apontado.

Pisa; Comparabilidade; Idade escolar; Composição do alunado por série; Proficiência

Ruben Klein

Ph. D. in Mathematics from Massachusetts Institute of Technology (MIT), retired researcher from LNCC/MCT, consultant of Fundação Cesgranrio, works in the area of educational statistics and assessment. E-mail: [e-mail]

ABSTRACT

This article looked at a problem of comparability of PISA results in some countries, in various years and between countries. This problem occurs because the school age of each country is not considered and because there is no rule for the date the test is held in relation to the school year of each country. The changing of the date of testing in a country modifies the composition of the student body among the different grades, making the universes incomparable. The Brazilian case was looked at in detail and it was shown that the 2000 and 2009 universes are not comparable. Some other countries were also analyzed, including Luxemburg and Chile, which together with Brazil, were highlighted as having made the greatest progress in performance since 2000. In Luxemburg there was no improvement in performance between 2003 and 2009, while the reason why the results of 2000 and 2003 are so different needs to be investigated. Chile improved its performance, but probably not as much as was highlighted.

Keywords: Pisa. Comparability. School age. Composition of the student body per grade. Proficiency.

RESUMEN

En este artículo se presenta un problema de comparabilidad de resultados entre las aplicaciones del PISA en algunos países, en diversos años y entre los países. Ese problema ocurre porque no se considera la edad escolar de cada uno de los países ni tampoco existe una regla para la fecha de aplicación de la prueba con relación a los años lectivos de cada país. El cambio de fecha de la aplicación en un determinado país modifica la composición del alumnado entre los grados, tornando los universos incomparables. Se analiza con detalles el caso de Brasil y se muestra que los universos de 2000 y 2009 no se pueden comparar. Se analiza también algunos otros países, entre ellos Luxemburgo y Chile, que, juntos con Brasil, se destacaron por tener la mayor evolución de desempeño desde 2000. En Luxemburgo no hubo mejora de desempeño de 2003 para 2009 y es necesario investigar porque los resultados de 2000 y 2003 son tan distintos. Chile presenta mejora de desempeño, pero quizás no tanto como lo señalado.

Palabras-clave: PISA. Comparabilidad. Edad escolar. Composición del alumnado por grados. Proficiencia.

Introduction

Program for International Student Assessment (PISA) is a research program that measures the knowledge and abilities of fifteen year old students in the former seventh grade (now the eighth year in the Brazilian primary education system) and upwards, in Organization for Economic Cooperation and Development (OECD) member states, as well as invited countries. Pisa assesses reading, mathematics and science every three years. Each assessment year greater emphasis is given to one of these areas. In 2000 it was reading, in 2003 mathematics, science in 2006, and reading again in 2009.

PISA uses Item Response Theory (IRT) with a single parameter, the Rasch model, and places the results on the same proficiency scale for each area over the years. In 2000 the arithmetic average of OECD countries in reading was set at 500 with a standard deviation of 100. The same was done for mathematics in 2003 and science in 2006.

The age of 15 was chosen because at this age the students are reaching the end of compulsory education in most OECD countries (ORGANISATION FOR ECONOMIC..., 2009). In Brazil this is the correct entrance age for the first year of secondary school, the ninth grade for PISA.

In this article it is shown that PISA suffers from a serious problem of comparability of results in many countries among the different assessment years, and probably also between countries in each assessment year. The problem is that there is no fixed rule for the application of the test in relation to the beginning of the school year and/or in relation to school age (age in terms of years completed at a particular month for entrance to a particular school grade, which may sometimes only be a recommended age).

For each school grade there is usually a correct (or recommended) school age and it is the school age of the student which determines in which grade he or she should be. The student can be ahead or behind this grade. One difficulty is that the definition of school age can vary between countries.

The age of 15 for PISA is defined as follows: "at the time of the test, the student should be between 15 years and 3 months and 16 years and 2 months" (ORGANISATION FOR ECONOMIC..., 2010, p. 7). This definition has no relation with school age.

Changing the month when the test is given implies changing the composition of students in relation to the various school grades and thus a change in the population, which makes them incomparable. This has been happening.

Another problem with changing the test month is that the 'school time' of students changes. There is a difference between giving the test at the beginning of the first semester and the beginning of the second.

What would be most appropriate would be to select students using the school age of 15 years in the country in question and for the test to be given a fixed number of months after the beginning of the school year. These rules should apply to all countries and for all years.

International comparisons are difficult, since there are great variations among the educational systems of different countries. For example, the school entrance age for the first grade in the regular educational system (primary education in Brazil) varies from country to country. In some countries this age is six years and in other seven. In the countries where school is entered at six, the grade corresponding to fifteen years of age is the tenth grade, while in countries where students start at seven it is the ninth grade. Brazil is in the latter case, although it is moving towards the first. Since PISA does not make this distinction, the seventh grade of those countries where school starts at six corresponds to the sixth grade of the others, who are ineligible for the test. The rules of PISA should cover these possibilities.

This problem also appears in the definition of years of schooling. How can years of schooling upon completion of basic education be compared, when in one country basic education lasts for twelve years (students start first grade at the age of six) and in the other it last eleven years (students start first grade at the age of seven)?

It should be noted that PISA has made great contributions in the development of how to assess and what should be assessed, and the comparison between countries is extremely important. The problems of comparability highlighted in this article do not affect the qualitative interpretation of how things are in countries, but they do affect the comparison of different years in the same country due to changes in the definition of age, as well as comparisons between countries.

Announcing the 2009 results, PISA stated that Luxemburg, Chile and Brazil were the three countries that had progressed best between 2000 and 2009. However, the student populations between 2000 and 2009 are not defined in the same way in these three countries.

In section two the case of Brazil will be analyzed in detail - which has three age definitions -, section three will examine the cases of Luxemburg, Chile, and Argentina, which have two different situations, and Mexico and the United States, both of which have three different situations. Also analyzed will be four examples of countries with an equal age definition over the four assessment years.

Finally, in section four the conclusions will be presented.

The Case of Brazil

Using the PISA data set, available at its site, the birthdates of the students in each PISA assessment year were found. The following information was obtained for Brazil:

• 2000 (years completed on 30 June). 1 July 1984 - 30 June 1985

• 2003 (years completed on 30 April). 1 May 1987 - 30 April 1988

• 2006 (years completed on 30 April). 1 May 1990 - 30 April 1991

• 2009 (years completed on 31 December). 1 January 1993 - 31 December 1993

As mentioned in the introduction, the eligible population for PISA is the 15 year old population studying in the seventh grade/eight year of basic education and above. Tables 1 - 3 show the proportion of the 15 year old population in school, the eligible proportion and the proportion per school grade among those eligible for each of the PISA assessment situations, age in terms of completed years on 30 June, 30 April and 31 December. The errors in the estimates are in brackets.

These estimates were made using the Pesquisa Nacional por Amostra de Domicílios (PNAD) (National Household Sample Research) data set. PNAD is carried out annually by the official statistical agency of Brazil, Instituto Brasileiro de Geografia e Estatística (IBGE ) (Brazilian Geographical and Statistical Institute), except for those years when the demographic census is carried out. The date for collecting the data is the Friday of the last full week in September.

Tables 1 - 3 include the years 1999 and 2001 in place of 2000, since this was a census year. In the three tables it can be seen that there was an increase in the proportion of the population in school and in the eligible population between 1999 and 2009. The proportions in school are greater in the case of years completed on 30 June and lower for those completed on 31 December. Nevertheless the proportions of those eligible are very close.

In Brazil, the 9^th, 10^th and 11^th grades correspond, respectively, to the 1^st, 2^nd and 3^rd grades of the secondary school.

In the three cases it can also be seen that there was an improvement in the distribution of eligible students among the school grades, increasing the proportion in the more advanced school grades.

In Table 1, years completed on 30 June, in which the proportion of eligible students in the first year of secondary school (9^th grade) is greater, this grade is clearly the modal school grade, which agrees with the idea that years completed on 30 June is the definition of school age in Brazil. Nevertheless, it can be noted that there was an increase in the proportion of students in the second year of secondary school (10^th grade) between 1999 and 2009.

Table 2 shows that the change in the definition of age in terms of years completed from 30 June to 30 April increases the proportion of advanced students, i.e., in the second and third years of secondary school, while that of the other grades falls.

The change in the definition of age to years completed on 31 December of the previous year increases even further the proportion of advanced students and reduces those in the other school grades, as shown in Table 3. This table also shows that the proportions of eligible students in the first and second years of secondary school are very similar, though in 2009, the proportion in second year of secondary school is greater. If all the students were in the correct school grade for their age, and if this were defined in terms of years completed on 30 June, half of the students should be in the first year in secondary school and the other half in the second year of secondary school.

Table 4 shows the proportions expected for each year in accordance with the definition of age used for that year. For 2000, data from 1999 and 2001 was used.

It can be seen in Table 4 that the composition of students changes in accordance with the definition of age and its influence on the second year of secondary school. In this grade the proportion in 2000 is around 11%, rising to around 20% in 2003 and 2006 and to 36% in 2009. In 2009 the proportion of students in the first year of secondary school falls a lot, as well as in the grades of the primary school. Another notable fact is that the proportion of students in the third year of secondary school also rises with the change of criteria in the definition of age.

Finally, Table 5 shows the proportions observed in PISA. It should be noted that all the calculations using the PISA data set complied with the recommendations of the PISA 2003 Data Analysis Manual (ORGANISATION FOR ECONOMIC..., 2005).

In Table 5 the proportions for 2000 are per discipline, since the student sample depends on the discipline, but all the proportions per grade are essentially the same.

In general terms, it can be noted that the proportions observed per grade follow the expected proportion pattern, indicating three different populations, one for 2000, another for 2003 and 2006 and finally a third for 2009. The proportion of students in the tenth grade (second year of secondary school) rises from around 9% in 2000 to 18% in 2003 and 2006, and finally reaches 36% in 2009. In the only two comparable years, 2003 and 2006, there is an improvement in the school flow, the proportion of students in the seventh and eighth grades falls, while that in the ninth grade (first year of secondary school) rises. It is stable in the tenth and eleventh grades.

Below are the averages obtained per grade and the total per discipline for each year PISA was held, with the standard error of the estimates in brackets.

It can be noted in Table 6 that the fall in the averages in reading in the first year of Secondary school between 2000-2003 and 2006-2009 is significant. Furthermore, the averages in the seventh and eighth grades are significantly lower in 2006 than in 2000. The average of the eighth grade in 2009 is significantly lower than in 2000. This suggests a fall in performance between 2000 and 2009. In the second year of secondary school the differences in averages are not significant. Nevertheless, the general average in 2009 is significantly greater than the averages for 2000 and 2006. It is almost significantly greater than the average in 2003. The only explanation of this is that the change in the composition of the student body per class and the improvement in school flow were such that they greatly compensated the fall of averages in some grades. In short, it cannot be stated that there was an increase in learning per grade.

In relation to mathematics, Table 7 shows that the increases in the averages of the seventh and eighth grades of basic education are in general significant, but not those of the other grades. Nevertheless, the general averages rose from 2000 to 2009, with the differences being significant. Thus, in mathematics there was an improvement in performance only in primary education. The general gain is once again due to the change in the composition of the student body per grade and an improvement in school flow.

Finally, in science, as shown in Table 8, the differences in averages between the different grades are not significant. As stated above, the increase in the general average in 2009 is due to the change in the composition of the study body per grade and an improvement in school flow.

It was stated in the introduction to this paper that a disturbing factor in the analysis is the different times at which the tests are held in relation to the school year, which is something that cannot be controlled. The difference in the time the test was held of around six months between 2000 and 2009 is very large. It is possible that if it had been held in the second semester of 2009, with the same population, the grade averages would have been greater. The only comparison with the same definition of age is between 2003 and 2006.

To better understand the change in the composition of populations and the improvement in the distribution of students among different grades, a simulation of results is presented in Tables 9-11, where each entry on the table is the weighted average of the averages per grade for the year indicated in the columns, while the weights are given by the observed proportion of students per grade for the test year indicated in the lines. In each column of the tables the averages per grade were kept fixed and in each line the proportions. The diagonals reproduce the real averages.

It can be noted that in each column in the three tables the values are rising, indicating that the improvement in the distribution of students per grade raises the 'general' averages.

Looking at the individual lines, different conclusions can be drawn for each discipline. In Reading practically equal general averages were obtained between 2000 and 2003, and between 2006 and 2009, but there was a fall between 2003 and 2006.

In mathematics very similar values were found between 2006 and 2009, though values rose from 2000 to 2009, indicating an increase in performance.

In science there were very similar values between 2003 and 2009, while those of 2000 were a little lower, though the differences are probably not significant, indicating stability during the decade.

Another type of comparative analysis is to consider only those students who satisfy the age definition. We have two cases, 2003-2009, with the months of May to December common to both definitions, and 2000-2009 with only the months of July to December common to the three situations. The results of the first case are contained in Tables 12-15 and those of the second in Tables 16-19.

In Tables 12 and 16 which contain the proportions per grade, an improvement can be noted in the distribution of students among grades, with the proportions in the higher grades increasing. However, it can also be observed that the lower the number of common months among the age definitions, the lower the students in the second and third years in secondary school and the higher the proportion of students in the first year of secondary school. In the primary education grades the proportions tended to fall with the years.

The analyses of averages per grade are analogous to what has already been observed in the real case. There is some evidence of an increase in the average in the seventh and eighth grades in science in 2009.

Other Countries

For the other countries we do not have PNAD (IBGE, 2011) type data to calculate the proportion of the population in school, the eligible population, and proportions expected per grade among those eligible. It would be interesting if the PISA reports highlighted the proportions of students in school and of eligible students.

Therefore, we will only consider PISA data sets.

We will start with Luxemburg and Chile, countries mentioned in the announcement of the results of PISA 2009.

The Case of Luxemburg

The PISA data set indicate that the birthdates of students in each assessment year for Luxemburg were as follows:

• 2000 (years completed on 30 January). 1 February 1984 - 31 January 1985

• 2003 (years completed on 30 January). 1 February 1987 - 31 January 1988

• 2006 (years completed on 31 December). 1 January 1990 - 31 December 1990

• 2009 (years completed on 31 December). 1 January 1993 - 31 December 1993

As can be seen there are two age definitions, one in 2000 and 2003 and another in 2006 and 2009. The change in definition is small, only one month.

According to the data there was an improvement in the flow from 2000 to 2003. There was an increase in the proportion of the tenth grade from 25% to 29%, the proportion in the ninth grade remained stable, it fell from 17% to 15% in the eighth grade, while the almost 2% in the seventh grade ceased to exist.

The change of a month brought new alterations: a new increase in the tenth grade, which rose to 35%, a fall in the proportion in the eighth grade, and a small fall in the ninth.

In this way there was a continuous improvement in the composition of the student body per grade, with the proportion of students in the highest grade increasing, while those in the modal grade fell slightly and those in the lowest grade declined. In 2009 approximately 52% of students were in the ninth grade, the modal grade, approximately 36% were advanced in the tenth grade, and around 12% in the eighth grade. What is the definition of school age in Luxemburg? It does not seem to be years completed in December, since the proportion of advanced students is high.

The modal grade is ninth grade, indicating that students enter first grade at the age of seven.

In the tables with the averages per discipline, a great jump can be observed from 2000 to 2003 in all the grades and in all disciplines. With this and with a small improvement in the flow, the general averages increased a lot. There was an increase of around 40 points (40% of the standard deviation) in reading and science and 50 points (half the standard deviation) in mathematics in only three years.

Maintaining the averages per grade from 2003, there should have been an increase in the general averages. But this was not what happened. The general average fell in reading and remained stable in the other two disciplines.

In reading and mathematics the average of the ninth grade fell in 2006, but the general average remained the same, since the number of students rose in the tenth grade and fell in the eighth. The general average remained the same due to changes in the composition of the student body.

In 2009 the average in the eighth grade fell in all disciplines and in reading, also in the tenth grade.

How can we say that there was a great rise between 2000 and 2009? The increase that occurred was only between 2000 and 2003, a very large increase for a three year period, even taking into account the small improvement in the distribution of students per grade. It is important to investigate what happened.

If we simulate the general averages in 2006 and 2009, with the proportion of students per grade in 2003 (age definition in 2003), we can see that there was a fall in performance in relation to 2003, Table 24.

The Case of Chile

PISA 2000 was only held in Chile in 2001, a year after it was scheduled. Chile did not participate in PISA 2003. For Chile the following data applies:

• 2001 (years completed on 30 June). 1 July 1985 - 30 June 1986

• 2006 (years completed on 30 April). 1 May 1990 - 30 April 1991

• 2009 (years completed on 30 April). 1 May 1993 - 30 April 1994

In Table 25 it can be seen that the modal grade in Chile is the tenth grade, indicating that students enter first grade at the age of six. As a result, Chile has eligible students in the seventh grade, corresponding to the sixth grade in Brazil and Luxemburg. In 2003 this contingent was 2%, falling to 1% in 2006 and 2009. This is not fair for the countries whose students enter first grade at the age of six, since they may have one lower grade more than the others.

The change in the age definition by two months had the effect of increasing the percentage of students in the tenth grade by 10 percentage points, resulting in the appearance of approximately 6% of advanced students in the eleventh grade and a reduction of the percentage of students in the seventh, eighth and ninth grades. This modified the composition of the student body, giving more weight to the correct and advanced grades.

In Chile there was a real increase in the tenth and eighth grades in all disciplines, and in the ninth grade in 2009, in reading and mathematics. This, together with a change in the composition of the student body, implied real progress in the general averages between 2000 and 2009. However, the real increase is lower than the approximately 30 points in each discipline.

If we change the age definition in 2000 (2001), i.e., if we calculate the 2000 general average with the 2000 averages per grade and the 2006 proportions, we would obtain the values of 427.62, 400.33, 430.48, respectively, for reading, mathematics and science. There would be an increase in approximately 20 points for each discipline.

The Case of Argentina

Like Chile, Argentina only held the 2000 PISA in 2001 and did not participate in PISA 2003. In Argentina children start first grade at six, thus the modal grade is the tenth. Table 28 shows that there are students in the seventh grade, equivalent to the sixth grade in Brazil, which is unjust for the comparison. The age definition in 2001 was equal to that of Chile, but the change of definition in 2006 was different. While Chile moved forward by two months, Argentina only moved one.

• 2001 (years completed on 30 June). 1 July 1985 - 30 June 1986

• 2006 (years completed on 31 May). 1 June 1990 - 31 May 1991

• 2009 (years completed on 31 May). 1 June 1993 - 31 May 1994

In relation to Argentina, it can be seen in Table 28 that there has been a worsening of the distribution of students per grade, with more students in the lower grades. This can be very clearly seen from 2006 to 2009, which have the same age definition. From 2001 to 2006, with the moving forward of the date by one month, it was expected, as had happened in the other countries analyzed, that the percentage of students in the tenth grade would rise, that students would appear in the eleventh grade and the percentage of students in the lower grades would fall. But it was not this that happened. In relation what was expected, only the appearance of students in the eleventh grade occurred, and to the contrary, the percentage of students in the tenth grade fell.

Another surprising find was the considerable fall observed in the averages per year in reading from 2000 to 2006. The same was not observed in mathematics and science. What happened in reading?

Simulating the 2000 averages with the proportion observed in 2006, the following averages were obtained, respectively for reading, mathematics and science: 413.37, 382.03, 392.70. An enormous fall was observed in reading between 2000 and 2006, (40 points), though mathematics and science were stable.

Even with a worse distribution of students per grade in 2009 in relation to 2006, the increase in the averages per grade in 2009 was sufficient to guarantee an increase in the global averages from 2006 to 2009.

The Case of Mexico

Mexico, like Brazil, had two alterations of its age definitions. Years completed on 30 November, 28 February and finally 31 December. The first change tended to worsen the distribution of students per grade and the second to improve it, but there was no return to the first situation.

• 2000 (years completed on 30 November). 1 December 1983 - 30 November 1984

• 2003 (years completed on 28 February). From 1 March 1987 - 28 February 1981

• 2006 (years completed on 31 December). From 1 January 1990 - 31 December 1990

• 2009 (years completed on 31 December). From 1 January 1993 - 31 December 1993

Table 33 corroborates this expectation. The proportion of students in the modal grade, the tenth, fell from 50% to 43% between 2000 and 2003, rising again to 48% in 2006. A curious fact is that the percentage of advanced students, in the eleventh grade, was always lower than 1%, except in 2006, when it rose to 5%. In 2009, apparently these 'advanced students' were in the tenth grade. It can also be noted that the proportion of late students fell a little in 2009.

Mexico, like Chile and Argentina is penalized in the calculation of the general average, since it incorporates students from the seventh grade, equivalent to the sixth in Brazil.

In Mexico there was a fall in the averages per grade between 2000 and 2003 in reading and a less accentuated fall in science. There were increases in the averages per grade in all the disciplines between 2006 and 2009.

In Table 37 we simulated the averages of 2000, 2003 and 2006 with the 2009 proportions per grade. What was observed was a fall and recovery in reading, an advance in mathematics and a fall and a slight recovery in science.

The Case of the United States

The United States also had three age definitions, with a great change between 2000-2003 to 2006-2009 - six months-, as in Brazil, and with great changes in the composition of students per grade. The distribution of students improved dramatically between 2000 and 2009. The proportion of students in the modal grade, tenth grade, rose from 56% in 2000 to 69% in 2009, and that of the advanced grade, eleventh grade, from less than 0.5% to 20%, while those of the lower grades, fell from 40% to 11% for the ninth grade, and from 3% to almost 0% for the eighth grade.

It should be noted that 2003 included students born between 1 January 1987 and 31 May 1988, more than a complete year.

• 2000 (years completed on 31 December). 1 January 1984 - 31 December 1984

• 2003 1 January 1987 - 31 May 1988

• 2006 (years completed on 30 June). 1 July 1990 - 30 June 1991

• 2009 (years completed on 30 June). 1 July 1993 - 30 June 1994

In 2006 there were no results in reading for the United States.

In the United States students also enter first grade at the age of six, with the tenth grade being the modal for 15 year old students. Once again, in some years, there appear students from the seventh grade.

The averages per grade in mathematics and science improved between 2006 and 2009, but in all the disciplines the averages fell significantly between 2000 and 2009 in the relevant grades, the ninth, tenth and eleventh, which implies enormous falls in the general averages, not reflected in the results because of changes in the composition of the student body per year. Table 42 shows the general average for 2000, per discipline, simulated with the 2009 proportions, where the falls from 2000 to 2006 appear, as well as the beginning of a recovery in 2009. As in the case of Brazil, it should be emphasized that the interval of around six months between the months when PISA was held in 2000 and 2006/2009 is very large and it might be possible that the result of the same students in 2006 and 2009 could have been better, if PISA had been held in the same month as in 2000.

The Cases of Japan, Korea, Finland and Sweden

Finally, four high performance countries are presented which had the same age definition in all four test years, namely:

• Japan: years completed in March. (There are a few 14 year old students who were born in April).

• Korea: years completed in February.

• Finland: years completed in January.

• Sweden: years completed in December.

In Japan 100% of students are in the tenth grade, Korea starts with 98% in this grade in 2000 and finishes with 95% in 2009, while the percentage in the ninth grade rises from 1% to 4% (Table 43). Finally in Finland around 88% are in the ninth grade and 12% in the eighth (Table 43). This results in a question: in Finland are there approximately 12% of students late every year or is there a problem of date? If the age in Finland was years completed in December would it still have this delay? In the neighboring country, Sweden, the students assessed by PISA use years completed in December, while the percentage in the modal grade, ninth grade, is around 95%, in all test years (Table 45).

The entrance age in the first grade is six years of age in Japan and Korea, while in Finland and Sweden it is seven.

The performance of the students of these countries shall not be discussed here, as they do not involve any novelties for the analysis.

Conclusion

Presented in this paper were the results of six countries, three from the northern hemisphere (Luxemburg, Mexico and the United States) and three from the southern (Brazil, Chile and Argentina).

It was noted that in these six countries the age definition varied between the different test years and between the different countries. Also shown was the age definition of another four countries (Japan, Korea, Finland and Sweden), all of which have different age definitions.

This fact shows the difficulty of international comparisons. When does the school year begin in each country? What is the age definition used in schools? What is the correct grade for 'fifteen year old' students? The answer to the last question depends on the entrance age in first grade, which varies between countries.

Age is a continuous variable, but school age is discrete. The month and the year of students' birth define in which grade they should be. As shown, the change in the month PISA is held and the consequent change in the age definition, changes the composition of the student body among the different grades and impacts students' global averages.

Also shown in the article was that the three countries with the greatest progress during the decade, namely Luxemburg, Chile and Brazil, had changes in the date the test was held and consequently changes in the age definition.

Luxemburg made great advances between 2000 and 2003, but afterwards remained stable or fell back, depending on the discipline. Chile had consistent advances in all the disciplines, but less than pointed out by PISA and, finally, Brazil, which probably had a fall in Reading, advanced in mathematics and remained stable in science. The change of around six months in the date the test was held between 2000 and 2009 hinders any affirmation.

Of the other three countries analyzed, Argentina presented a worsening flow, a large fall in performance in reading between 2000 and 2006, but an improvement of the global and grade averages from 2006 to 2009. Mexico had a fall in 2003 and a recovery in reading and science only in 2009. In mathematics there was an advance. Finally, the United States appeared to have a large fall in performance between 2000 and 2006 and the beginning of a recovery in 2009. As in the case of Brazil the change in the month the exam was held between 2000 and 2006/2009 was approximately six months.

It is urgent to correct PISA 2012. The most appropriate correction is to select students by the school age of 15 and hold the test a fixed number of months after the beginning of the school year. This rule should apply for all countries and for all test years.

References

• IBGE. Pesquisa Nacional por Amostra de Domicílios Rio de Janeiro, 2011. Disponível em: <http://www.ibge.gov.br/home/estatistica/populacao/trabalhoerendimento/pnad2009/>. Acesso em: 22 nov. 2011.
• ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. Pisa 2003 Data Analysis Manual Paris, 2005. Disponível em: <http://www.oecd.org/dataoecd/53/22/35014883.pdf>. Acesso em: ago. 2011.
• ______. Pisa 2009 assessment framework: key competencies in reading, mathematics and science. Paris, 2009. Disponível em: <http://www.oecd.org/document/44/0,3746,en_2649_35845621_44455276_1_1_1_1,00.html#TOC>. Acesso em: ago. 2011.
• ______. Pisa 2009 at a Glance Paris, 2010. Disponível em: <http://www.oecd.org/dataoecd/31/28/46660259.pdf>. Acesso em: jul. 2011.

A reanalysis of PISA results: comparability problems

Un reanálisis de los resultados del PISA: Problemas de comparabilidad

Publication Dates

• Publication in this collection
  16 Oct 2012
• Date of issue
  Dec 2011

History

• Received
  28 Sept 2011
• Accepted
  03 Nov 2011