RATIONALES FOR AND RECEPTION OF THE OECD'S PISA

ABSTRACT:

Set against the backdrop of the education work of the Organization for Economic Co-operation and Development (OECD) and the changing place of Program for International Student Assessment (PISA) within it, the paper analyzes the OECD's rationale for PISA. It involves an account of the media strategies, the OECD's education, and skills that the directorate utilizes to manage the PISA message. The document then outlines the various rationales that the nations use for participating in PISA. The national usage of PISA often involves externalization rather than policy learning, with comparative performance on PISA being used to legitimize the policy reforms which are usually not connected to the policies in top performing nations. Three cases of such externalization are proffered as responses to the United States, England, and Australia in comparison to Shanghai's outstanding performance on PISA 2009.

Keywords:
PISA; externalisation; policy rationales; policy reception

RESUMO:

Levando em consideração o trabalho da Organização para a Cooperação e o Desenvolvimento Econômico (OCDE) na educação e o lugar transformador que o Programa Internacional de Avaliação de Alunos (Pisa) ocupa nela, o presente artigo analisa a fundamentação da OCDE para o PISA. Para tal, abordam-se as estratégias de comunicação, a educação da OCDE e as habilidades que a Diretoria utiliza para administrar a mensagem do PISA. O documento também descreve as várias fundamentações que as nações empregam para participar no PISA. Os usos que são feitos do PISA no contexto nacional envolvem mais frequentemente a externalização do que a aprendizagem política, sendo os resultados comparativos do PISA aplicados para legitimar reformas políticas geralmente desconectadas das políticas de países com melhor desempenho no teste. O artigo apresenta três casos de tal externalização, levando-se em conta as respostas dadas nos Estados Unidos, na Inglaterra e na Austrália ao excelente desempenho de Xangai, China, na prova do PISA de 2009.

Palavras-chave:
PISA; externalização; fundamentações políticas; acolhimento de políticas

RÉSUMÉ:

Tenant compte l'intervention de l'OCDE sur l'éducation et la place changeante du PISA en son sein, cette article analyse la rationalité de l'OCDE sur PISA. Cela implique un compte des stratégies de communication utilisées para la Direction-Générale de l'Éducation et de la Culture pour gérer le message du PISA. On décrit les divers arguments que les nations utilisent pour participer au PISA. Les utilisations qui sont faites du PISA dans des contextes nationaux impliquent plus souvent des processus d''externalisation que d'apprentissage politique, et les résultats comparatifs du PISA sont souvent utilisés pour légitimer des réformes politiques généralement déconnectées des politiques des pays meilleurs performantes au test PISA. On présente trois cas d'externalisation, en prenant en compte les réponses données aux États-Unis, en Angleterre et en Australie face aux excellentes performances de Shanghai dans le test PISA 2009.

Mots-clés:
PISA; externalisation; rationalités politiques; réception de politiques

Introduction

First of all, this paper considers two sets of rationales for the Organization for Economic Co-operation and Development (OECD)'s Program for International Student Assessment (PISA): the OECD's rationales and the rationales for the nations to participate in this most influential of the International Large Scale Assessments (ILSAs). The participation at PISA is nowadays more than double of the 34 OECD member nations and it is still expanding. The second focus is the "reception" of PISA results, particularly with respect to the international performance league tables that are constructed by the OECD and are usually expressed in a grid mapping both equity and quality. An important aspect of PISA testing is that, as per a central OECD finding, there have been two dimensions of educational outcomes - quality and equity -, which actually go together and do not come against each other. As Condron (2011) has clearly shown, these are compatible goals for the educational systems in affluent countries. Reception here refers to the policy usage made by the nations and to the schooling systems with their fluctuating comparative performance on PISA over time. The first test was administered by the OECD in 2000 and after that every three years.

In terms of the OECD's rationale for PISA and for the nations' participating in it, the paper will draw on the research that I have conducted with others at the OECD, on its education work in the latter part of the last century (e.g., Henry et al., 2001), as well as the investigation conducted there more recently involving more than 50 research interviews (e.g., Lingard & Sellar, 2016). This analysis is going to use a heuristic typology of rationales for their participation in PISA developed by Addey and Sellar (2016). This typology was developed to cover the rationales for the participation of the nations in multiple kinds of ILSAs, including PISA, but also the International Association for the Evaluation of Educational Achievement's (IEA) Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS), as well as the testing under the jurisdiction of the United Nations Educational, Scientific and Cultural Organization (UNESCO) and some regional cross-national testing. However, their typology aptly applies to national rationales for participation in PISA.

The PISA reception part of the paper will demonstrate the "externalization" involved in this usage (Schriewer, 1990; Waldow, 2012). Externalization here refers to the nations using the PISA results to drive reforms which are already underway, and using PISA performance as an argument for doing so as a legitimate narrative. We will see in these cases that policy borrowing or even policy learning (Steiner-Khamsi & Waldow, 2012) do not occur. We are also going to see what we might regard as "PISA shocks" being particularly powerful in terms of practices of externalization, specifically in respect of Shanghai's 2009 PISA performance. The OECD's PISA has also provoked the construction of new reference societies or reference systems for national schooling systems (Sellar & Lingard, 2013). From 2000 to 2009, Finland was the PISA poster child because of its outstanding PISA performance, resulting in enhanced educational tourism to Finland by policy makers from around the world.

The OECD's PISA has constituted the globe as a commensurate space of measurement and, as such comparative performance on it, has reconstituted reference systems away from older colonial, trade and cultural alliances and alignments. So, for example, out of their performance on PISA, Finland and Shanghai have become reference societies for many educational systems.

In what follows, a brief historical background to the OECD, its work in education, and the changing place of PISA within it is provided. Then the paper turns to a commentary on the OECD's rationale for and usage of PISA. Next, it considers the rationales for the participating nations. The essay then looks over the case of Shanghai's 2009 PISA performance and its reception in three nations, the United States, England, and Australia. There will be an acknowledgment here as well, that media representations of national performance on PISA have been very important in terms of national reception. The paper concludes with the salient points from the argument proffered.

The OECD's education work and PISA

The OECD was created in 1961 with 20 member countries, all of them European, except for the United States, and with Japan becoming a member in 1964. Membership nowadays stands at 34 nations. The OECD from its beginning focused on economic policy for its rich member nations and served as a bulwark against Soviet communism. The closure of the Cold War ended that ideological role as the organization repositioned itself in the context of a global capitalist economy framed by neo-liberalism, and also contributed to such developments.

Membership of the OECD requires commitment to three broad values: a market economy, liberal democracy and, more recently, human rights. Its chief aims are enhancing economic growth, international trade, and economic development. The conclusion of the Cold War precipitated a minor crisis of purpose for the organization, and subsequently it has become more important to produce international comparative statistics for policy-making.

In 2008, the Global Financial Crisis (GFC) presented another challenge for the OECD, and subsequently it has begun to rethink its economic stance, including a focus beyond gross domestic product (GDP) as the primary growth indicator to include well-being and concerns about inequality and social cohesion. In that context, its skills strategy was an attempt to ensure policy coherence across all the OECD's directorates and an indicative of the enhanced significance of education in the work of the organization.

Because of its complex relationships with members and committee structures, the OECD is an important node in a network of relationships, including non-member countries and other international organizations. The OECD has been depicted as a "geographical entity, an organisational structure, a policy-making forum, a network of policy makers, researchers and consultants, and a sphere of influence" (Henry et al., 2001, p. 7). Carroll and Kellow (2011) demonstrate that the OECD exerts soft power through processes of mutual surveillance and peer pressure. These processes have grown with the enhancement of the OECD's statistical work from the mid-1990s. It included the creation of Education at a glance indicators, a data compendium from member countries, as well as the development of the PISA, which was first administered in 2000.

Up until the late 1990s, education had an "inferred role" inside the OECD (Papadopoulos, 1994). The establishment of the Centre for Research and Innovation (CERI) in 1968 provided education with a firmer position within the organization. In 2002, a separate Directorate for Education was established, and in 2012 it became the Directorate for Education and Skills, indicative of the enhanced importance of education within the OECD and globally. The OECD's education policy analyses have included national reviews of school systems and more thematic, comparative reviews across national systems. More recently, however, the balance of the organization's education work has shifted from reviews of this kind towards large-scale quantitative assessments such as PISA and PIAAC, the Program for the International Assessment of Adult Competencies, a PISA for 16-64-year-old ones.

Across the 1990s, there was pressure from members for the OECD to develop international comparative educational statistics. While the organization conducted national reviews of systems, from that time onwards, there was a focus on more thematic reviews and international comparisons. Statistical data became increasingly significant in relation to these country and thematic reviews. Henry et al. (2001) have documented the increasing significance of quantitative data at the OECD across this period and especially during the 1990s, with strong support from the United States. In 1988, the CERI governing board created the Indicators of National Education Systems (INES) project. The initial Education at a glance indicators were released in draft form in September 1991 and have become a more significant part of the OECD's statistical work. Henry et al. (2001, p. 90) sum up this shift in the OECD's stance, "From philosophical doubt to statistical confidence; from covering some countries to covering most of the world; from a focus on inputs to a focus on outputs; and from occupying an experimental status to being a central part of the organization's work."

It was in this context that a 1997 OECD report recommended the creation of what was to become the PISA. The United States was an important player in the ranks for this development.

The PISA tests were first administered in 2000, with the analyses and performance rankings released in 2001. This led to the now well-documented PISA shock in Germany (Grek, 2009), which performed much worse than the German self-perceptions of the quality and equity of their schooling system. There have been other PISA shocks as well, including Japan (Takayama, 2008), and following the release of the 2009 results, which show Shanghai, China, as a top performing system (Sellar & Lingard, 2013; Waldow, Takayamaand & Sung, 2014), and thereby becoming a major focus of this paper. The impact of PISA resulted in varying forms and intensities (Carvalho & Costa, 2014, p. 3). The PISA shock in Japan, for example, led to the inclusion of PISA type application of knowledge questions into the three layers of testing at the national, prefecture and municipal levels. A broader effect of PISA has been the growth of complementary national testing of schools.

Since 2000, PISA has been carrying out national samples of 15-year olds, every three years, with increasing numbers of nations participating since the first test, including 2015, when there were more non-members than member countries. PISA measures reading, mathematical and scientific literacy tests, and putative tests that all 15-year olds ought to know and ought to be able to apply in "real world situations". It was conducted online for the first time in 2015.

PISA tests have helped constitute a global educational policy field through its work, in constituting the globe as a commensurate space of measurement (Lingard & Rawolle, 2011). Wiseman (2010, p. 8) notes:

What widely available international data on education has done is create an intellectual space where educational policy-making is not geographically or politically bounded but is instead bounded by the extent of the legitimated evidence used to support one decision or policy versus another.

Performance on PISA by the national systems is ranked according to quality (scores on the tests) and equity (the spread of the scores and strength of correlations between socio-economic background and performance). As noted, PISA has differing effects on national schooling systems and their policies. More recently, the OECD has sought to expand the scope of PISA (what is tested), scale (number of participating nations), and explanatory power (linking PISA results to other tests) (Sellar & Lingard, 2014). There has also been the development of complementary PISA tests, for example, PISA for Schools (Lewis, Sellar & Lingard, 2016) and PISA for Development, a version of PISA for developing nations.

The OECD and PISA rationales

The United States pressured the OECD throughout the 1990s to create what became PISA. America wanted a measure of the international comparative performance of American schooling. With the globalization of the economy, such measures of the quality and quantity of nation's human capital became surrogates for the putative competiveness of the national economy within the global one. In this context, PISA became attractive to many nations, including beyond the membership. Here education policy was reframed as central to economic policy and under the "control" of the nation.

Insiders at the OECD have described it to me as a "non-academic university" that does research for policy, as opposed to the more academic research of policy (Lingard, 2013). PISA testing is part of this research, or here, data, for policy-making.

Mention has been made of so-called "PISA shocks". In a sense, this is the desired effect of PISA as seen by the OECD. Indeed, this rationale has been expressed to me in research interviews. OECD documents express this rationale for PISA in this way: "[E]vidence suggests that international pressure and competitive environments are more likely to diffuse a sense of ineluctability of some reforms among the various stakeholders and the public at large" (OECD, 2010, p. 315, emphasis in original) and again, "[E]xperience shows that more comprehensive reforms are possible when there is a widespread recognition of the need for a change to take place - e.g. in case of external pressure, competitive threat or common enemy" (OECD, 2010, p. 335). PISA is thus designed to unsettle complacency in education policy terms within participating nations.

My OECD research would suggest that the organization has become more of a "policy actor" in education (Henry et al., 2001; Lingard & Sellar, 2016). This is not to deny the complex relationships that the organization has with member nations and its complex committee structures in each policy domain that conjointly set the agendas, nor it is to deny the complexity and contestation inside the organization and the directorate. However, this role as policy actor can be seen in the OECD's use of the media in relation to getting a particular message across about PISA to participating nations.

In the research that I carried out at the OECD, the Media Relations manager, who handles the media coverage of PISA, suggested that his role was "to liaise between OECD experts and the outside world" (interview, OECD media officer, 2013). He added that he strived to do the work of the directorate more appealing to the media, while acknowledging that the media was more interested in certain sorts of stories as opposed to others. He worked closely with the head of the directorate and of PISA to get the message the OECD wanted to deliver to the media. In seeking to "control" the media message, extensive work was involved. So a few weeks before the first release of PISA results in December or thereabouts of the year following that in which the test was conducted, a notification was sent to a list of 4,500 journalists, for whom the OECD has contact details globally, alerting them to the imminent publication of PISA results. The media team then worked with the Education Directorate on a choreographed release of PISA results. One OECD interviewee noted that not all OECD reports had such focused media releases as the PISA report did: "We do these big media launches because it's a flagship, and in this particular case, it's also seen as breaking news, which is unusual for us" (interview, OECD media officer, 2013). The results are made public in press conferences in Paris, London, Brussels, Washington, and in other large cities.

The media team at the OECD is also involved in the production of the PISA report and of the short, media-oriented country notes on PISA performance. The Media Relations manager stated, "We work with [the Directorate] to work out a strategy for releasing it to the media" (interview, OECD media officer, 2013). The Education Directorate briefs journalists about the PISA results, seeking to manage the message that gets reported, and attempting to improve journalists' understanding PISA and PISA results. The Media Relations manager also noted that at this time the director is available for multiple media interviews: "Yeah. [The Director] thinks the media are very helpful in getting his message across to people he wants to. He's quite incredible - you know, you will send him an email and he will respond to the journalist himself. Never says 'no' to an interview" (interview, OECD media officer, 2013).

Besides, the Media Relations manager verified that the director's

core audience is with, ideally, the Education Ministers; if we had to choose one person to go to, it would be education. He sees the value of media as another way to reach Education Ministers and also reaching to a more general public; especially with the policy-makers. If the OECD brings out a report, it is very easy for the government Ministers to ignore it or be told about it by their staff, top down. Whereas, I am sure with PISA, the fact that it's all over the BBC in England, the Education Minister is going to be very aware of it (interview, OECD media officer, 2013).

We see in these quotes the OECD's attempt to manage representations of PISA as part of the organization's role in enhancing the policy usage of PISA.

Another OECD interviewee noted specifically the use of media to strengthen the political impact of PISA: "I think this organisation has made PISA more media focused and also more political... Of course, you tend to use that to reinforce the impact" (interview, OECD media officer, 2013). We see here the OECD in the role of "policy actor" working through the media linked to the organization's agenda setting role.

The Media Relations manager also noticed how journalists within nations wanted league tables of performance on PISA and regularly sought explanations of a specific nation's standing or ranking on the test. He added how he spent a lot of time explaining national performance, including international comparative performance to national journalists. Indeed, much of his time was taken up on this task in the weeks following the initial release of the PISA results. While he realized how the OECD was in some ways opposed to league tables of performance, he suggested:

It was obvious there was going to be a league table. But we don't want to encourage people to go down that line. Because with PISA, the frustration with the league table is: even though you have the top five countries or countries with half points difference, there's no difference at all, but the countries still say "we are third and above X and Y," and you can't stop them (interview, OECD media officer, 2013).

He then added, "I wouldn't say we discourage that but we certainly don't encourage it. It is a lost cause, really" (interview, OECD media officer, 2013). The OECD's media strategy around PISA thus gives emphasis to league tables and not equity, with most nations responding to quality rather than equity concerns (Wiseman, 2013).

We can see the active role the OECD plays in seeking to frame the PISA stories reported in the media around the globe. We also verify the acceptance that the media want league tables, despite the OECD's reluctance about these. Besides, it should be noted that national media representations of a nation's PISA performance are also very important in the ways nations respond to the release of PISA results (Martens & Niemann, 2013).

National rationales for participation in PISA

Addey and Sellar (2016) have created a heuristic typology for understanding the various rationales for nations participating in ILSAs, which I accept as being by and large applicable to participation in PISA. They argue that nations make use of a mixture of these rationales and these can change over time. The first rationale they consider is the "evidence for policy." Across the period of the existence of PISA, we have witnessed the rise of "audit cultures" (Power, 1997) within nations, linked to restructured state bureaucracies and systems of governance. It has been accompanied by the rise of policy as numbers, and the usage of comparative data as a mode of governance linked in education to top-down, test-based modes of accountability. PISA data provide another set of evidence for policy-making. However, as the cases dealt with what will be showed ahead, this evidence for policy usage rationale does not always result in policy learning or borrowing, but more often results in "externalization" (Schriewer, 1990; Waldow, 2012), that is, the use of PISA data as a justification for domestic reforms. It is also important to note that OECD analyses of PISA tend to overplay the significance of policy in systemic performance and downplay structural inequalities (Meyer and Schiller, 2013).

The second rationale considered by Addey and Sellar (2016) is that of "technical capacity building". It sees a nation participating because it helps build technical capacity within the nation in respect of psychometrics and the most advanced modes of test development and data analysis. They include in these technical elements: item response theory, Differential Item Function (DIF), online adaptive testing and the like. While they correctly argue that this rationale is used most often by low and middle income nations, they also note that such a rationale was used by France for participation in ILSAs.

"Funding and aid" is the next rationale Addey and Sellar (2016) document. It applies most often to developing nations and is often linked to performance benchmarking and impact requirements of monetary aid. Sometimes, they suggest, this is linked to the necessity of appearing transparent and accountable. They speculate that it will be interesting to see how the new PISA for Development is going to be utilized in this way.

Belonging to the international community of nations, the international relations justification, according to Addey and Sellar (2016), is the next rationale for participation. Participation itself is the rationale here, rather than one based in the need to access useful performance data. Peer pressure from other nations is an element of this international relations rationale, as is the membership of an organization, which sees participation as a responsibility of membership. It should be pointed out that in 2000 not all OECD member nations participated in PISA. Nowadays, they all do. There is also a way in which the neo-institutional or global polity theory offers an explanation of this type, for participation in PISA: the global dispersal of a kind of Weberian technical-rationality as a demonstration of the modernity of any given nation. Addey and Sellar (2016) suggest nation-building aspirations can also constitute an element of this international relations rationale. Besides, they ask for many nations non-participation in such tests even an option. Complementary here is the desire of the OECD to expand the scale of participation of nation on PISA (Sellar & Lingard, 2014).

National politics provide another rationale for participation in PISA, according to Addey and Sellar (2016). This might result from the political goals and career ambitions of politicians, desires of those ones inside the various government bureaucracies and also pressures exerted by several lobby groups within nations. As noted before, in considering OECD rationales for PISA, international comparative performance measures such as PISA serve as a surrogate measure of the putative future competitiveness of national economies globally. The authors also see such an economic rationale for nations participating in PISA. Many nations, as Addey and Sellar (2016) notice, now regard educational expenditure as an economic investment and this serves as a rationale for participation in PISA. Attracting foreign investment and strengthening domestic investment in some industries are also seen by some policy makers within nations as being closely linked to evidence of good educational outcomes as measured on global metrics like PISA. Importantly, Addey and Sellar (2016) suggest that in some nations PISA performance is a way in which national education ministers are held accountable for their portfolios. They also note the ways that results on such tests can be used to glorify or alternatively scandalize national performance.

The final rationale they mention for nations participating in ILSAs, namely curriculum and pedagogy, does not so much apply to PISA, as PISA, unlike the IEA's TIMSS and PIRLS, is not based on national curricula, but rather purports to measure the capacity of 15-year olds to apply knowledge in maths, literacy, and science. Despite this, following some decline on PISA, Japan modified its national curricula to give more emphasis to the application of knowledge.

PISA 2009, Shanghai's performance and reception in the United States, England and Australia

Shanghai's outstanding performance in reading, mathematics, and science on PISA 2009 has drawn much attention to its schooling system. Shanghai is a top performer within China, and a leader in terms of educational reforms, which is providing a basis for policy learning in other parts of the country - Shanghai is an internal reference system. It is the only region of mainland China that participated publicly in PISA 2009 with its results published as part of the international data set. However, a number of other provinces in China was involved in PISA 2009 on the basis that they were exempted from making results public (Chan & Seddon, 2014). More results from China will be made public from the 2015 PISA. A research interviewee explained that China was "using PISA as a lever for improvement in other provinces, for the western provinces for example" and "[t]hey are going to ensure that they get things right before going fully into it. I think the plan is that more and more Chinese provinces come in as and when the infrastructure is there" (interview, OECD policy officer, 2013).

On PISA 2009, Shanghai out-performed all other participating nation countries, scoring 556 in the reading assessment (Finland 536, Australia 515, United States 500, England 495, average 493), 600 in mathematics (Finland 541, Australia 514, average 496, England 493, United States 487), and 575 in science (Finland 554, Australia 527, England 515, United States 502, average 501). Additionally, 12.3% of variation in reading performance in Shanghai can be explained by socio-economic background, compared with 12.7% in Australia, 14% in England, and 16.8% in America. This suggests that not only Shanghai's students were the top performers on the reading assessment, Shanghai's schools were also comparatively successful at enabling students to overcome disadvantageous socio-economic backgrounds. Shanghai was successful in terms of both performance quality and equity. A research interviewee observed that "people who know Shanghai were not surprised by those outcomes, but for people who don't know Shanghai's education system it was sort of a wakeup call" (interview, OECD policy officer, 2013). Many are very skeptical of the equity measure, given the exclusion of the poorest internal migrants from government schools and thus from the PISA sample.

The United States' response

President Barack Obama has been an activist education president, strengthening the role of the national government vis-à-vis the states. He supported state governors in the development of the common core curriculum. Race to the top demanded states test in relation to the common core and introduced particular reform agendas.

PISA had limited impact in the United States until 2010 (Breakspear, 2012). This remained the case despite comparatively poor performance on the test. Shanghai's outstanding performance on PISA 2009 changed this. Prior to the release of the 2009 PISA results in December, 2010, the director of PISA at the OECD (now director of the Directorate for Education and Skills) addressed the American Senate Education Committee. He warned that the United States was falling behind on high school completion rates and PISA results.

Following the release of the 2009 PISA results, there was a huge amount of media coverage. The headline in The New York Times (TOP TEST SCORES..., 2010) read, "Top test scores from Shanghai stun educators." In this report, former head of president Ronald Reagan's Department of Education, Chester E. Finn Jr., observed, "Wow, I'm kind of stunned, I'm thinking of Sputnik" (TOP TEST SCORES..., 2010). He continued, "I've seen how relentless the Chinese are at accomplishing goals, and if they can do this in Shanghai in 2009, they can do it in 10 cities in 2019, and 50 cities by 2029" (TOP TEST SCORES..., 2010). Arne Duncan, then secretary of Education, said, "We have to see this as a wake-up call." He added, "The United States came in 23rd or 24th in most subjects. We can quibble, or we can face the brutal truth that we're being out-educated" (TOP TEST SCORES..., 2010). President Obama was also quoted in The New York Times story noting how the 1957 launch by the Soviet Union had precipitated an educational shock in the United States, which led subsequently to increase investment in Science, Technology, Engineering and Mathematics (STEM) in schools and universities. He argued that with the stellar performance of Shanghai, America was facing another Sputnik moment. In his State of the Union Address on January 25, 2011, the president observed, "We know what it takes to compete for the jobs and industries of our time. We need to out-innovate, out-educate, and out-build the rest of the world" (TOP TEST SCORES..., 2010). In this media coverage and in the political responses to it, we see a PISA shock.

Reflecting a human capital construction of schooling, in a press release dated December 7, 2010, Arne Duncan quoted the president saying that the nation that "out-educates us today will out-compete us tomorrow" (TOP TEST SCORES..., 2010). Importantly, Duncan suggested that the United States nowadays had to look outwards to other national systems as a way to move American schooling forward. He subsequently commissioned a study from the National Centre on Education and the Economy (NCEE) and the OECD. The findings of this research were published in the book, Surpassing Shanghai: an agenda for American education built on the world's leading systems , edited by Marc Tucker (2011). The point to make about this case is that the PISA shock was used for externalized purposes. There has been little evidence of policy learning.

England's response

The case here is England because, while the United Kingdom is the unit of analysis for the OECD's PISA, and one of the OECD's 34 members, each member country of the United Kingdom pays for oversampling. This enables the disaggregation of PISA data at the level of England, Scotland, Wales and Northern Ireland. Under the New Labor government (1997-2010), England's self-perception was as a leader rather than follower of international testing trends. The New Labor government was confident about the quality of national test data and its usefulness for policy-making. That government expressed some skepticism about the usefulness of international comparative data like PISA for policy purposes. In a research interview, a policy maker noted, "up until now we've been very focused on our national data, we are probably ahead of other countries in terms of data" and "for the actual conduct of the [PISA] studies. I don't think we get anything new because we are so far ahead in terms of data collection than most countries" (interview, policy maker, 2009). This neglect of PISA ended in the lead-up to the change of government in 2010, and after Shanghai's 2009 PISA performance.

On being elected in 2010, the Coalition government used the comparative decline in England's PISA performance between the 2000 and 2006 PISA to criticize the previous Blair/Brown administrations. The Coalition government (conservatives and liberal democrats) utilized these data to articulate a narrative of declining standards. This narrative in turn was used to legitimize the new government's agenda for yet more school reform. This is an exemplary case of externalization. The Coalition and subsequent Conservative government (from 2015) increased the usage of international performance data.

The coalition released the white paper The importance of teaching in 2010, which emphasize comparative performance on international tests as a justification for reform. The white paper states, "What really matters is how we are doing compared with our international competitors. [...] The truth is, at the moment we are standing still while others race past" (DfE, 2010, p. 1). England's performance on PISA 2006 was utilized to frame the policy positions set out in the document, with the "Far East" (and Scandinavia) identified as having top performing systems from which England must learn. A research interviewee noted, "The focus on international evidence sharpened hugely" (interview, policy maker, 2013), while under the Coalition, "That has all changed. Ministers are absolutely clear that every policy that is developed, they want to see underlying evidence not just from the national side, also the international level" (interview, policy maker, 2013). However, this shift since 2006 has not only occurred in terms of the political use of PISA, but has also been reflected in the systemic and policy usage of PISA data. This new emphasis on PISA results was very evident in speeches made by the secretary of Education at the time, Michael Gove.

There were other reference societies in addition to China-Shanghai used by Gove. For example, policy learning taken for the United States was manifested in free schools along the lines of the Swedish model and the creation of more academies, which like Charter Schools, are publicly funded, but privately managed. In the education policy work of the Coalition and subsequent Conservative governments, one witnesses the political and legitimatized usages of PISA performance and the political rhetoric surrounding Shanghai's PISA performance. The reforms implemented by these governments actually entrenched further a neo-liberal educational reform agenda, rather than policies based on practices in Shanghai. Interesting fact here is that the policy learning in respect of free schools and academies came from the ideological alignment with Swedish and American reforms, while Shanghai was used largely as externalizing legitimizations.

Australia's response

In early PISA, Australia performed well in reading, mathematics, and science. A new narrative of declining performance, however, was set in process following the public release of PISA 2009. The narrative of decline through this PISA shock was catalyzed by the reality that the top performing nations on both PISA 2010 and 2012, apart from Finland, have been in Asia. Australia is heavily dependent economically on the strength of the Chinese economy, and regards its economic future as dependent on Asia. Interestingly though, the PISA shock was precipitated more by think tank and consultancy firm rehashes of the PISA results, released in 2012 and media coverage of them, than the PISA results themselves. Just as the OECD seeks to manage the media messages regarding PISA, media representations of Australia's PISA performance since 2010 have contributed to the now pervasive discourse of decline in Australian schooling performance.

A lot of media coverage of two reports, based inter alia on PISA results, produced by consultancy firm, the Nous Group, and by think-tank, the Grattan Institute, helped shape current Australian education policy. Both highlighted Australia's declining PISA performance, directing attention to Asian schooling systems. The Nous Group's Schooling challenges and opportunities: a report for the review of funding for schooling panel (2011) was framed in terms of Australia's comparative performance globally. The report opens with an analysis of PISA performance, focusing on Australia's declining reading, and mathematics performance. The risk of Australia "falling behind" Asian systems such as Shanghai, Korea, Hong Kong and Japan was stressed (Nous Group, 2011, p. 7). The then federal Labor government's 2012 White Paper on Australia in the Asian century used this argument to press for school reform in response to Australia's declining comparative PISA performance.

Catching up: learning from the best school systems in East Asia report (Jensen et al., 2012), a policy report by the Grattan Institute, also emphasizes the rise of Asian systems: "Today's centre of high performance in school education is East Asia" (Jensen et al., 2012, p. 2). The influence of spending and cultural factors - "Confucianism, rote learning or Tiger Mothers" (Jensen et al., 2012, p. 2) - on this test performance is significantly downplayed. Instead, education reform agendas and policies in countries such as Shanghai, Hong Kong and Singapore are stressed. The report was derived from a 2011 roundtable, titled Learning from the best . This roundtable, convened by Grattan, included amongst its participants the Australian prime minister and federal Education minister, education academics from Shanghai, Singapore, Hong Kong and Korea, and interestingly, Andreas Schleicher from the OECD, now director of the OECD's Education and Skills Directorate. The author of the report also wrote an open piece for the national newspaper, The Australian (SHANGAI SUCCESS A..., 2012) entitled, "Shanghai success a lesson in delivery".

This activity and media coverage around PISA 2009 and the "rise" of Asian schooling systems precipitated the PISA shock in Australia. Indeed, Julia Gillard, then prime minister, entered the debate. She argued that Australia needed to "win the education race" in Asia (WE RISK LOSING...,2012), and continued, "Four of the top five performing school systems in the world are in our region and they are getting better and better" (WE RISK LOSING...,2012). She did not want Australian students to become "workers in an economy where we are kind of the runt of the litter in our region and we've slipped behind the standards and the high-skill, high wage jobs are elsewhere in our region" (WE RISK LOSING...,2012).

Australia's perceived economic future and specific geopolitical location were important factors in Australia's PISA shock. Shanghai's performance was used as an externalizing legitimization for subsequent school reform, which has been framed by political party ideology in Australia, rather than by policy learning from Shanghai or elsewhere in Asia. The Conservative governments in Australia from 2013 have continued to use Australia's comparative PISA performance more generally as an externalizing legitimating device.

There are two features that should be noted about Australia's changing PISA performance: while 40% of students in the five top performing systems are in the top category of performance on mathematics and literacy, only 15% of Australian students are, and this percentage has declined over time. Second, there have been strengthening of socio-economic correlations with performance on PISA and also a decline in the number of "resilient students," that is, those from the bottom quartile of socio-economic background, who perform in the top two categories on PISA measures. The policy legitimization use of PISA, particularly by Conservative governments in Australia from 2013, only ever mentions the decline in quality and totally ignores the declining equity performance.

Conclusion

The multiple rationales for participation in PISA have been documented, including the OECD's and those of participating nations. The OECD's attempt to pass on the message that national media and policy makers take up has also been outlined. The paper has also been shown, through three cases, how national systems of schooling often use comparative performance on PISA and changes over time as external justifications for reforms of their systems, framed largely by internal political imperatives and ideologies, rather than so much by policy learning from top performing national systems. The United States', England's, and Australia's responses to Shanghai's outstanding performance on PISA 2009 well illustrate this. It was also shown how important the media is in constituting national representations of PISA performance and also in framing national policy responses. It is interesting to think about the relationships between nations' various rationales for participation in PISA, given that participation is expensive, and the ways in which PISA data are used or not for policy-making within nations.

Referências

• ADDEY, C.; SELLAR, S. Rationales for participation in large-scale international assessments. In: VERGER, A.; NOVELLIN, M.; ALTINYELKEN, H. (Eds.). Global education policy and international development: new agendas, issues and policies.2ed. London: Bloomsbury, 2016.
• BREAKSPEAR, S. The policy impact of PISA: an exploration of the normative effects of international benchmarking in school system performance. OECD Education Working Papers , Paris, n. 71, 2012.
• CARROLL, P.; KELLOW, A. The OECD: a study of organizational adaptation. Cheltenham: Edward Elgar, 2011.
• CARVALHO, L.M.; COSTA, E. Seeing education with one's own eyes and through the PISA lenses: considerations of the reception of PISA in European countries. Discourse: Studies in the Cultural Politics of Education , v. 6, n. 5, p. 1-9, 2014. http://dx.doi.org/10.1080/01596306.2013.871449
  » http://dx.doi.org/10.1080/01596306.2013.871449
• CHAN, P.; SEDDON, T. Governing education in China: PISA, comparison and educational regions. In: FENWICK, T.; MANGEZ, E.; OZGA, J. (Eds.). Governing knowledge: comparison, knowledge-based technologies and expertise in the regulation of education. 2014 World Yearbook of Education London: Routledge, 2014. p. 200-217.
• CONDRON, D. Egalitarianism and educational excellence: compatible goals for affluent societies? Educational Researcher , v. 40, n. 2, p. 47-55, 2011. DOI: 10.3102/0013189X11401021
  » https://doi.org/10.3102/0013189X11401021
• DEPARTMENT FOR EDUCATION - DfE. The importance of teaching: the schools white paper 2010. Norwich: HMSO, 2010.
• GREK, S. Governing by numbers: the PISA 'effect' in Europe. Journal of Education Policy , v. 24, n. 1, p. 23-37, 2009.
• HENRY, M.; LINGARD, B.; RIZVI, F.; TAYLOR, S. The OECD, globalization and education policy . Amsterdam: Pergamon Press, 2001.
• JENSEN, B.; HUNTER, A.; SONNEMANN, J.; BURNS, T. Catching up: learning from the best school systems in East Asia. Australia: Grattan Institute, 2012.
• LEWIS, S.; SELLAR, S.; LINGARD, B. PISA for schools: topological rationality and new spaces of the OECD's global educational governance. Comparative Education Review , v. 60, n. 1, p. 27-57, 2016. DOI: 0010-4086/2016/6001-0004.
  » https://doi.org/0010-4086/2016/6001-0004.
• LINGARD, B. The impact of research on education policy in an era of evidence-based policy. Critical Studies in Education , v.54, n. 2, p. 113-131, 2013. DOI: 10.1080/17508487.2013.781515.
  » https://doi.org/10.1080/17508487.2013.781515
• LINGARD, B.; RAWOLLE, S. New scalar politics: implications for education policy. Comparative Education , v.47, n. 4, p. 489-502, 2011. DOI: 10.1080/03050068.2011.555941.
  » https://doi.org/10.1080/03050068.2011.555941
• LINGARD, B.; SELLAR, S. The changing organizational and global significance of the OECD's education work. In: MUNDY, K.; GREEN, A.; LINGARD, B.; VERGER, A. (Eds.). Handbook of global education policy . Oxford: Blackwells, 2016.
• MARTENS, K.; NIEMANN, D. When do numbers count? The differentiated impact of the PISA rating and ranking on education policy in Germany and the US. German Politics , v. 22, n. 3, p. 314-332, 2013. DOI: 10.1080/09644008.2013.794455.
  » https://doi.org/10.1080/09644008.2013.794455
• MEYER, H-D.; SCHILLER, K. Gauging the role of non-educational effects in large-scale assessments: socio-economics, culture and PISA outcomes. In: MEYER, H.-D.; BENAVOT, A. (Eds.). PISA, power, and policy: the emergence of global educational governance. Oxford: Symposium Books, 2013. p. 207-224.
• NOUS GROUP. Schooling challenges and opportunities: a report for the review of funding for schooling panel. Melbourne: Nous Group, 2011.
• ORGANIZATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT - OECD. Making reform happen: lessons from OECD countries. Paris: OECD Publishing, 2010.
• PAPADOPOULOS, G. Education 1960-1990: the OECD perspective. Paris: OECD, 1994.
• POWER, M. The audit society: rituals of verification. Oxford: Oxford University Press, 1997.
• SAHLBERG, P. Finnish lessons: what can the world learn from educational change in Finland?New York: Teachers College Press, 2011.
• SCHRIEWER, J. The methodology of comparison and the need for externalization: methodological criteria and sociological concepts. In: SCHRIEWER, J.; HOLMES, B. (Eds.). Theories and methods in comparative education . New York: Peter Lang, 1990. p. 25-83.
• SELLAR, S.; LINGARD, B. Looking East: Shanghai, PISA 2009 and the reconstitution of reference societies in the global education policy field. Comparative Education , v. 49, p. 464-485, 2013. DOI: 10.1080/03050068.2013.770943.
  » https://doi.org/10.1080/03050068.2013.770943
• SELLAR, S.; LINGARD, B. The OECD and the expansion of PISA: new global modes of governance in education. British Educational Research Journal , v. 40, n. 6, p. 917-936, 2014. DOI: 10.1002/berj.3120.
  » https://doi.org/10.1002/berj.3120
• SHANGAI success a lesson in delivery. The Australian , Sidney, 18 Feb. 2012.
• STEINER-KHAMSI, G.; WALDOW, F. (Eds.). World yearbook of education: policy borrowing and lending in education. New York: Routledge, 2012.
• TAKAYAMA, K. The politics of international league tables: PISA in Japan's crisis debate. Comparative Education , v. 44, n. 4, p. 387-407, 2008.
• TOP test scores from Shanghai stun educators. The New York Times , New York, p. A1, 7 Dec. 2010.
• TUCKER, M. (Ed.). Surpassing Shanghai: an agenda for American education built on the world's leading systems. Cambridge, MA: Harvard Education Press, 2011.
• WALDOW, F. Standardisation and legitimacy. In: STEINER-KHAMSI, G.; WALDOW, F. (Eds.). World yearbook of education: policy borrowing and lending in education. New York: Routledge , 2012. p. 411-427.
• WALDOW, F.; TAKAYAMA, K.; SUNG, Y.-K. Rethinking the pattern of external policy referencing: media discourses over the "Asian Tigers" PISA success in Australia, Germany and South Korea. Comparative Education , v. 50, p. 302-321, 2014. DOI: 10.1080/03050068.2013.860704.
  » https://doi.org/10.1080/03050068.2013.860704
• WE risk losing education race, Julia Gillard warns. The Australian , Sidney, 24 Jan. 2012.
• WISEMAN, A. The uses of evidence for educational policy-making: global context and international trends. Review of Research in Education , v. 34, n. 1, p. 1-124, 2010. DOI: 10.3102/0091732X09350472.
  » https://doi.org/10.3102/0091732X09350472.
• WISEMAN, A. W. Policy responses to PISA in comparative perspective. In: Meyer, H.; Benavot, A. (Eds.). PISA, power, and policy: the emergence of global educational governance. Oxford: Symposium Books , 2013.

Publication Dates

• Publication in this collection
  Jul-Sep 2016

History

• Received
  15 July 2016
• Accepted
  08 Aug 2016