PISA FOR DEVELOPMENT AND THE SACRIFICE OF POLICY-RELEVANT DATA^*

ABSTRACT:

This paper examines how PISA is being extended to include lower and middle income countries, raising questions about its significance in such contexts and its claim to produce more policy-relevant data. The paper tells the history of PISA for Development (PISA-D), before discussing how PISA-D is negotiated as a policy tool. Drawing on Haas' (1992) epistemic communities, and on socio-material semiotics (LAW, 2008), this paper discusses qualitative data consisting of document analysis and interviews on PISA-D carried out at the OECD, The Learning Bar (a private contractor), and with high level policy actors in Ecuador and Paraguay (two PISA-D countries). By unpacking the negotiation process upon which PISA-D's policy-relevance threshold was established, the paper unravels the multiple interests invested in PISA-D. These interests, and PISA-D's policy-relevant data being more PISA-relevant, challenge PISA-D's policy-relevance claim and argue its relevance threshold is driven by a commitment to the PISA epistemic community (BLOEM, 2015). The paper concludes questioning the meaning of "policy-relevance" in the PISA era. In the days of epistemological governance (SELLAR; LINGARD, 2013), the global education community appears to have moved away from what knowledges are relevant for policy to whose knowledge counts for policy.

Keywords:
PISA for development; OECD; Policy-relevance; Paraguay; Ecuador

RESUMO:

Este artigo examina como o PISA vem sendo alargado de modo a incluir países de 'renda baixa e média', e levanta questões sobre a importância do teste em tais contextos e sobre a sua pretensão de produzir dados politicamente mais relevantes. O artigo começa por apresentar a história do PISA para o Desenvolvimento (PISA-D) e, depois, discute como este programa é negociado enquanto instrumento de política. Com base na noção de 'comunidade epistémica' (Haas, 1992), bem como na semiótica sócio-material (Law, 2008), este artigo analisa entrevistas sobre o PISA-D realizadas com atores da OCDE, da The Learning Bar (uma empresa privada) e, ainda, com atores políticos de alto nível do Equador e do Paraguai (dois países PISA-D). Ao debruçar-se sobre o processo de negociação em função do qual se estabeleceu o 'limiar de relevância política' do PISA-D, o artigo revela os múltiplos interesses envolvidos. Ao fazê-lo, o artigo põe em questão a reivindicação da relevância política do PISA-D e argumenta que a aceitação do 'limiar de relevância política' do PISA-D é determinada pelo compromisso com a 'comunidade epistêmica do PISA' (Bloem 2015). O artigo conclui questionando o que significa 'relevância política' no mundo PISA. Em tempos de 'governança epistemológica' (Sellar e Lingard 2013), a comunidade mundial da educação global parece ter deslocado a sua atenção dos conhecimentos são relevantes para a política para os conhecimento que contam para a política.

Palavras-chave:
PISA para o desenvolvimento; OCDE; Relevância política; Paraguai; Equador

RÉSUMÉ:

Cet article examine comment PISA est élargi pour inclure les pays à revenu faible et moyen, soulevant des questions sur sa signification dans tels contextes et sa prétention à produire plus de données pertinentes destinées aux politiques. Le texte raconte l'histoire du 'PISA pour le Développement' (PISA-D) et discute comment il est négocié comme outil politique. Appuyée sur le concept de communautés épistémiques (Haas, 1992) et aussi sur la théorie de l'acteur-réseau (Law, 2008), l'article analyse des entrevues sur PISA-D menées à l'OCDE, à The Learning Bar (un entrepreneur privé), et avec les acteurs politiques de haut niveau en Equateur et le Paraguay (deux pays PISA-D). En exposer le processus de négociation sur lequel le seuil de pertinence politique de PISA-D a été établi, le texte identifie les multiples intérêts investis dans PISA-D. Ce faisant, le texte mette en question la pertinence politique demandé par PISA-D. Dans les jours de la gouvernance épistémologique (Sellar et Lingard 2013), la communauté mondiale de l'éducation semble avoir être déplacée l'attention des connaissances qui sont pertinentes pour la politique à ceux qui comptent pour la politique.

Mots-clés:
PISA pour le développement; OCDE; Pertinence politique; Paraguay; Equateur

Introdution

The Organisation for Economic Co-operation and Development (OECD) Programme for International Student Assessment (PISA) is being extended to include lower and middle income countries, raising profound questions about the significance of PISA in such contexts and its claim to produce more policy-relevant data¹. This paper draws on empirical data to tell the history of PISA for Development (PISA-D) - an extended version of PISA - and highlights the contradictory forces at work. On the one hand, there is reason to measure on the single universal PISA metric and compare internationally, driving countries toward compliance as regards standardization and centralization of data. On the other hand, there is commitment and rationale for making data more relevant to lower and middle income policy contexts. This paper investigates how this tension is negotiated by high-level PISA-D actors at the OECD, a PISA-D contractor², and by high-level policy actors in Ecuador and Paraguay.

The OECD claims that PISA-D will provide countries with PISA data which is more policy relevant in lower and middle income contexts, and is implementing policy activities to ensure the PISA-D data (expected at the end of 2018) will impact on policy. It is for this reason that this analysis focuses on PISA-D as a policy tool.

After a note on the theoretical tools applied and methodology, the paper briefly reconstructs the history of PISA-D beyond the official documents made available on the OECD website, and outlines the activities implemented to make PISA-D impact on policy. The paper then explores how PISA's instruments were enhanced and how the limits of this enhancement were established by the OECD, by one of its private contractors (The Learning Bar) and by the PISA-D countries.

By unpacking the negotiation process upon which the PISA-D policy-relevance threshold was established, this paper discloses the OECD's rationales of PISA-D actors' participation - including sharing policy knowledge beyond membership and geopolitical expansion, business opportunities, and political ties with the global PISA community. Consequently the paper challenges PISA-D's policy-relevance claim and argues that the relevance threshold is driven by a commitment to the PISA epistemic community (BLOEM, 2015) rather than policy information needs. Finally, this paper concludes questioning the meaning of "policy-relevance" in the PISA era. How has the global education community come to value PISA-relevance and the PISA epistemic community values and policy enterprise over context-relevant policy knowledge? It appears that in the days of epistemological governance, we have moved away from what knowledges are relevant for policy to which knowledge counts for policy.

A theoretical toolbox: epistemic communities and socio-material semiotics

Theoretically, this paper draws on Haas's (1992) analytical approach to epistemic communities and on notions from socio-material semiotics (LAW, 2008; LATOUR, 2005) to theorize how the OECD and its PISA-D partners negotiate consensus on PISA-D as a policy-relevant instrument. I relate this understanding to the work by Bloem (2015) on the PISA epistemic community, and by Sellar and Lingard (2013) on the OECD and PISA as part of epistemological governance.

Haas' understanding of epistemic communities is that of a knowledge-based network of recognized experts in particular domains (though they can have different backgrounds) who makes "authoritative claims to policy-relevant knowledge" (HAAS, 1992, p. 3). The prestige, professional training, and expertise reputation of the members legitimizes an epistemic community's activities and claims to knowledge (HAAS, 1992, p. 17). In Haas' words, an epistemic community has:

(1) a shared set of normative and principled beliefs, which provide a value-based rationale for the social action of community members; (2) shared causal beliefs, which are derived from their analysis of practices leading or contributing to a central set of problems in their domain and which then serve as the basis for elucidating the multiple linkages between possible policy actions and desired outcomes; (3) shared notions of validity - that is, intersubjective, internally defined criteria for weighing and validating knowledge in the domain of their expertise; and (4) a common policy enterprise - that is, a set of common practices associated with a set of problems to which their professional competence is directed, presumably out of the conviction that human welfare will be enhanced as a consequence (HAAS, 1992, p. 3).

Haas (1992) also argues that the dynamics of uncertainty, interpretation, and institutionalization are crucial to understanding epistemic communities. Uncertainty (MURPHY, 2010, on policy uncertainty of PISA data users) drives policy knowledge needs, with epistemic communities emerging to provide such information based on valued scientific and technical expertise.

HAAS' (1992) theoretical approach resonates with the way PISA-D policy relevance is negotiated and established, and helps understand how the relevance threshold is more PISA relevant than policy relevant. Ensuring PISA-D data are comparable with PISA data, rather than creating a different metric for the PISA-D contexts, allows PISA-D countries to access and benefit from the prestige PISA has obtained and the legitimacy it allocates.

To further nuance the use of Haas' (1992) epistemic communities, this paper draws on notions from socio-material semiotics (LAW, 2008; LATOUR, 2005): assemblages, black boxes and piggy-backing. An assemblage is an ensemble of human and non-human actors which come together temporarily to further the assemblage's aims (i.e. knowledge production). The actors' agency is given by their relationships, making power in actu and distributed amongst actors. The black box concept suggests that in order for actors to join an assemblage, previously unstable truths need to be stabilized and accepted by all joining actors - these sealed black boxes become unquestioned, shared foundations (LATOUR, 1987). The concept of black boxes is used to understand which assumptions have become temporarily accepted truths, beyond contestation in PISA-D (including the scientific, technical approach to education, and the ideological and political dimension embedded in PISA). The concealed truths of black boxes nuance Haas' understanding of shared beliefs and notions of validity by suggesting that these truths are temporary and go unchallenged in order for actors to join and maintain the assemblage. The third notion of socio-material semiotics relevant in this paper is piggy-backing, used by Latour (1987) to theorize how an alliance can serve multiple, even contradictory interests of all actors involved. Latour argues that by pushing the interests of an assemblage, an actor profits from it whilst furthering his own interests (LATOUR, 1987, p. 108). In the context of the PISA epistemic community, the piggy-backing notion may help understand what drives the participation of actors with diverse rationales.

By combining Haas' (1992) epistemic communities with socio-material semiotics, all the actors involved acquire agency by being part of the assemblage and power is understood as distributed. This theoretical approach is conducive to understanding how the negotiation of PISA-D's policy-relevance threshold is a shared process, rather than imposed by the OECD. As we shall see, all actors decide to black box temporary truths and adapt to the PISA-D assemblage - not because one of the actors (the most obvious one being the OECD) has decided how relevant PISA-D data should be, but because all actors have an interest in establishing a threshold of relevance which allows PISA-D data to compare with the main PISA.

Participating in the PISA-D assemblage assumes there is agreement amongst involved actors regarding a shared set of normative, principled and causal beliefs, notions of validity, and the policy enterprise. Among these we can list that:

• there is a shared problem concerning the need to reform policies which will produce globally competitive knowledge societies and make educational systems better respond to globalized market pressures;

• there is a need for evidence in the form of comparative data to inform policy (ROSE, 1999; LINGARD, 2010);

• quantification and international comparisons of educational performance are a valid practice;

• validity can be obtained scientifically and it is not socially constructed; PISA's decontextualized global skills metric, data interpretations and policy recommendations provide scientific, objective knowledge for policy;

• the OECD's global educational agenda of quality and equity is the most valued policy enterprise;

• the OECD's skills and competencies agenda (GREK, 2014) is a shared policy enterprise;

• an economistic approach to education is a valuable policy approach;

• measuring 15-year-old skills provides a measure of future national and global competitiveness (SELLAR; LINGARD, 2013); and

• PISA's comparative data allows the identification of best policies and practices.

Although driven by different rationales for participation and development, the OECD, the private contractors, and the PISA-D countries, have agreed to the shared values of PISA-D, and by valuing PISA relevance over policy relevance, to the values of the PISA epistemic community. The paper argues that the PISA epistemic community's shared policy enterprise and beliefs have played a key role in the side-lining of PISA-D's policy relevance.

Sellar and Lingard (2013, p. 16) argue the OECD and its PISA have created a mode of global epistemic governance by constituting the education policy field and constructing "the globe as a commensurate space of measurement." Epistemological governance is to be understood as a combination of Woodward's (2009) normative and cognitive modes of governance: the former functions through agreed sets of values (i.e., the need for comparative data-based knowledge for policy), and the latter functions through cooperatively moving a transnational agenda forward (i.e., increasing education quality and equity by searching for best practices through comparative performance data). Sellar and Lingard (2013) argue that epistemological governance is strengthened by the expansion of PISA in scope and scale, but also through the OECD's creation of a PISA epistemic community. Carrying out ethnographic research at the OECD, Bloem (2015) captures the OECD's shift from describing the data produced in the first PISA implementations, to the OECD's transformation into an independent knowledge producer. Keen to remain politically relevant but also building on the legitimacy and prestige acquired by the PISA instruments, data and its data sets with historical value, the OECD stopped outsourcing the PISA data analysis to scientists and took over the data analysis as of 2006. As argued by Bloem (2015), transforming quality into quantity through commensuration (ESPELAND; STEVENS, 2008) and then back into quality requires decisions and interpretations (based on assumptions, values, and agendas) to be taken. Taking control over the PISA data interpretation allowed the OECD to direct the interpretation process and increase the political relevance of the analysis. Also drawing on Haas (1992), Bloem (2015) argues that this shift allowed the OECD to position itself at the center of the epistemic community around PISA.

At the time of PISA-D development and implementation, PISA is no longer only an OECD international assessment which provides countries with comparative data, but a prestigious global epistemic community - with the OECD as the center of it - providing data and an interpretation of the data in the form of policy solutions (i.e., the PISA-D report includes a chapter on "Policy options in Zedland"). As we shall see in this paper, all actors involved in PISA-D are driven by the prestige of PISA and its epistemic community, though how they negotiate the PISA-D process suggests the agreed values and transnational agenda are owned and pursued differently.

A note on methodology

This paper draws on data gathered within the research project PISA for Development for Policy (P4D4Policy), which looks into how PISA-D is changing education policy processes in Ecuador and Paraguay, but also the main global policy actors involved. The P4D4Policy research project is carried out with a qualitative research design. The data discussed in this paper draw on four years of PISA-D working documents³, the OECD's "PISA-D Call for Tender", participant observations of a PISA-D International Advisory Group meeting which took place in Paraguay in March 2016 but also on discussions at PISA-D social events, and semi-structured interviews at the OECD, at The Learning Bar⁴, and with high-level policy actors in Ecuador and Paraguay in late 2015 and early 2016. It is worth noting that the data were gathered whilst the program was being developed and implemented.

The semi-structured interviews carried out in Canada, Paris, Ecuador, and Paraguay are the result of a long process of negotiating access and building trust, which took place over four years. The data presented here are the result of high-level access and trust relationships which meant I interviewed the highest levels involved in PISA-D (including Andreas Schleicher, Director of Education and Skills at the OECD, who asked not to be anonymized; key decision makers; and minister level actors). High levels of trust also explain my invitation to meetings which are routinely carried out behind closed doors and to which independent researchers are not invited. This has important implications for research ethics applied, and have led me to exclude compromising data. Interview extracts which were deemed too specific (i.e., relating to personal roles) or compromising were altered to protect anonymity. Although interviewees shared extensive information on the process being investigated in this paper, the data selected to be discussed in this paper highlights information which stands out as significant to the research questions discussed in this paper. Interviews extracts are presented in English, although the interviews were carried out in Spanish in Paraguay and Ecuador.

A history of Programme for International Student Assessment for Development

After five PISA implementations since 2000, the OECD publicly recognized that PISA had poor policy value for the lower and middle income countries that had diagnosed their educational outcomes with PISA. Although the OECD claims PISA provides countries with input for legislation and policy and that it motivates educational reforms (OECD, 2014a, p. 16), in 2013, the OECD published a working paper stating the "lack of institutional capacity and less relevant results due to a non-representative sample of 15 year-olds and clustering of students at low proficiency levels are (the) main challenges for the effective use of PISA" in lower and middle income contexts (BLOEM, 2013, p. 4).

The poor policy-relevance acknowledgment is the result of a process initiated by the participation of India in PISA. Following the British Department for International Development's (DFID) involvement in India's national education plan, DFID concluded that the Indian national assessment program was leading to a poor quality report. DFID, thus, contracted Education Testing Service (ETS) to carry out a technical capacity package to build international large-scale assessment capacity in India and decided that participation in PISA would allow India to leap forward. It was at this stage that DFID and the OECD discussed how India could join and put together a PISA support package for India's 2009 PISA participation. India implemented PISA in two high-performing states: Tamil Nadu and Himachal Pradesh. However, they performed very badly, "only above Kyrgyzstan" (Interview, OECD2015#30) and the Ministry of Development defended the results suggesting the tests were inappropriate to the Indian context, contesting the validity criteria shared by the PISA epistemic community (which had been previously been accepted). It was observing the Indian PISA experience, that high-level staff at the OECD got together and decided "Ok, let's do something" (Interview, OECD2015#30).

The Indian PISA experience informed the kind of assistance the OECD saw as necessary for lower and middle income country participation, but also the need for the PISA data to be meaningful. Although the OECD was keen to enhance PISA in the most challenging contexts for PISA (lower income contexts), it also determined PISA-D to be a success and decided it could not take risks. The Organization established three criteria to start the project: countries had to be low income, have already implemented an international assessment, and there had to be an aid partner willing to fund each country's participation. The aid partners were approached with the list of potential countries and asked who they would fund.

Having identified countries and funding partners⁵, the OECD's Directorate for Education and Skills and its Directorate for Development Cooperation started developing partnerships and collaborations with agencies and international organizations⁶ with educational expertise in lower and middle income contexts, and initiated the development of PISA-D. In March 2013, the OECD sought to confirm its partners' support and started a dialog with the selected countries. In 2015, Ecuador, Guatemala, Paraguay, Honduras, Zambia, Senegal, and Cambodia officially signed up to implement PISA-D⁷. Ecuador described being the first country to sign the PISA-D contract as crucial in opening the way to other countries' formal commitment. In 2016, Panama, which had already participated in the main PISA, joined the out-of-school (strand C) part of PISA-D to assess the skills of their out-of-school 15-year olds.

What is Programme for International Student Assessment for Development?

he first technical working meeting on PISA for Development in 2013⁸, held with the PISA-D partners, country candidates, experts, and international advisory board members, included a welcoming introductory presentation by Andreas Schleicher, in which he stated that PISA-D would be "as comparable as possible to facilitate peer-learning" and "as country-specific⁹ as necessary to be meaningful and interpretable in national contexts" (Initial Technical Meeting power points: slide 9). The OECD also argued that lower and middle income countries need to measure educational outcomes in an internationally comparable manner, as measuring educational progress within an internationally agreed framework provides a basis for international collaboration on designing and implementing educational policies. Resonating with Haas' (1992) epistemic communities, the OECD states¹⁰ that PISA helps governments with shared policy interests set policy targets in terms of measurable goals achieved elsewhere and facilitates peer-learning on policy and practice; and that by participating, countries "join international community focused on improving learning outcomes based on benchmarking from PISA results" (Second Technical Workshop power point). The OECD also clarified that PISA-D countries are not joining a second class and that publication of the PISA-D data remains at the discretion of participating countries.

As with PISA, the OECD outsourced PISA-D to private contractors through a competitive tendering process. The OECD divided PISA-D into three strands:

• strand A: the cognitive development of the testing instruments to highlight the distribution of skills below baseline proficiency levels in PISA;

• strand B: the development of PISA-comparable but lower and middle income context-relevant contextual questionnaires; and

• strand C: the development of the PISA-D test and background questionnaires for out-of-school 15-year olds.

In 2015, a consortium of Global North-based contractors was awarded the strands A, B, and C contracts but also the management of the entire program. Strand A was contracted to ETS in collaboration with cApStAn, Westat, and Pearson; strand B was contracted to The Learning Bar; and strand C was contracted to all the above-mentioned private companies working in partnership.

Developed as a "powerful tool for policy guiding and making" (OECD PISA-D slides), PISA-D's policy-relevance is to be obtained by redeveloping the PISA instruments to respond to the policy priorities of the first seven participating countries. However, the PISA-D data also have to be comparable with the main PISA data. Making PISA into an extension of PISA and not something different implies the PISA-D countries are required to share the values and policy agenda of the PISA epistemic community. Driven by the need for PISA-D to be as comparable as possible with PISA, it is helpful to clarify in what way the OECD describes PISA and PISA-D as different in six points:

• the design targets 15-year olds at the lower end (in the main PISA 20% of test items are level 2 and below, the proportion of these items in PISA-D will be substantially higher in PISA-D, potentially 60%);

• the test items are enhanced to be more relevant to lower and middle income contexts - using items from PISA, PISA for Schools, PIAAC¹¹, STEP¹², and LAMP¹³;

• the background questionnaires include questions relevant to lower and middle income contexts and policy priorities;

• the tests and questionnaires are tailored to reduce the reading burden;

• PISA-D includes an assessment for out-of-school 15-year olds; and

• capacity building and policy learning activities are provided.

The PISA-D capacity building and policy learning include policy knowledge exchange amongst PISA-D participating countries, policy knowledge exchange amongst main PISA and PISA-D countries, policy analysis capacity building, collaborative policy-oriented report writing, and OECD policy support on the ground during data dissemination. The OECD argues that the policy learning dimension of PISA is the overarching aim of PISA. Drawing on an interview with Andreas Schleicher, the PISA tests can be understood as an Esperanto of the global education community which is conducive to policy learning:

The global purpose is simply to allow those countries to engage with global best expertise. It isn't really about the scores; the scores would be a product of this. But the idea really is to connect the expertise in those countries with the best expertise which is around anywhere in the world. There was no global dialogue, no willingness to engage with other people's ideas, other peoples' missions. (...) We need to find a common language to learn from each other. The testing was just the instrument. (...) What I am most proud of is that now, when ministers meet or educators meet, or when scientists meet, they may not agree but they listen to each other. And they learn from each other. (Interview, Andreas Schleicher in 2015)

What is not said about the global education community' Esperanto, is that a constructed language is a political project - which in the case of PISA could be described as spreading an economistic approach to education - that is channeled through the conceptual and methodological frameworks of the tests. The capacity building and the policy learning activities planned as part of PISA-D illustrate how a process has been put in place to ensure the values and policy enterprise underpinning PISA are adopted and maintained by the actors joining PISA-D assemblage. This process is ongoing at all stages of PISA-D, but it is strengthened when the data are made available for analysis and publication as countries are more likely to leave the assemblage and question the previously agreed notions of validity, the policy enterprise, and shared beliefs (i.e. the case of India's participation in PISA 2009).

How does the Programme for International Student Assessment for Development make its data policy-relevant claim?

In 2013, the OECD commissioned Willms and Tramonte (2014) to investigate how the background questionnaire of PISA-D could better respond to the newly reached contexts given the challenge of maintaining comparability between the main PISA and PISA-D data but also developing the instruments so that the data would be sufficiently policy-meaningful and interpretable in national contexts. In response Willms and Tramonte (2014) suggested the contextual assessment tools include:

• items on students' early learning experience;

• items on students' familiarity with the language of the test;

• measures of parental involvement, social capital, and cultural capital;

• measures of the role of other community members and of types of community;

• measures of school attendance; and

• items on participation in formal and informal labor market.

Willms and Tramonte (2014) argued that Caroll's (1963) learning model upon which the main PISA questionnaire is based, is limited, given the scope of PISA aims. PISA does not aim to understand how students can apply skills as a result of the formal schooling year they are tested in, but as a result of cumulative formal, informal, and non-formal learning which occurs from birth (and before). Willms and Tramonte (2014) paper therefore strongly recommended that PISA-D, but also the main PISA background questionnaires, be rewritten. In April 2014, it was agreed that the paper presented by Willms and Tramonte (2014) would represent the technical basis to enhance the PISA-D contextual questionnaires - and thus The Call for Tender terms of reference for strand B.

By contract, The Learning Bar (TLB), which was contracted to carry out strand B, has to develop the background questionnaire on the basis of the policy priorities identified with the participating countries, further to assisting the OECD in implementing the PISA-D policy learning activities¹⁴. TLB was also requested by the OECD to form and administer a questionnaire expert group (QEG), which advises TLB on its work, but also has a legitimizing role: the experts are all highly respected experts whose advisory role and engagement transfers expertise and authority to TLB's PISA-D work. As theorized by Haas (1992), an epistemic community relies on the prestige and professional training of the experts it enrols to legitimize its authoritative claims to knowledge. In June 2015, TLB visited each of the PISA-D countries "to identify policy questions unanswered by PISA-D contextual questionnaire data" so that PISA-D countries could "work with The Learning Bar to identify questions (or measures) to include in the PISA-D contextual questionnaires" (International Advisory Group meeting, May 2015: slide 61) before meeting with the QEG in August 2015, to discuss and agree on the country policy suggestions. As we shall see in the next part of this paper, by entering the PISA epistemic community, the values and policy agenda driving these negotiations had already been settled before the consultations.

Programme for International Student Assessment for Development and its policy-relevant data claim: relevant to whom?

This section discusses how "policy-relevance" is understood and negotiated by the PISA-D actors interviewed. Understanding how PISA-D policy-relevance is negotiated, goes hand in hand with the rationales which are driving the OECD, TLB, Ecuador, and Paraguay's involvement in PISA-D.

The Organisation for Economic Co-operation and Development

By shifting from describing data to interpreting data for policy (BLOEM, 2015), the OECD also became aware that without innovations, PISA and its PISA epistemic community would soon lose traction. It was decided that PISA would have to innovate by broadening its reach, but in order to supply more countries with PISA information, the OECD had to understand what countries needed:

That's kind of like a 'supply' part of the story - we want to make this available to lower income countries. Then you have more like a demand side of the story, which is, the OECD as a whole, basically trying to position itself globally. How can the OECD continue to be relevant? Well it has to provide things that countries beyond the OECD countries want. (Interview, OECD2015#30)

Acknowledging PISA's data-irrelevance for policy in lower and middle income countries, the Organization started nurturing a demand for more policy-relevant PISA data¹⁵. To further increase this demand, the OECD strategically ensured a PISA-like universal learning benchmark be needed for the Sustainable Development Goals:

The MDGs were coming to an end. And so there was a whole load of discussion, processes moving ahead to look at the post-2015 framework, and so we positioned ourselves very early on as a voice arguing that the post-2015 world has to be focused on learning outcomes and quality, we cannot have another fifteen years where all talk about this, getting kids into school. And of course that immediately begged the question "How are we going to measure quality, how are we going to measure learning outcomes?", so we offered PISA as a global metric, and said 'Look, the work on PISA for Development is going to help us make this available to a wider range of countries. (Interview, OECD2015#30)

Lastly, and key to understanding the OECD's recent work, is the OECD's 50th Anniversary Council Meeting at Ministerial level in 2012, during which concern was expressed about the OECD membership being a historical relic (Interview, Andreas Schleicher in 2015). The OECD Member States agreed that the global economic gravity had moved over the previous fifty years and with it, global economic governance had shifted (OECD, 2012). The meeting was followed by the publication of an OECD Strategy for Development (OECD, 2012) in which the Ministers stated the Organization's new vision: a more inclusive policy sharing Organization, sharing its evidence-based approaches to policy with what it defines "developing countries". In the strategy, the OECD recognizes it needs to make its policy recommendations relevant to different contexts:

That strategy on development states very clearly the ambition for OECD to be relevant in a changing world and to address the needs of middle income countries, in particular emerging economies. And the key clank of that strategy is to take the best of the OECD's policy instruments and adapt them and make them relevant for the middle income countries. And of course, that's where we were with PISA for Development in 2012, we kind of shaped it and got the idea. And so the strategy on development gave us this impetus so we could go forward. (Interview, OECD2015#31)

In 2014, the OECD stated that PISA-D "supports directly the OECD Strategy on Development and the global relations strategy of the PISA Governing Board" (OECD, 2014b, p. 19). The Organization also admits satisfaction in its ability to combine its strategy for development¹⁶ with PISA as "the key policy instrument" in the global education agenda driven by the Sustainable Development Goals (SDGs):

PISA is the key policy instrument for the education and skills directorate. PISA for Development isn't a sort of discreet project, it's part of that effort to make PISA available to a wider range of countries. I think it fits very neatly into what Education and Skills are trying to do, and fits absolutely into the strategy on development. And now that strategy on development is further reinforced by the SDG framework, and certainly in education, the education SDG 4.1.1 on proficiency levels in reading and Maths at the end of primary and lower secondary, that's PISA. I think that all fits together very neatly now. (Interview, OECD2015#30)

What we understand from the extracts here above is that PISA-D is a form of global expansion in sharing education policy expertise for the OECD. Although the OECD's strategy and work program suggest that the Organization is aware it needs to follow the global economic center of gravity and that it "needs to learn from diverse growth and development experiences" (OECD, 2012, p. 4), the OECD's rationales for the development of PISA-D also imply a limited inclination to risk-taking (i.e. developing new instruments rather than another PISA version). This helps understand why the OECD has been keen to establish PISA-D with a threshold of policy-relevance which can be described as more PISA-relevant than relevant to the newly reached contexts.

The Learning Bar

Stating that the OECD's PISA claims and policy advice were not substantiated by PISA data, TLB claimed it had a model upon which PISA data could improve its relevance for PISA-D countries but also for the main PISA countries if the OECD were willing to improve its policy instruments:

The PISA lemon has been squeezed too much. (...) The whole model is flawed, and has been from the beginning. Learning scores are affected from birth. The school effect is done by subtraction. The input-process-output model is not the right way to go about it, but it underlies the whole of PISA. The model has no clear link to policy, so the whole idea is flawed. Countries set their benchmarks to become number five in the world rankings, and they then revise their curriculum. But that is not what PISA says and it will not change the scores in the next PISA round. There is such a weak link between what is taught and PISA. It is amazing PISA has got as far as it has. (Interview, PISA-Dcontractor2015#34)

TLB identified where the problem with PISA's poor policy relevance and offered the OECD and all its PISA-D partners a solution, at one condition: the OECD would have to be willing to change the background questionnaires of PISA. The OECD expressed scepticism justified by the need to maintain PISA's longitudinal data sets but also because it was keen to ensure comparability between the main PISA and PISA-D data. With PISA as it was developed in 1997 at the heart of the PISA epistemic community and what every country believes will provide it with the data and policy recommendations it needs to improve its education, the OECD was unwilling to take risks:

We did not want to take any risks, we are only going to get a chance to do this once, so we have to do it right. We were not going to give this to somebody we did not have total confidence in being able to do it. (Interview, OECD2015#30)

From a socio-material semiotics perspective, changing the PISA questionnaires would have required questioning the agreed truths upon which the PISA assemblage had settled and potentially putting the assemblage in danger of falling apart, and consequentially also the basis for the development of PISA-D.

However, TLB managed to convince PISA-D partners and countries, leading to greater influence over the OECD:

Based on previous work of mine, I was inspired to develop the Prosperity Model. The countries were pretty much happy with what we proposed, what we are giving them. Paraguay and Ecuador they love the model. They get it. We went round the table in Ecuador, there was absolute unanimous, some even said 'We love this, it's what I've always wanted. (...) We have to market the Education Prosperity Model to make it have an impact in practice. You need information campaigns. If you are going to bring around change, it's about creating a new framework of understanding of what education is about. It's about having a vision - like in marketing where you bombard the message of a different model of learning. (Interview, PISA-Dcontractor2015)

The OECD ended up agreeing to adopt TLB's Education Prosperity Model at the condition that a substantial percentage of the PISA-D background questionnaires remain the same as the main PISA. This way, from a socio-material semiotics perspective, the truths upon which the PISA rests are assembled are not called into question.

Having successfully sold the Education Prosperity Model as the best global policy solution framework, TLB's contract with the OECD required the Model be revised together with the PISA-D countries, taking the countries' contextual and policy needs into consideration:

When we went to the countries, we presented the Prosperity Model and basically showed them why it would work. It's pretty well all there. When we presented the Model, it was hard to say this does not make sense, because it does, everything is there, and if they thought that something was missing, they had lots of opportunities to tell us, and by and large they didn't because everything is there. (Interview, PISA-Dcontractor2015#36)

Although the negotiation process appears unidirectional, the next section suggests that the PISA-D countries did not negotiate the background questionnaires in the interest of maintaining PISA as it is. Having attained consensus on the Education Prosperity Model reflecting countries policy priorities, TLB went through a series of rounds of consultations to finalize the PISA-D background questionnaires. In 2016, these were agreed upon by all PISA-D actors involved. It is on the basis of this model that the OECD, TLB and the PISA-D countries that the PISA-D country reports will be written in 2018.

Obtaining a prestigious contract with the OECD, TLB became a globally recognized company working in educational assessment, boosting both its global credibility and reach. Being part of the PISA-D consortium of contractors not only allows the company to network widely with global actors investing in education, but also to broaden its business reach to countries where its expertise and products (i.e. pre-school assessment packages) have been displayed to show the company as the most competitive actor on the market. It is not surprising therefore that TLB's packages are being considered by governments in PISA-D countries. Through its PISA-D activities, the TLB has improved its reputation but also its business opportunities.

Programme for International Student Assessment for Development countries

Although keen to obtain more policy-relevant data than the main PISA would offer, PISA-D countries were presented with a dilemma. Countries were invited to discuss how the background questionnaires and the instruments could be made more relevant; however, they were warned that beyond a certain threshold, the assessment would no longer be considered PISA and the data would not be comparable with the main PISA data.

Interviewees do recount that after a lengthy process, some adaptations were made (in particular, thanks to the mediation of the OECD between countries and contractors), but that overall the program has been set up in a way that policy relevance is not about what knowledges but which knowledge, begging the question of whose policy relevance PISA-D responds to:

We would have to rethink the study. This kind of work would have required different types of relationships, with different timeframes, different forms of working, because if you think of it, throughout the whole period, the OECD does three or four meetings a year, where you basically just revise. The specialists from these Organizations and companies present their instruments and there is not much time. And one is just concerned with understanding what they are saying, and sometimes with trying to adapt, but in most cases you do not question the content. (...) There is really a lot of asymmetry, it is not a shared building process, it is not that we are building the study together. The background questionnaires is where I noticed more openness, but not in the rest. There was a lot of distance in the technical management, between the specialists, who are the very top, and the countries, and so the reality is it is not a dialogue. (Interview, PISADcountry2016)

Participant observations substantiate the country perceptions of PISA being structured in a way that is not conducive to the consideration of multiple concerns because the structure of the instruments, the working modes, the relationships, the timeframes, and the valued knowledge are not in place to rethink the concept of policy relevance in the PISA era. Although this may be read as an imposed policy-relevance threshold, socio-material semiotics suggests that power is disseminated and that power is acquired through the assemblage stability and size. Assemblages which do not further the interest of its allies fall apart but when the PISA-D countries were faced with the dilemma of policy relevance or PISA relevance, there was unanimous consensus amongst PISA-D countries:

It's as if you have all the freedom in world but with limits, you cannot play that much and these limits are not only OECD-imposed limits, the countries impose them too. You take part in PISA because you want PISA, you don't take part in PISA because you want TERCE. At the end of the day, they want to preserve PISA and we also want to preserve PISA, so I would not see any reason to be in PISA-D if not to be part of PISA. (...) When we all took part, it was 'Well, I want PISA and if I am in PISA-D it's because I want to compare myself with other countries that are part of PISA and not just to compare with you lot'. And so yes, I do need those questions there, yes, I do need the socio-economic index fits with the PISA one, even if it does not reflect my reality. It cannot reflect my reality much, but I can sacrifice that. (...) We managed the best we could. Well, not the best we could. Within those limits, the best we could. (...) And yes, many things countries wanted were omitted. At the end of the day you need to decide what you prefer. And what was preferred was to have a stronger tie with PISA.' (Interview, PISADcountry 2016)

Countries were willing to sacrifice the relevance of the data, what they are not willing to sacrifice is being part of PISA and the epistemic community they have accessed through PISA-D. This extract suggests PISA relevance is not an imposed threshold but driven by unanimous consensus: getting PISA is what counts. From an epistemic community perspective, the PISA-D countries appear to share the causal, principled, and normative beliefs upon which PISA is built, the test's validity, and the overarching PISA policy enterprise.

Conclusions

This paper has told the history of PISA for Development - from the first experiences that led to the idea and creation of PISA-D. This adds a layer to the official PISA-D history told in the OECD documents online, by illustrating how PISA-D draws on the frustration of the OECD and India's experience with the main PISA's poorly relevant policy data; how the OECD managed to fit PISA-D neatly into the post-2015 global education agenda whilst aligning it with the OECD's strategy on development, and how the Organization took a no-risks approach to ensure PISA-D become a success story.

The paper then discussed how the OECD and its PISA-D partners negotiated the "policy-relevance" threshold. Through this negotiation process, the multiple rationales for the involvement of PISA-D actors emerged: policy knowledge sharing and geopolitical expansion, business opportunities, and political ties with valued global PISA communities. Although actors joining the PISA epistemic community appear to share the same beliefs, the notion of black box suggests they are temporarily believed truths accepted only in so far as PISA-D furthers their diverse interests. Having temporarily black boxed the shared beliefs and policy agendas of the PISA epistemic community facilitated the policy-relevance threshold negotiation process amongst all PISA-D actors.

The paper challenges PISA-D's policy-relevance claim driven by lower and middle income context policy priorities, arguing that the policy-relevance threshold has been driven by the multiple concerns of actors involved. However, all actors were concerned with maintaining a strong tie with PISA, resulting in a threshold of policy relevance which is best described as PISA relevant. In the days of epistemological governance, it might be suggested that PISA relevance provides more than policy relevance can provide.

This paper concludes by inviting all those involved in international assessment data production and use to reflect on how policy relevance is understood in the PISA era. It appears that the global education community has moved away from policy-relevant data to PISA-relevant data. Should we be asking which data and data interpretations count and not how can data be made more relevant to inform policy? What ethical implications might this have for the production of public policy knowledge?

Referências

• BLOEM, S. PISA in Low and Middle Income Countries . Paris: OECD Publishing, 2013.
• BLOEM, S. The OECD Directorate for Education as an Independent Knowledge Producer through PISA. In: ESPELAND, W.N.; STEVENS, M.L. A sociology of quantification. European Journal of Sociology v. 49, n. 3, p. 401-436, 2008.
• CARROLL, J.B. A model for school learning. Teacher College Record , n. 64, p. 723-733, 1963.
• GREK, S. OECD as a site of coproduction: European education governance and the new politics of 'policy mobilization', Critical Policy Studies v. 8, n. 3, 2014.
• HAAS, P.M. Introduction: Epistemic communities and international policy coordination. International Organization v. 46, n. 1, p. 1-35, 1992.
• LATOUR, B. Science in Action . Milton Keynes: Open University Press, 1987.
• LATOUR, B. Reassembling the social: an introduction to Actor-Network-Theory . Oxford: Oxford University Press, 2005.
• LAW, J. On Sociology and STS. The Sociological Review v. 56, n. 4, 2008.
• LINGARD, B. Policy as numbers: Ac/counting for educational research. The Australian Educational Researcher , v. 38, n. 4, p. 355-382, 2010.
• MURPHY, S. The Pull of PISA: Uncertainty, Influence, and Ignorance. International Journal of Education for Democracy , v. 3, n. 1, p. 28-44, 2010.
• OECD - ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. OECD Strategy on Development Paris: OECD, 2012.
• OECD - ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. Call for Tender 100000990 PISA for Development Strand A and B . Paris: OECD , 2014a.
• OECD - ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT. 2014 Report on the Implementation of the OECD Strategy on Development . Paris: OECD , 2014b
• ROSE, N. Powers of Freedom: Reframing Political Thought . Cambridge University Press, 1999.
• SELLAR, S.; LINGARD, B. PISA and the Expanding Role of the OECD in Global Education Governance. In: MEYER, H.D.; BENAVOT, A. PISA, Power, and Policy the emergence of global educational governance . Wallingford/GB, Symposium Books, 2013.
• WILLMS, D.; TRAMONTE, L. Towards the development of contextual questionnaires for the PISA for Development Study . Paris: OECD , 2014.
• WOODWARD, R. The Organization for Economic Cooperation and Development (OECD) . Abingdon: Routledge, 2009.

• *The author is grateful to the Fritz Thyssen Foundation for supporting the P4D4Policy research project.
• 1
  For clarity, it is worth noting that in this paper, policy-relevant data are understood as information which responds to policy needs of a specific educational system (i.e. data which represent students' backgrounds in a lower income country based on a socio-economic framework which fits the context rather than the context of a higher income country).
• 2
  A contractor is a private firm which responds to an OECD competitive bidding process to provide a service or product to the OECD.
• 3
  The documents I draw on are the power point presentations presented between 2013 and 2015 at three technical working meetings (27-28 June 2013 in Paris, 8-11 April in Washington DC, 1-2 October 2014 in Paris) and two International Advisory Boards (27-28 May 2014 in Paris, 11-13 March 2015 in Paris). All documents are available on the OECD PISA-D website page: https://www.oecd.org/pisa/aboutpisa/pisa-for-development-meetings.htm
• 4
  The Learning Bar describes itself as "a research-based education company that provides [...] tools and solutions designed to inform educators, guide school planning, and give a voice to students, parents, and teachers." Based in Canada, it was co-founded in 2005 by Douglas Willms. The company is now led b y Wills who is professor and director of the Canadian Research Institute for Social Policy at the University of New Brunswick (UNB). Although TLB is a new actor in the field of international education assessments, Willms has been involved in PISA activities since the inception of PISA: developing the instruments, chairing the PISA Technical Advisory Board, and writing international PISA reports.
• 5
  France, Inter-American Development Bank (IADB), Korea, World Bank, Global Partnership for Education (GPE), Norway (Norad), UK (DFID), Germany (BMZ/GIZ), Japan (JICA), and Ireland (Irish Aid).
• 6
  UNESCO, UNESCO Institute of Statistics (UIS), Education For All Global Monitoring Report (EFA GMR) team, UNICEF, WEI-SPS, Education International, PISA SDG, PIAAC team, and the assessment programs: ASER; EGRA; EGMA; SACMEQ; PASEC; Pre-PIRLS and PIRLS; TIMSS; LLECE; STEP; LAMP; UWEZO.
• 7
  For five of the participating countries a funding agreement was established: Senegal is financed by the World Bank and France (AFD), Honduras by the Inter-American Development Bank, Guatemala by the Inter-American Development Bank but also by Norway and Germany, Zambia by the World Bank and DFID, and Cambodia by South Korea. Ecuador and Paraguay are self-funding their participation in PISA-D. Furthermore, each country has a main PISA country as a peer.
• 8
  During this initial technical meeting, PISA-D was officially presented and aimed at reaching: (1) a "shared understanding among participants and partners" - as presented in the power points available online at http://www.oecd.org/pisa/aboutpisa/pisafordevelopment.html
• 9
  Although this aspect highly debated, large-scale international assessments do this by using culturally neutral items.
• 10
  This is drawn from Andreas Schleicher's welcoming introductory power point presentation at the Initial Technical Meeting.
• 11
  Program for International Assessment of Adult Competencies carried out by the OECD.
• 12
  Skills measurement study carried out by the World Bank.
• 13
  Literacy Assessment and Monitoring Program carried out by the UNESCO Institute for Statistics.
• 14
  TLB is to assist the PISA-D countries to develop data analysis into policy devices (i.e. national reports which respond to the specific policy needs of each country's educational policy priorities and challenges) and develop capacity of national policy makers to use the PISA-D data for policy.
• 15
  In 2016, the OECD invited countries in Asia and Latin America to consider participation in a second round of PISA-D, asking countries to present (at international OECD-sponsored events) their motivation to participate, expected outcomes, potential challenges, and relationships between national assessments and PISA/PISA-D).
• 16
  This is further stated in the 2014 revision of the: "PISA for Development is well placed to support global efforts to frame a learning goal in the context of the post-2015 agenda and to provide a single universal metric for measuring progress on this. PISA is regarded as one of the most important education policy instruments in the world today. There is a compelling logic for making this more relevant to developing countries so that greater numbers can benefit from the surveys and analysis. The project therefore addresses squarely the core elements of the Strategy on Development, particularly the adaptation of the most successful OECD policy instruments to make these more relevant for developing country contexts." (OECD 2014b, p. 10)

Publication Dates

• Publication in this collection
  Jul-Sep 2016

History

• Received
  30 June 2016
• Accepted
  05 Sept 2016