SciELO - Scientific Electronic Library Online

 
vol.43 issue4Teaching Work and Capes Assessment: estrangement and naturalizationInterfaces between Evaluation and History Curriculum at High School author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Journal

Article

Indicators

Related links

Share


Educação & Realidade

Print version ISSN 0100-3143On-line version ISSN 2175-6236

Educ. Real. vol.43 no.4 Porto Alegre Oct./Dec. 2018

http://dx.doi.org/10.1590/2175-623684893 

THEMATIC SECTION: EDUCATIONAL MANAGEMENT AND EVALUATION POLICIES

National and International Assessment in Brazil: the link between PISA and IDEB

Marialuisa VillaniI 

Dalila Andrade OliveiraII 

IInternational Network of Research on Reestructuring of Educational Professions (WERA/INRREP), Paris - France

IIUniversidade Federal de Minas Gerais (UFMG), Belo Horizonte/MG - Brazil

Abstract:

This article analyses the relationship between the Program for International Student Assessment (PISA)designed by OECD and the Index of Development of Basic Education (IDEB), designed by INEP. Currently, educational systems are subjected to implement national and international standard assessments. This process is the result of the New Public Management (NPM) policy implementation, which in Brazil starts at the beginning of the nineties. In this context, statistical data play a relevant role as a government policy tool. This work presents the development of PISA and IDEB, how they relate - from a statistical, methodological and political perspective, and how these dynamics influence the Brazilian educational policy.

Keywords: PISA; IDEB; National Standard Assessment; International Standard Assessment

Introduction

The aim of this article is to analyse the link between national assessment programs and indicators, specifically we present the relationship between the Index of development of Basic Education (IDEB), designed by the Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (Inep), and the Program for International Student Assessment (PISA), developed by the Organization for Economic cooperation and Development Cooperation (OECD), that is also implemented in Brazil. This research inquires how those two educational measurement tools influences the educational policies of the country. This paper will present data from a research focused on the production of PISA data in Brazil. This work was funded by REFEB13 fellowship of the French Embassy in Brazil in 2016. We will present data from interviews conduct with Inep researchers who participated in the PISA process in Brazil in the 2012 and 2015 editions. The qualitative part of the research used a semi-structured interview, in order to allow the approach of several topics that way, adapting the tool to each participant interviewed14.

The contemporary policies are increasingly the results of an analytical process based on the “evidence-based policies”. This policy framework has been developed in the educational field in Brazil. According to Solesbury (2001) the policies are produced and disseminated under accountable politicians’ control, and this process the “pragmatism” replaces the ideologies. To define the concept of “evidence”, Solesbury refers to the Oxford English dictionary definition “the available body of facts or information indicating whether a belief or proposition is true or valid. He uses two key concepts “availability” and “validity”. It is from those two factors that international large-scale assessments, as well as national, evaluation programmes are developed in different parts of the world. This model is related to the New Public Management paradigm, which in Brazil started to be introduced in the nineties (Oliveira, 2015).

National and international large-scale assessment programmes use statistical data as an assessment and knowledge tool to reason their “objectivity” nature. Therefore, according to Foucault’s (2004) theories, they develop a link between statistics and policy which is not linear. Foucault analysed in several of his works the relationship between power, policy design and technical tools, Foucault developed the governmentality concept to describe the social and political control made by the governments:

[...] With this word governmentality I want to say three things. By governmentality I mean a whole represented by institutions, procedures, analysis and reflection, calculations and tactics that allow me to exercise this specific and complex type of power which has as a principal target the population, and as a major type of knowledge the political economy, as essential technical tool the securities feature [...] (Foucault, 2004, p. 111)15.

Lascoumes (2004) affirms that during the last 30 years statistics, and consequently international scale assessment became the new technologies of the power. In this view, the quantification process of the social phenomena influences not only the statistics, but also the performance indicators and all the tools implemented by the New Public Management (Desrosières, 2014). The use of statistics according to the NPM framework produces discontinuity with the oldest application made by the governments. According to Desrosières (2014), statistics indicators designed under the NPM framework produce a retroactive effect on the action and behaviours of political and social actors. This retroactive effect also influences the national programmes what is intended to be analysed in the text.

OECD as Transnational Actor of Development Policies in the international scenario

For a better understanding of International Large-Scale Assessment, we use as reference the distinction proposed by Nóvoa and Yariv-Mashal (2003): in the first case the assessment programmes focusing on comparison of national cases without a specific time space context; in the second case the comparison focusing on studies of variables, that confront results of different sets of variables. According to Lindblad, Pettersson and Popkewitz (2015), the first perspective is linked to human sciences (particularly this heuristic is used by the disciplines of literature or science of educational field); the second one is related to social sciences (this second paradigm is more developed in sociology of education, economy of educational or social statistics field). This kind of debate developed, also in comparative studies field (Steiner-Khamsi, 2013). Lindblad, Pettersson and Popkewitz (2015) propose several examples. Standardised comparisons measure outcomes against norms, school indicators and gender equality. Amongst those type programmes there are, for examples International Large-Scale Assessment (ILSA). The International Large-Scale Assessment, which is an assessment based on standardised tools that promote the comparison of pupils’ or adult’s knowledge and competencies in different countries in the world. The agencies that designed and further implement this kind of assessments are Association for the Evaluation of Educational Achievement (IEA) and the OCDE.

PISA16 is one of the most developed and applied international assessment programme in the world. The peculiarity of this programme is the use of the Item Response Theory (IRT). This method offers the possibility of applying the test by generalising the predictive value of individual performance, without requiring the application of the whole test. The use of this method, allows highlighting the problems of applying the test. The IRT is an assessment tool where technical characteristics produce changing in function of the sample, thus developing measurement tools whose characteristic is not influenced by a reference group, and it defines an “absolute” scale (Pini, 2012).

The Latin America countries started assessment programmes in numeracy and literacy with the help of United Nations for Educational, Scientific and Cultural Organization (UNESCO). The main purpose with these results were: to reduce the cultural gap with other assessment programmes and improve the relevance of the assessment; conducting international comparative analysis and explanatory analysis of pupil’s success; identifying the socio-familiar factors that influence the pupil’s success; providing useful information to design and implement educational policy in this region (Lindblad; Pettersson; Popkewitz, 2015).

Bottani and Vrignaud (2005) attribute the development of school assessment driven by international agencies as OECD to political, economical and social conditions that allowed the expansion of these surveys. They define three elements that increase this process: expansion of educational system in OECD area, the uniformization of educational models that reduced the gaps between the school systems, producing uniformisation, the increasing dichotomy in educational area between scientific purposes in one hand, and political goals in the other hand. Creating and implementing international assessment programmes, focusing on knowledge and competencies of pupils requires interlinking of complementary sectors that includes: administration and management of the assessment program, the start-up funding, accounting plan and final account, design of research tools, work of data centre design and data processing, data analysis and dissemination of results (Bottani; Vrignaud, 2005).

Nowadays ILSA are the key players of the educational assessment scenario and its dissemination representing the development of methodological measurement dominance, according one part of academic world (Lindblad; Pettersson; Popkewitz, 2015).

According to Mons (2009), standardized assessment programs are part of Policy Evaluation dynamics, defined by Spenlehauer (2003) as policy evaluation. This paradigm is based on experimental Science model that proposes, with a pragmatic approach, to organise in three sequences or system of policy action. The model is structured in three steps: project; measurement, analysis of effects.

In order to analyse these dynamics that represent the link between standardisation assessment and policies, Mons also uses the economic logics of human capital and the school effectiveness principle. According to the author the economists refer to the paradigm of the production function and they use production factors (inputs) in order to elaborate a product (output). When these models are translated into educational field, according to Mons (2009), it is possible to obtain as results several inputs - educational material, teachers with specificity (time of training, different period of work, etc.) - in order to produce a quantitative evaluation school system (year of schooling) and qualitative (measurement of students acquires). These two elements are needed by economists to run the analysis models. For the data gathering they use a standardized assessment. This explains the fact that the analysis approach of economists for the educational systems, and also in general, is based on results regulatory (Wöessmann, 2007).

In the Brazilian context this dynamic defined by this approach develops with the Ideb implementation. With the Ideb design in 2007, within the Educational Development Planning (Plano de Desenvolvimento da Educação PDE), as reference to design and guidance of policies and public funding of education at national level, it was applied the economic logic to govern and manage the public educational system. Using the Prova Brasil and SAEB (National Assessment System of Basic Education) the Ideb produces a direct influence on the regulation of pupils’ performances, but also it influences the action of the other scholar actors, and it impacts the organizational logic of the schools including curricula organisation and management planning. Based on that, the accountability starts to disseminate across the country, in the small village as in the bib capitals, educational policies established goals and negotiation process to achieve its with the schools. It assist to introduction of transaction between schools and federal governments, having as parameter to achieve the Ideb average (Oliveira; Jorge, 2015).

This effect produced by Ideb create a reference for educational policies, including The Ministry of Education, school management, it is considering as an international trend that happens with PISA. Bottani e Vrignaud (2005, p. 17) relatively to PISA program they affirm that is an assessment predominance:

[...] OECD obtained that which IEA did not accomplish for forty years, which means they not only brought the attention of policymakers to these assessments, but they also reoriented policies in several countries; PISA became reference to justify all kinds of choices and reforms. We can almost affirm that PISA assessments changed educational policy landscape at world level [...]17.

According to Morgan OECD with PISA program positions itself as a leader in statistical data design and production in educational field:

[...]By creating the PISA, the OECD has positioned itself as a leader in the international statistical and data gathering and production infrastructure that exists within a global architecture of education. This global structure includes several key international governmental organizations such as the World Bank, the International Monetary Foundation (IMF), and the United Nations. Regional organizations such as the European Union (EU) are also connected to it. In addition, there is the OECD’s ‘rival’ in educational measurement - the International Association for the Evaluation of Educational Achievement (IEA) which was created under the auspices of the UNESCO Institute for Education (UIE) [...] (Morgan, 2011, p. 4).

The PISA programme is recognised at international level as the producer of the major part of knowledge about educational systems in several part of the world. At the same time, the production of the assessment programme influences at different levels the policy making. PISA can be considered as a tool of knowledge policy. As Carvalho states (2012, p. 174), it works as a knowledge generating device:

To repeat, this means that PISA is a knowledge-based and knowledge-generating device which, combining technical and social components, takes part in the setting up of rules for the coordination and the control of public action in education.

This knowledge production develops also through the methodology of production and application of PISA assessment. That process creates a knowledge monopole governed by OECD, and this was realised through PISA design. According to Carvalho exists a group of experts from several OECD sectors and other public and private organisation that participates in different way in PISA coordination. In the Brazilian case these experts work at the Inep, that is the agency in charge of PISA in the country, but also in private companies that contribute to logistic part of the program.

Pettersson (2014) defines the impact of PISA on educational policies as an international comparative hegemony. In the article Three Narratives: national interpretations of PISA (Pettersson, 2014), the author identifies three narratives as interpretative models used at national level to analyse the influence of PISA on educational policies.

The first narrative is defined as ontological. In this model national reports define the way whom PISA function can be interpreted and the way the countries accept the comparative monitoring as a tool of improvement of educational offer. This model develops the concept of effectiveness, governmental capacity, globalised market economy. According to the author, these elements are prerequisite to the ontological narrative.

The second narrative is the epistemological. In this perspective is analysed the way to interpret the concepts of competences and knowledge. The OECD declares that through PISA assesses pupil’s skills reflecting on knowledge and experiences, the way they apply its on daily problems. According to Pettersson (2014), this theoretical paradigm is embodied in the literacy concept. The literacy is not shape by national curriculum, but by the common sense. The third narrative is defined as narrative of the reform, that describes as recommendation made by PISA national reports.

According to the author and using these three narratives simultaneously it is possible promotes the national perception of PISA for each country (Pettersson, 2014). In the Brazilian case we have the possibility to identify, in data from interviews, models based on epistemological and ontological narratives, which are introduced to Inep through the implementation of the program, but also with the training of researchers that participate and disseminate at national level OECD methodologies. The research data made at Inep show trough the narrative of interviewees how this process is embodied by the involved actors:

[...] One of the thing that I did during... I guess that was interesting, as I was learning on PISA, every time I learned something in depth, I called my colleagues that worked on national assessments and I gave a seminar on it…so I gave seminar on mathematics, on Portuguese performances in 2009, literacy 2009, on science matrix…I organized a seminar on PISA questionnaire… there went not a lot of persons, what I say…I had this willingness to make: let’s disseminate the good part of the project that is the knowledge that people has, not only at the Inep, but also I was invited to give a post-graduation seminar on PISA for the educational assessment class, at the São Paulo University, I was at Carlos Chagas Foundation [...] (Interviewee 2, 2016)18.

The definition of OECD as educational transnational actor is considered due to an expansion process depending on several dimensions. According to Carvalho (2012) it is possible identify a dimension of expansion related to the time (this program exists since 2000), but also it is possible to consider the geographical dimension, currently 75 countries participate to the programme and other are involved trough the PISA for Development (Addey, 2017). This expansion is also related to the assessed competencies, as the ICT (Information Communication Technology) competences. Furthermore, Carvalho (2012) describes an expansion that produce interconnection between PISA and the other OECD programs.

According to Mons (2007), PISA produces a universalistic idea of educational comparison that aims to show the isomorphism process and sharing of goals of considered countries. Consequently a comparison based on same analysis criteria and representation seems methodologically justifiable. On the basis on this elements, Felouzis and Charmillot (2012) propose an analysis of several edition of PISA and show how concepts of efficacy and equity are underling elements of theoretical framework of the programme.

According to Morgan (2011), supranational structures as OECD approach issues linked to comparative educational theories in vertical way. In line with the authors, agencies as European Union through the idea of (life-long learning), UNESCO (Educational for All) and OECD (literacy concept through PISA) develop concepts that became determinant factors in national educational policies all around the world and also in international assessment programs. The countries that use these theoretical and methodological frameworks have as goal the harmonisation of their educational system and they want to analyse to which extent they are achieving to international criteria (Morgan, 2011).

Data shows that in the Brazilian case, the interest to develop this kind of connection with the international criteria is encouraged by Inep, that continuing to be the Brazilian institution with major knowledge on PISA. It is important to highlight that the expansion of PISA in Latin-American countries as Brazil, regarding the production of statistical educational data, produce a misalignment process (Carvalho, 2012) at methodological, logistical and analytical level (Villani, 2018). The results of a research regarding the production of PISA data, show several criticalities related to the gathering and dissemination of information regarding the country scholar system. We observe that PISA application in Brazil reflects made a partial representation of brazilian scholar reality that is complex, very diverse and with a high level of economical and cultural inequality (Villani, 2018).

It is important to highlight besides that OECD produces a soft influence on policy, due to the fact that it not has legal tools to promote educational policies (Pettersson, 2014). This happens in Brazil except for the inclusion on Educational National Plan (Plano Nacional de Educação PNE) Federal Law nº 13.005, of June 2014, in 7.11 strategy, of 7th goal, the PISA average as educational quality indicator in the country. In the meanwhile, OECD reach to produce an international political discourse based on international rankings and assessments. Therefore, it is import to understand that PISA develops two function in the construction of the international political discourse: one economic, another educational (Pettersson; 2008; 2014).

The Double Role of Inep as National and International Actor of Educational Policy in Brazil

Nowadays, the Inep is the agency that produces more data on Brazilian educational system, as an autarchic organism depending on the Ministry of Education. It was funded in 1938, but in 1950’s that Anísio Teixeira started to develop the Institute goals defining one of them as: movement of educational re-checking and re-assessment (reverificação e reavaliação educacional). In 1955 there were created the Educational Research Centres. During the military dictatorship (1964-1985) the functions of Inep have been limited and Anísio Teixeira has been persecuted and dead. In the 1990’s, with the democratisation process of society and the approval of 1988 Federal Constitution, that guaranteed more rights for the population in the Brazilian history, the Inep becomes has principal missing the assessment and knowledge production on Brazilian scholar system (Ferreira Santos, 2008; Schwartzman, 2013). Currently, according to one of the interviewees the principal goal of Inep is to guide the work of MEC (Ministry of Education):

[...] I think is important talking with you about how we organize here, in this institution that is Inep, National Institution of Educational Research Educacionais AnísioTeixeira, that is an autarchy of Ministry of Education…it has link in metaphoric way, we use to say that Inep is 1% of educational budget that guides or should guide the remaining 99%, it is an institution that plans all the strategy of research school education system policy as well as higher education system and also it is in charge of the educational assessment, that is a recent responsibility, the institute aquired its in the 1990’s, when it took charge the responsibility for the implementation of SAEB assessment (Scholar Education Assessment Program)…so we have several department, and we have the department of scholar system assessment DAEB, that is in charge of the implementation of the scholar system strategies in the country... it has the responsibility of the management of the assessment of all scholar grades assessment, it manages o National High School Exam (Exame Nacional do Ensino Médio Enem), that is an important exam created in 1998, but that since 2009 is used also as admission test for the Public University entrance[...] (Interviewee 1, 2016).

As the interviewee declares, the major role of Inep is the educational guidance in the country. Consequently, it is possible to affirm that the institute play a double role in relation to the three factors mentioned: statistical, political and geographical. Statistical because it is the organism with the major production of educational data through the Censo Escolar19 and the several assessments of scholar and higher education system. Political because, as interviewees declare, the Inep experts realize strategies of educational policies guidance in the country. Finally, under a geographical point of view, because Inep produces a national and international work, not only through the PISA, but also with other international institutions as the UNESCO.

The Inep acts at national and international level under the NPM paradigm. That is how in 2005, the Institute decided to produce a quality indicator of the Brazilian scholar system that fits with a dynamic of accountability implementation (Alves; Soares, 2013), already implemented in other Latin-American countries and all over the world. This aim has been declared by MEC in the Ideb design:

[...] It is an initiative of Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (Inep) to measure the performance of Brazilian educational system, combining the pupil’s performances in external standardized assessment(Prova Brasil e Saeb) and the approval rate, the indicator has influence on efficiency of scholar flux, in other words, in pupils ‘progression between steps/years (Brasil, 2015 p. 6)20.

The Ideb is the result of a product between the Brazilian pupils’ flux in the school system and its performance, calculated through SAEB21 (Fernandes, 2009), represented by the following formula (Figure 1):

Source: Fernandes (2009). i= year of the exam (SAEB) and of Censo Escolar; Nij= the average of Literacy and Numeracy performance standardized for an indicator between 0 e 10 of pupils of j unit, obteind in a specific edition of exem realized in the end education level em; Pij= performance indicator based on approval rate of pupils ‘educational stage of j unit

Figure 1 Statistical Formula of IDEB 

IDEB and PISA a Statistical and Political Relationship

The relationship between Ideb and PISA started in the indicator design. In the indicator reports on Ideb (Fernandes, 2009; Brasil, 2015) we can find methodological references linked with these two devices:

The definition of a national goal for the Ideb in 6,0, it means that the country has to achieve in 2021, considering the starting years of basic education, the quality level of education, related to the proficiency and income (approval rate) of the developed countries average (the average of OECD member countries) currently observed. This international comparison was possible due to standardisation between the distribution of the PISA (Programme for International Student Assessment) average and SAEB (Fernandes, 2009 p. 2).

The Inep choice of using PISA as a reference model (statistical and epistemological) of school system quality is an example of expansion process described by Carvalho (2012) and Pettersson (2014). Using the Pettersson (2014) paradigm it is possible highlight an epistemological dynamic and reform to. PISA is used not only as a methodological reference (Carvalho, 2012), but also to develop national educational policies aims.

The Entry since October 201322 of Brazil in the PISA Governing Board (PGB) is another important element. This event place Brazil in a new position between the non-member countries, it acquires political power within the Program and OECD. From this moment each Brazilian delegate becomes automatically the PGB vice-president. It is this organism that decides all the actions and policy choices related to PISA.

It is important to emphasize how this process that happens in Brazil within Inep points out the strong role of OECD as transnational actor. OECD exercises a soft power interfering in national policies, producing transformation and uniformization of educational quality idea that becomes incorporated by governmental actors, academic sectors, social sectors (Foucault, 2004; Pettersson, 2014).

The use of PISA as a reference in the Brazilian school system measures design can be considered also an Inep policy action. In June 2013 meeting of PISA for Development, the Brazilian delegate, the Professor Luiz Claudio Costa, the-then Inep president, presented a graphic (Graphic 1) that defined the comparison process between PISA and Ideb.

Source: INEP presentation of Luiz Costa, OECD (2013)

Graphic 1 Conparison between Ideb and PISA scale presented in June 2013 meeting Comparação entre as escalas do IDEB e do PISA apresentado na reunião PISA for Development, June 2013  

The Inep interest in developing a relationship between Ideb and PISA has been publicly declared by the Institute, as it is possible to see on Inep site, among their goals23. This action is structured also in PNE policy, Ideb and PISA have been introduced in PNE 7 paragraph, as already mentioned. Analysing the comparison process between Ideb and PISA, it is possible to evidence several misalignment points from a statistical point of view. The Ideb is designed from results of performances based on Brazilian curriculum knowledge. The PISA is designed on a test made to assess pupils ’competences in literacy, numeracy and sciences. In order to realize this comparison between these two tools, it would be necessary using a similar item in each test to assess the pupils’ performance (Soares; Xavier, 2013). Consequently, this comparison process is made in two devices designed with a different nature. These critics already have been formulated by Soares e Xavier (2013). They point out issues related to the statistic reliability of indicator. Moreover, according these two authors, the Ideb success ad quality indicator in the Brazilian academic, political and scholar context, obtained so relevance that it was used, nowadays, as the only indicator of Brazilian school quality reference. It is possible arise from that PISA, for the fact that is used as a reference model, it has been included in the Brazilian educational legislation, It plays a reference model of educational quality, thus far in indirectly way. This indirect impact on educational system acquires more relevance if we consider, unlike the Ideb, that PISA is not a reference in the Brazilian public debate, including the scholar actors indeed. In the meanwhile, it is presents in the debate between actors linked with the national agency, especially between public administrators that using NPM criteria and embedded its principles in the promotion of competitive education as synonymous of quality. This is observed by one of the interviewees that has been a local public administrator:

[...]24 There is something that emerged and me as local public administrator used, we have Ideb that I mentioned, you already know that the Ideb as a quality tool is limited, the quality concept beyond it is strength, so It is not a good reference for example for a networks (of schools)that have already a good Ideb (results), in that place they had already 6 as a Ideb result, so It was already good, but It was no good, when you talked with teachers, with persons, you wen at the schools and you saw that you had a lot of thing to do, their own teachers and community understood that. So, I used PISA results to develop a dialogue with the community, with teachers to say: “People is our municipality good? Yes, it is, but please focuses on my discourse: what here is 6 when we observe PISA and Brazil, we see that we need to improve more and more…”, so It (PISA) helps to creates a debate on quality for whom (schools) are already, let say, at advanced level. (…) And also PISA helps to give this dimensions of what can be defined as “advanced”, in educational terms as global competencies, with a spectrum of needs in the Sciences field, of literacy and problem solving of problem with a big complexity [...] (Interviewee 1, 2016).

This misalignment process is evident also in the relationship between PISA and SAEB, this latter is one of the elements that form the Ideb. SAEB test “[...] has reference matrix elaborated starting from a summary of state and municipal curricula propositions, and PCNs” (Becker, 2012, p. 38). Particularly this difference is noticeable in PISA learning outcomes (that assesses pupils’ of 15 years old competences) and the test used in the SAEB exam (that assesses the knowledge through cognitive tests and contextual questionnaires based on Brazilian scholar curricula), and event more visible between PISA learning outcomes and the organization of national curricula that present different goals.

Another relevant factor is the use of PISA and Ideb results by public school networks and private schools. This difference between the public and private schools reality in Brazil is a relevant characteristic of the system. It is important to consider the inequalities of the country, which are reflected on educational offer, above all, referring the public offer. Moreover It is more disparate the school segmentation, due to the fact that pupils from middle and upper classes go to private schools. This difference, is also evident in PISA average between public and private schools (Villani, 2017), and that private schools have a result above the PISA average. However, It is important to highlight that federal public schools have good PISA performances as well, overtaking private school pupils’ results, but they are in charge of only 5% of educational offer at this stage.

In terms of that, PISA seems to be a reference more important for the private schools, that for the public one, as observed by one of the interviewees who answered about the use of PISA data:

In the public schools no, in the public schools I do not see that, I see that in some private institutions, that ask to Inep, for example, to realize a PISA for Schools, that is an initiative implement in some countries (Interviewee 1, 2016).

In that case, it is possible identify the PISA role as a new devices and pedagogical tools producer (Carvalho, 2012). It is possible that the actions promoted by Inep, - as Inep promotes debates, seminars, meetings on basic education assessment, that try to involve state and municipal public administrators and academic actors - are a way to increase its role, in addition to be an organism that produces statistical data. These actions reflect, to a certain extent, a political internationalization from Inep using a recognized international reference as PISA, and also they exercise an impact on government and policy decision-makers.

This concern is also present between Inep experts that are aware of methodological weakness of Ideb and inadequate of the programme the implementation (Ideb and PISA) as a reference for the same policies. We take it from the narrative of one of the interviewees:

[...] I think that the initial error was…the Ideb says…When you achieve the 6 average you will achieve the 500 PISA average…you download the technical report…It makes a very simple calculation, you take the pupils that are enrolled in the school year considered in the SAEB average and compare, you take out and you will create a scale, basically they did that…but this assessment they…they are not at the same time, let me say, you can not known compare a group of pupils in the same way, in a so simple way, you do not know if is the same pupil, if the test is applied in the same day. In order to make a scale comparison, usually you need to use more sophisticated models, as to collocate some PISA anchor items in a national assessment and analyse the pupils’ performances, in order to create a standardized comparison. What has been done, has been badly done, I think that the 6 average is important because it fix a goal for schools [...]

[...] PISA is um cognitive tool while Ideb is made by two elements, a cognitive and flux factors, therefore there are two distinct measurements, you can improve your Ideb performance due to a flux change, and at the same time the PISA performance can get worse [...] (Interviewee 2, 2016).

In addition to methodological issues, in these data it is possible to underline the technical difference between PISA and Ideb. The interview points out the discrepancies that are produced by a technical-statistical process and pedagogical tools, in this case the scholar curriculum. This difference produces a misrepresentation between what is taught (scholar curriculum) and what is assessed.

Collected data highlight a retroaction effect (Desrosières, 2014) that the Ideb implementation developed in Brazilian scholar system. These dynamics influence policies because they have an impact on local school actors, teachers and families (Oliveira; Duarte; Clementino, 2017). These policies designed on Ideb impact the school management. The school establishment receives a lot of pressure to achieve the performance goals established starting from Ideb. The school then negotiates the goals, it does not organize following its principals or purpose, but focusing on test results. Soares e Xavier (2013), para descrever esse fenômeno se referem à teoria de Campbell (1979, p. 85):

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

To describes this phenomenon, Schwartzman (2013) use the Goodhart effect or Goodhart law, referring to the economist that in 1975 developed a theory to demonstrate that when the governments try to regulate a particular sector of financial actions, these latter become not reliable as indicator of economic trend. According to Schwartzman this theory in contemporary Brazilian educational indicator scenario explains the current assessment phenomenon and the fact that indicators lose their explanatory power when the government acts in them.

Conclusion

Brazil presents a particular scenario, in the respect of the context within assessment policies at global scale, due to the implementation as educational quality reference a national and international indicator. These indicators have as function the standardisation of process, actions and values in a context diversified and inequality context. In this way the standardization of goals can produce new imbalance and can cover issues and criticalities related to local context that would need attention. In the last 10 years, with the Ideb consolidation as national educational quality measurement indicator, we assisted to a standardization process, in Brazilian reality, that influence not only educational policies decision at federal level, but also at state and local level, due to country federative repartition. Several researches show the Ideb effect on scholar actor at local level (Almeida; Dalben; Freitas, 2013; Oliveira, 2017; Clementino, 2018). These dynamics develop an educational market that acts, in one hand through the retroaction effect of indicators on choices of families, and in the other hand, directly on teachers’ career.

This article tried to present some issues, both in the epistemological that in policies area, with regard to these two assessment devices, used in the Brazilian school system: Ideb and PISA.

It is possible highlight an expansion process developed by OECD in the Brazilian context, this kind of dynamic is developing in all PISA participant countries. Nevertheless, for Brazil this influence can produce a double effect: on the one hand we find a harmonisation effect, related to the standardization process that concerns all PISA participant country; in the other hand an uniformization process related to a western educational model. This second effect disregards (or not recognizes) the specific characteristics and needs of a scholar system situated in Latin-American region, with its socio-historic specificities. This disregard dynamic produces a misalignment (Carvalho, 2012) that we underline in relation to the statistic part as well as the scholar system, through the different purposes between the PISA learning outcomes and the Brazilian scholar curricula learning outcomes.

It must also put to produce, in consideration of institutional desire from Inep, an international relationship. The Ideb design reveals the agenda to participate at the standardization game with the powerful institution in this field, developing an own indicator that can contributes to create an educational model more harmonized and comparable. However, some weaknesses point out by researches on indicator, as already mentioned, demonstrate that besides the methodological issues, this process was not able to embrace the complexity represented by Brazilian educational system.

The central element that characterizes the relationship between OECD and Inep in the design and implementation of assessment device is a dynamic determined by the indirect influence of embody process of international reference model. This relationship is very different from the influence that PISA has in the other countries, and in the daily practices hardly appears in the public debate, but at the same time it is enclosed in the assessment tool and creates a major reference, at least according the issues emerged in the analysis.

REFERENCES

ADDEY, Camilla. Golden Relics & Historical Standards: how the OECD is expanding global education governance through PISA for Development. Critical Studies in Education, Melbourne, v. 58, n. 3, p. 311-325, 2017. Disponível em: <Disponível em: https://doi.org/10.1080/17508487.2017.1352006 >. Acesso em: 29 maio 2018. [ Links ]

ALMEIDA, Luana; DALBEN, Adilson; FREITAS, Luiz Carlos. Ideb: limites e ilusões de uma política educacional. Educação & Sociedade, Campinas, v. 34, n. 125, p. 1153-1174, 2013. Disponível em: <Disponível em: http://dx.doi.org/10.1590/S0101-73302013000400008 >. Acesso em: 6 mar. 2018. [ Links ]

ALVES, Maria Teresa Gonzaga; SOARES, José Francisco. Contexto Escolar e Indicadores Educacionais: condições desiguais para a efetivação de uma política de avaliação educacional. Educação e Pesquisa, São Paulo, v. 39, n. 1, p. 177-194, 2013. Disponível em: <Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S1517-97022013000100012 >. Acesso em: 7 jun. 2018. [ Links ]

BECKER, Fernanda da Rosa. Avaliações Externas e Ensino Fundamental: do currículo para a qualidade ou “da qualidade o currículo”. Revista Iberoamericana sobre Calidad, Eficacia y Cambio en Educación, Madrid, v. 10, n. 3, p. 37-48, 2012. Disponível em: <Disponível em: http://www.rinace.net/reice/numeros/arts/vol10num4/art3.pdf >. Acesso em: 6 maio 2018. [ Links ]

BOTTANI, Norberto; VRIGNAUD, Pierre. La France et les Évaluations Internationales. Paris: Rapport pour l’haute Conseil de l’évaluation de l’école; Documentation Française, 2005. Disponível em: <Disponível em: http://www.ladocumentationfrancaise.fr/var/storage/rapports-publics/054000359.pdf >. Acesso em: 5 abr. 2015. [ Links ]

BRASIL. Ministério da Educação. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Resumo Técnico, Resultado do Índice de Desenvolvimento de Educação Básica 2005-2015. Brasília: INEP, 2015. [ Links ]

CAMPBELL, Donald. Assessing the Impact of Planned Social Change. Evaluation and Program Planning, New York, v. 2, n. 1, p. 67-90, jan. 1979. [ Links ]

CARVALHO, Luis Miguel. The Fabrication and Travel of a Knowledge-Policy Instrument. European Educational Research Journal, London, v. 11, n. 2, p. 172-188, 2012. [ Links ]

CLEMENTINO, Ana Maria. Os Docentes frente a Política de Avaliação do IDEB: responsabilização e resistência. In: INTERNATIONAL CONGRESS OF THE LATIN AMERICAN STUDIES ASSOCIATION - LATIN AMERICAN STUDIES IN A GLOBALIZED WORLD, 36., 2018, Barcelona. Anais... Barcelona, 2018. [ Links ]

DESROSIÈRES, Alain. Prouver et Gouverner: une analyse politique des statistiques publiques. Paris: La Découverte, 2014. [ Links ]

FELOUZIS, Georges; CHARMILLOT, Samuel. Les Enquêtes PISA. Paris: PUF, 2012. [ Links ]

FERNANDES, Reynaldo. Índice de Desenvolvimento da Educação Básica (IDEB): metas intermediárias para a sua trajetória no Brasil, estados, municípios e escolas. Brasília: INEP , 2009. Disponível em: <Disponível em: http://download.inep.gov.br/educacao_basica/portal_ideb/o_que_sao_as_metas/Artigo_projecoes.pdf >. Acesso em: 7 jul. 2016. [ Links ]

FERREIRA, Marcia Santos. Os Centros de Pesquisas Educacionais do INEP e os estudos em ciências sociais sobre a educação no Brasil. Revista Brasileira de Educação. São Paulo. v. 13 n. 38 maio/ago, p. 279-292, 2008. [ Links ]

FOUCAULT, Michel. Sécurité, Territoire, Population: cours au Collège de France (1977-1978). Paris: Seuil, 2004. [ Links ]

GRENET, Julien. PISA: une enquête bancale? La Vie des Idées, Paris, 8 fev. 2008. Disponível em: <Disponível em: http://www.laviedesidees.fr/PISA-une-enquete-bancale.html >. Acesso em: 24 set. 2015. [ Links ]

LASCOUMES, Pierre. La Gouvernementalité: de la critique de l’ État aux technologies du pouvoir. Le Portique - Revue de philosophie et de sciences humaines, Strasbourg, n. 13/14, p. 1-15, 2004. Disponível em: <Disponível em: http://leportique.revues.org/625 >. Acesso em: 12 jun. 2016. [ Links ]

LINDBLAD, Swerker; PETTERSSON, Daniel; POPKEWITZ, Thomas. International Comparison of School Results: a sistematic review of a research on large scale assessments in education. Stockholm: Swedish Research Council, 2015. [ Links ]

MONS, Nathalie. Les Effets Théoriques et Réels de L’Évaluation Standardisée. Paris: EACEA Eurydice, 2009. [ Links ]

MONS, Nathalie. Les Nouvelles Politiques Éducatives. Paris: Puf, 2007. [ Links ]

MORGAN, Clara. Constructing the OECD Programme for International Student Assessment. In: PEREYA, Miguel (Org.). PISA under Examination: Changing Knowledge, Changing Tests, and Changing Schools. Rotterdam: Sense Publishers, 2011. P. 47-59. [ Links ]

NÓVOA, António; YARIV-MASHAL, Tali. Comparative Research in Education: a mode of governance or a historical journey? Comparative Education, Abingdon, v. 93, n. 4, p. 423-438, 2003. [ Links ]

OLIVEIRA, Dalila Andrade. Nova Gestão Pública e Governos Democrático-Populares: contradições entre a busca da eficiência e a ampliação do direito de educação. Educação & Sociedade , Campinas, v. 36, n. 132, p. 625-646, jul./set. 2015. [ Links ]

OLIVEIRA, Dalila Andrade. O Governo das Escolas e a Nova Gestão Pública. In: LIMA, Licínio; SA, Vírginio. O Governo das Escolas: democracia, controlo e performatividade. Famalicão: Editora Húmus, 2017. P. 61-86. [ Links ]

OLIVEIRA, Dalila Andrade; DUARTE, Alexandre William Barbosa; CLEMENTINO, Ana Maria. A Nova Gestão Pública no Contexto Escolar e os Dilemas dos(as) diretores(as). Revista Brasileira de Política e Administração da Educação, Porto Alegre, v. 33, n. 3, p. 707-726, dez. 2017. Disponível em: <Disponível em: http://seer.ufrgs.br/rbpae/article/view/79303 >. Acesso em: 3 jun. 2018. [ Links ]

OLIVEIRA, Dalila Andrade; JORGE, Tiago da Silva. As Políticas de Avaliação, os Docentes, e a Justiça. Currículo sem Fronteiras, Braga, v. 15, n. 2, p. 346-364, maio/ago. 2015. Disponível em: <Disponível em: http://www.curriculosemfronteiras.org/vol15iss2articles/oliveira-jorge.pdf >. Acesso em: 3 jun. 2018. [ Links ]

PETTERSSON, Daniel. International Knowledge Assessments: an element of national educational steering. 2008. 294 f. Tese (Doutorado em Educação), Uppsala Studies in Education, Uppsala Universitet, Uppsala, 2008. [ Links ]

PETTERSSON, Daniel. Three Narratives: national interpretations of PISA. Knowledge Cultures, Woodside, v. 2, n. 4, p. 172, 2014. [ Links ]

PINI, Gianreto. À Propos de la Théorie des réponses aux items: le cas d’itemsdichotomiques. Genebra: Groupe-Edumétrie, 2012. Disponível em: <Disponível em: https://www.irdp.ch/data/secure/1952/document/TRI_DICHO.pdf >. Acesso em: 9 jun. 2016. [ Links ]

SCHWARTZMAN, Simon. Uses and Abuses of Education Assessment in Brazil. Prospects, Paris, v. 43, n. 3, p. 269-288, 2013. Disponível em: <Disponível em: https://doi.org/10.1007/s11125-013-9275-9 >. Acesso em: 24 fev. 2016. [ Links ]

SOARES, Francisco José; XAVIER, Flavia Pereira. Pressupostos Educacionais e Estatísticos do Ideb. Educação & Sociedade , Campinas, v. 34, n. 124, p. 903-923, 2013. [ Links ]

SOLESBURY, William. Evidence Based Policy: whence it came and where it’s going. ESRC Centre for Evidence Based Policy and Practice, London, p. 1-11, 2001. [ Links ]

SPENLEHAUER, Vincent. Une Approche historique de la notion de politiques publiques. Informations Sociales, Paris, n. 110, p. 34-45, sep. 2003. [ Links ]

STEINER-KHAMSI, Gita. What is Wrong with the “What-Went-Right” Approach in Educational Policy?. European Educational Research Journal , Oxford, v. 12, n. 1, p. 20-33, jan. 2013. [ Links ]

VILLANI, Marialuisa. International Assessment, Standardisation, evidence-based policies et inégalités éducatives. 2017. 120 f. Tese (Mestrado em Sociologia e Estatística) - Master2 recherche Sociologie et Statistique, Sociologie, EHESS/ENSParis, Paris, 2017. [ Links ]

VILLANI, Marialuisa. The Production Cycle of PISA Data in Brazil: the history of data beyond the numbers. Sisyphus Journal of Education, Lisboa, v. 6, n. 3, p. 30-52, 2018. [ Links ]

WÖESSMANN, Ludger. International Evidence on School, Competition, Autonomy and Accountability: a review. Peabody Journal of Education, Nashville, v. 82, n. 2-3, p. 473-497, 2007. [ Links ]

Translated from portuguese by Natalia Santana Revi

Received: July 17, 2018; Accepted: September 26, 2018

Marialuisa Villani is a PhD doctor in Social Systems, Organization and Policies Analysis at Sapienza University of Rome and Ma in Statistic for Social Sciences at EHESS Paris. Her research interests are: The transformation process of young researchers work conditions in Europe; the link between educational assessment programs, statistical data and educational policies in Southern-America. E-mail: marialuisavillani@hotmail.com

Dalila Andrade Oliveira is Professor of Educational Policies (UFMG). She is Researcher and Coordinator of CA-Ed/CNPq. She is Coordinator of Latin-American Network in teacher work Studies (Red Estrado). E-mail: dalilaufmg@yahoo.com.br

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License