Brazil's one laptop per child program: impact evaluation and implementation assessment

Abstracts

Este artigo discute os rumos da política pública de inclusão digital nas escolas com base nos resultados da pesquisa de avaliação de impacto e de processo do Projeto UCA-Total (Um Computador por Aluno) realizada em cinco municípios onde foi implementado no decorrer de 2010 e 2011. A avaliação de impacto, de caráter longitudinal, foi feita através de um survey domiciliar aplicado em duas ondas. A pesquisa de processo associou diversas metodologias qualitativas, como grupos focais, entrevistas e visitas a campo. Limita-se aqui aos resultados derivados da pesquisa de painel, que revelam o baixo nível de aproveitamento do Projeto e um padrão de funcionamento bastante divergente em razão de vantagens e debilidades locais. Seu enorme potencial não foi até hoje completamente aproveitado.

Inclusão digital; Avaliação do programa; Políticas públicas

Este artículo discute los rumbos de la política pública de inclusión digital en las escuelas en base a los resultados de la encuesta de evaluación de impacto y de proceso del Proyecto UCA-Total (Una Computadora por Alumno) realizada en cinco municipios en los que fue implementado en el trascurso de 2010 y 2011. La evaluación de impacto, de carácter longitudinal, se llevó a cabo a través de una encuesta domiciliaria aplicada en dos fases. La encuesta de proceso asoció diversas metodologías cualitativas, como grupos focales, entrevistas y visitas a campo. Nos limitamos aquí a los resultados derivados de la encuesta de panel, que reveló que la ejecución del Proyecto se caracteriza por el bajo nivel de aprovechamiento y por un patrón de funcionamiento bastante divergente en función de ventajas y debilidades locales. Hasta hoy su enorme potencial no fue totalmente aprovechado.

Inclusión Digital; Evaluación de Programas; Políticas Públicas

OTHER ISSUES

Professor and Researcher at the Instituto de Economia of the Universidade Federal do Rio de Janeiro - UFRJ. E-mail: [e-mail]

Professor and Researcher at the Departamento de Ciências Sociais of the Universidade do Estado do Rio de Janeiro - UERJ. E-mail: [e-mail]

ABSTRACT

This article discusses the directions taken by the public policy of scholastic digital inclusion, based on the results of an investigation into the impact of and process employed by Project UCA-Total (One Computer Per Student), implemented in five municipalities over the period 2010-2011. Evaluation of the program's impact - conducted longitudinally - was carried out via a home survey, rolled out in two waves. The investigation of the program's process used a number of qualitative methodologies, such as focus groups, interviews, and trips to the field. This paper focuses on the results derived from the panels, revealing the project to be underperforming, with operating standards that vary considerably from site to site. The program's enormous potential has not yet been fully taken advantage of.

Keywords: Digital Inclusion; Programme Evaluation; Public Policies

THIS ARTICLE SEEKS TO DISCUSS the directions taken by the public policy of digital inclusion in public schools in Brazil. Firstly, it undertakes a periodization of the stages and the design of the program in this area present in federal schools since 2007, lending context to the institutional profile of these initiatives. Next come the results of the evaluation of the process employed by and impact of Project UCA-Total (One Laptop Per Child) implemented in five municipalities over the course of 2010 and 2011: São João da Ponta (Pará), Barra dos Coqueiros (Sergipe), Tiradentes (Minas Gerais), Santa Cecília do Pavão (Paraná), and Terenos (Mato Grosso do Sul).

From the perspective of the program's implementation, one must note that while it is currently being carried out in these five municipalities, it has been marked by underperformance and varying operating standards, reflecting local strengths and weaknesses. The program's enormous potential has not yet been fully taken advantage of.

The impact evaluation, conducted longitudinally, was carried out via a household survey in two waves (T₀ and T₁), with a six-month interval between them, along with a sampling of students and their guardians, representative of the whole range (10,484 students). The investigation into the program's implementation, meanwhile, brought together a number of methodologies, notably qualitative ones, such as focus groups with the agents who were directly affected by intervention (450 teachers), interviews with the various actors involved with implementation, and field visits in order to map the transformations seen in the wake of UCA-Total.

Although the implementation process was marked by noted deficits in the chain of transmission and a severe lack of coordination, it was possible to note relevant and original effects, the result of distributing laptops to students in the selected municipalities. In short, the impact evaluation revealed that the public school students targeted by UCA-Total - particularly those from poor families - discovered computer science and the internet and came to master them, and that schools were the great vector of this dissemination of technological innovation. Public schools play a central role in the promotion of new information and communication technologies in remote, underdeveloped areas. In this article, we will show how these results were attained through the combination of two specific methodologies for analyzing impact and process, and, using the results of this evaluation, discuss how essential changes to the policy of scholastic digital inclusion may make it more effective and introduce innovation as a practice and value into the Brazilian learning process in the public sector.

DIGITAL INCLUSION PROGRAMS IN SCHOOLS

The use of digital technologies within schools, and the challenges this poses, has been a recurring theme in investigations over the past years, in parallel with the adoption of ambitious and fairly onerous public programs aimed at introducing computers and internet access as a pedagogical tool to aid learning. Among the variety of initiatives created to stimulate digital inclusion within schools, two attracted interest and gained ground fast: long-distance learning programs, which for many provide a differential in terms of access to better-quality education in remote areas, or in the case of special target groups; and the One Laptop Per Child Program, which would donate laptops for individual use to primary- and secondary-school students, with the aim of substituting textbooks, as well as the traditional forms of structuring and disseminating information within the classroom. Both question the format of the school and the practice of teaching as they have been known to families, pupils, instructors, administrators, and society at large.

Reassessing school performance, reducing dropout rates, improving teacher training in order to change the ways in which teaching and learning take place, so as to allow children and youths to acquire a new type of knowledge aimed at creative problem-solving with a critical spirit: these are some of the aims cited by those who support the dissemination of information and communication technology (ICT) in schools, as part of the effort to revolutionize teaching and thus renew it. Or, at one extreme, remake it.

The same arguments were used in the United States in 1997 by the members of the independent committee of specialists in science and technology, as they formulated - after two years of reflection and analyses - a set of guidelines addressed to the office of the president.

The teacher remains central to the process, however. As the committee saw it, online interactive systems favor a new modality of learning in which the teacher is able to offer individualized teaching, taking into account the interests, needs, background knowledge, and learning style of each student. Teachers, while indispensable, would have to profoundly change their work process in order to integrate technology into their curricula and get the best work out of each student, without worrying about the class average. Digital technologies thus promise greater efficiency on the teacher's part in the effort to provide a permanent education, one lasting for the rest of the student's life.

This belief in information technology's power to revolutionize teaching methods is, however, by no means a consensus among educators, pedagogues, scientists, and other professionals directly or indirectly involved with education policy and schools. Evaluations based on longitudinal, rigorously conducted surveys show that the use of technology in the classroom presents fairly high costs and often show insignificant, subpar results (GOODWIN, 2011), especially in terms of performance. Many cities that switched over to an "all-digital" model for their schools - even substituting blackboards with large interactive screens -

Here lies the so-called digital enigma. The acquisition of information technology skills certainly allows for a new type of interaction between new generations, avid as they are to innovate, and schools, where everyone suffers from the fatigue of the old methods, the lack of public resources and the dwindling prestige of teachers. That being said, technology per se does not seem to guarantee success in learning.

A recent study by Greaves and collaborators (2010) carried out in 997 American schools, identified nine factors that, when present, seem to contribute to improving performance in schools offering individual laptops to their students. Among these, the three most crucial are:

1. Ensuring uniformity in the process of integrating the technology in all classrooms of the school, at a given site.

2. Giving teachers time for training and time to share experiences and collaborate with their colleagues.

3. Encouraging daily use of technology through online activities so as to promote cooperative learning.

The debate moves on, strongly polarized, although this has not inhibited the creation of similar initiatives across the world and on a large scale. In developing countries, the recent recovery of the economy, tied to the increasing affordability of digital technology, drove the creation of programs such as One Laptop Per Child (OLPC), distributing XO laptops - a model developed by MIT, created with schools in mind and costing around $100 per unit.

DECISIONMAKING, PROGRAM DESIGN, AND PROGRAM PROCESS ASSESSMENT

At the 2005 World Economic Forum, in Davos, then-president Luiz Inácio Lula da Silva was shown a cardboard model of what would be the new laptop destined to revolutionize education on a planetary scale, subverting the traditional paradigms of teaching and learning. At the time, Nicholas Negroponte promised to deliver a functional prototype of the future XO within twelve months if Lula would take on the challenge of setting up a public program for distributing laptops in Brazilian schools.

The challenge was accepted, and in late 2006, UCA-Total was launched in Brazil, the national equivalent to the global OLPC network and an important step in the production of knowledge via the dissemination of access to educational technologies in public schools. At the moment, the first generation of individual laptops in schools, distributed freely, presents a more modest product at an elevated price, in the light of its limited functionality. The same model has been adopted in Uruguay, for example, which introduced Programa Ceibol in 2007 and within three years managed to universalize primary- and secondary-school students' access to computers and the internet via ownership of an XO laptop. Argentina, Paraguay, Peru, Bolivia, and Nicaragua have followed in Uruguay's wake and developed similar projects, adopting the XO. Portugal has also launched a program of infoinclusion and e-schools, providing 1.7 million students with laptops.

From Nepal to India, Afghanistan, the Gaza Strip, Nigeria, Madagascar, and Ghana, promising experiments are multiplying, although in most cases their coverage is reduced and their funding only temporary, when supported by NGOs.

Joining this wave of enthusiasm, which left profound transformations and great hopes in its wake, the Brazilian government adopted Project UCA, meant to provide each public school student and instructor with a personal laptop.

The UCA was planted within the executive branch so as to facilitate decision making and ease of implementation. At the Ministry of Education (MEC), the Special Secretary for Long-Distance Education (SEED) became the direct supervisor for the program, although it was not taken on to the same extent by other areas of the ministry, which remained resistant to the experiment.

The concept for the UCA slowly took on clearer form - and, in the process, its own identity, prizing student autonomy in the learning process: "the student will choose how to learn." This focus had immediate consequences in terms of curriculum design, forcing them to adapt to an open-ended, nonstop educational experience, spreading beyond the walls of the school.

Apart from providing equipment to students and teachers, technological infrastructure and internet access, the federal government took on the task of providing access to new digital media for the continued education of teachers and students. Two educational portals with online pedagogical content became potential tools for this new virtual work and research environment: Portal do Professor (Teacher's Portal) and Portal do Aluno (Student's Portal). The former contains over 9,000 catalogued objects meant to help update the curricula of each subject to the uses of multimedia technology, as well as informing teachers about courses geared towards implementing ICT in the classroom. The second is meant for students' use. In both, users - provided with a password - may access digital resources.

The municipal governments in charge of primary school, were tasked with adapting their schools to this new style of teaching and learning, via the mobilization and activity of their Secretaries of Education. An additional challenge, moreover, would be the physical adaptation of classrooms: replacing seats with armrests with writing desks in order to facilitate laptop use in classrooms, adjusting the electric network to the new demands of intensive electronics use, the installation of special cabinets with plugs for the laptops to be recharged at night, etc. In terms of secondary schools, run by the state, these tasks were to be taken on by the state secretary of education.

In the first phase of Project UCA, pilot programs were brought to five Brazilian cities over the course of 2007: Piraí (Rio de Janeiro), Porto Alegre (Rio Grande do Sul), Palmas (Tocantins), Brasília (Distrito Federal), and São Paulo (São Paulo). At each site, a school was chosen to host the experiment, and all the students at the institution were given a prototype of an educational laptop, provided at no cost by its producers. A second phase was begun in 2010, this time expanding the program to 300 cities across the country and based on the first phase (operating in just one school per municipality). Approximately 150,000 UCA laptops were distributed, in total.

Finally, authorities opted for a third phase, dubbed UCA-Total - the subject of our evaluation - in which six municipalities from across the country were chosen, this time affecting all the schools (local and state) in each municipality, as well as all their students and teachers.

One observation, to begin with, speaks to the fact that Phase 3 of Project UCA - or UCA-Total - did not provide for a detailed, clear-sighted review of the evaluations undertaken in preceding phases. Not only was there no systematized document about the project's antecedents, which go back to 2005, but neither are the products developed as part of these experiments (referred to generically as Programa Nacional Um Computador Por Aluno, within the Política Nacional de Inclusão Digital) available under a single electronic address or in a single virtual library.

Moreover, detailed reports regarding the first phase - such as those drawn up by the World Bank - were disregarded and considered extremely pessimistic in their interpretation of the results, as they focused on the problems and failures registered when the first version of Project UCA was launched.

Even identifying the agencies and entities involved with UCA-Total was a challenge, as the project lacks supporting documents that might allow for one to trace precisely the process of formulating and decisionmaking in tailoring the Program.

It is true, however, that the Project's philosophy was clearly defined: to promote a radical change in the conception of public-school curricula via individual access to technology, so that "each protagonist of this process may be the coauthor in the construction of his or her education in the school - an open education, with multiple authors." This would thus be a "process with no owner," in theory, not being top-down, and which ought to be interpreted as a collective construction. Among these protagonists, emphasis is placed on students; it is up to them to choose how to learn. The teacher becomes yet another mediator in the learning process, rather than the only agent of learning.

This vision is strongly supported and driven by a group of prestigious educators in Brazil, who advocate the intensive use of ICT in schools to revolutionize public education and improve its quality.

The Project's direct link with the office of the president would eventually generate conflicts with other areas of government. The program was supported by the MEC and the National Network for Teaching and Research (RNP), and the Ministry of Science, Technology, and Innovation (MCTI), to provide the wifi infrastructure that would be necessary in each municipality. At the start, a standard network was developed that was meant to be replicated in each site. At MEC, the Secretary for Long-Distance Education teamed up with the nucleus linked to the office of the president to ensure smooth coordination of the project. This same group operated as an interlocutor with local and state secretaries of education, and with institutions of higher education, these directly responsible for training teachers at each site in the use of this new digital content.

The path to implementation was long, and some municipalities had to wait years for the Project to arrive (from 2007 to 2010). In theory, after the selection of a given municipality, the first stage of the UCA-Total implementation process would be the installation of connectivity technology the city and its schools. The second phase would bring a transformation of physical infrastructure - adapting schools to receive individual laptops in each classroom and installing a wifi network. The third stage would train teachers in using the UCA laptops, both as individual tools and a way of connecting to the universe of digital education, with all its online and interactive content. Finally, the laptops would be distributed to students.

Five training courses were arranged to attend to teachers and administrators. Each would take 180 hours - 40 hours in the classroom, per se, and the rest would be up to the initiative of the teacher and/or administrator. The instructors at the Institutes of Higher Education of each region would be tasked with providing this training and supervising their colleagues' progress in adopting ICT at the state and local levels.

There was no "acting out," a sort of simulation that would have allowed them to learn how to implement the Project simultaneously in five municipalities (ROSSI et al, 2004). Nor was there any monitoring strategy in place to systematically oversee the Project's progress or measure how closely what was implemented hewed to what had been planned, in step with the previously established timeline. The only date on a timeline of activities was the UCA-Total launch in the city, meant to lend the program visibility and guarantee its institutionalization.

The implementation model adopted in practice was that which the evaluation literature refers to as "unstandardized or uncontrolled intervention" (ROSSI et al, 2004), in which implementation methods (coverage, personnel qualifications, content) vary from site to site, which may hinder the generalization of the impact of a given model.

Another mark of the implementation model used was that it did not produce records for the program, meant to systematize administrative information and essential for any sort of follow-up on its execution and posterior corrections, should they be deemed necessary.

Another noteworthy point is the absence of a nucleus constituted by all five municipalities, where experiences, difficulties, and solutions might be shared between administrations, leading to an effective learning process among local administrators over the course of the implementation of this digital inclusion policy. Instead, the municipalities conducted their work in mutual isolation; administrators never even exchanged email addresses. Hence, coordination operated top-down (from the nucleus at the office of the president to local administrators), with few communication channels and no horizontal connections. Each municipality had a UCA-Total coordinator, the privileged interlocutor between Brasília and state and municipal Secretaries of Education, but these figures' interventions were largely ineffective. Direct administration of the program lay in the hands of the Secretaries.

In terms of the Project's implementation, the tasks left to the municipalities were unfulfilled in some cases. The problem of the cabinets for storage and recharging laptops is an illustrative case. Due to high prices,

In field visits, a number of problems were detected having to do with the misuse and improper conservation of laptops by some students, but it is not yet possible to predict the long-term effects of this.

What the case of the cabinets demonstrated was the difficulty in implementing initiatives that require cooperation between the municipal and state levels, and the absence of monitoring by the overall administration of the Project, which left it wholly up to the municipalities to abide by the stipulated standards. What prevailed was not quite the art of muddling through (LINDBLOM, 1979); rather, spontaneity and a lack of coordination prevailed.

In none of the five municipalities observed did state and municipal government coordinate their actions or find collective solutions. Table 1 summarizes the situation in late 2011, as regards the protocol for implementing Project UCA-Total's infrastructure.

In terms of implementation and connectivity infrastructure, the situation varies considerably across sites, especially when it comes to adapting classrooms (writing desks and charging cabinets). On one hand, the federal government's responsibilities in implementing a public broadband network were partially fulfilled: the logical network was installed with routers in strategic places within a number of classrooms in order to provide better wireless access, as may be seen in Table 1. On the other hand, tests carried out with the UCA laptop model showed that the signal was frequently null, preventing a network connection. When it did work, the flow of data was subpar; in general, rural areas were not adequately covered, significantly hobbling UCA's viability in non-urban zones.

The solution arrived at by a number of schools individually was to sign up with private broadband providers (Velox, for one) in order to guarantee internet access. This solution was viable for schools in urban areas. In some cases, the service was paid for by the teachers themselves, on an informal basis.

One thus sees that, in practice, implementation did not manage to attain the standards of the model in all cases. Two municipalities in particular, São João da Ponta and Barra dos Coqueiros, encountered significant difficulties; the prerequisites defined for the classroom operation of UCA-Total were practically ignored, while this did not lead the general administration of the Project, in its regular monitoring of the progress of activities, to demand that stipulated technical standards be met.

Operation of the Project was left up to the discretion of municipal administrators - or, in many cases, school principals. Adaptation of physical spaces, equipment, and support for installation of the logical network were dealt with at the municipal level, and their not having been completed did not trigger any course-correction on the part of the administration. The focus remained on the specific abilities of each school, to the detriment of a standard model.

In other words, when it came to decentralized activities, the Project took on different forms at each site. This was not always the product of specific choices, but rather of a lack of choices. And the absence of monitoring and coordination on the part of federal administrators wound up generating wildly varying dynamics between municipalities, which inevitably had an impact on the degree to which students and teachers were able to take advantage of this new tool. Coordination at the federal level was not able to fill the notorious gaps in implementation that marked the program's progress.

Another important point in understanding the implementation process for UCA-Total has to do with teacher training, since teachers were the key players in terms of disseminating digital education, as well as its success in the middle and long term. Training was envisioned in terms of five modules. The first, 40 hours of in-class training, was positively evaluated by teachers. The following modules were designed to be fully completed long distance, using the Project laptop and access to the internet. This is where the problems began. Not all schools had access to the web, which severely compromised the quality of training. In the second phase of investigation, eight out of ten teachers who underwent training reported difficulties in using the UCA laptop in the classroom, an indicator that the course was not sufficient preparation for instructors to be able to master the tool in their day-to-day work.

Two research methodologies were employed to evaluate the effects of instructor training: interviews with representatives from institutes of higher education selected to develop and supervise the training of teachers from the municipal and state public schools involved in the Project; and the application of a survey,

A minority of instructors (less than a third) reported having no problems in their use of the UCA laptop. Despite this, 72% said they believed that the use of UCA-Total would facilitate their work. More than 70% said they had some degree of difficulty in using the laptop; however, the frequency with which UCA laptops were used in the classroom, a year after the start of the Project, reveals the persistent underutilization of digital pedagogy and equipment, as illustrated by Graphic 1.

Indeed, one sees that instructors' use of the tool was not completely assimilated into classroom activities. Many of them experienced great difficulties in handling the laptop. And there remain those who resist incorporating the change into their work routine. Overall, teachers recognize that they have more difficulties than students in using the project laptops.

It also becomes apparent that the plan for the Project underestimated primary and secondary-school teachers' difficulties in adopting ITC in relatively poor communities, which led to the underutilization of UCA in the classroom. A year after the program's launch, nearly 20% of instructors reported that they had not yet used the new tools in their classes, and 22% said that they had received no training.

IMPACT EVALUATION

Sound evaluations of public programs and policies require knowledge of the principal characteristics of their target populations. With the aim of better understanding the world of the students participating in Project UCA-Total in the five pilot municipalities, the specificities and vulnerabilities of their situation, a longitudinal survey was undertaken at two points (two waves), in December 2010 and June 2011. The goal of this quantitative assessment, using a probability sample, was to gauge the impact and estimate the effects and results of Project UCA-Total's intervention in terms of the homes of the students provided with laptops.

The methodology chosen was the application of a survey in the form of a panel - before (T₀) and after (T₁) the laptops were distributed - alongside a probability sample stratified by school, year, and class, representative of the population provided with UCA laptops: that is, the students from municipal and state public schools in the five municipalities and their guardians. The allocation of the resulting sample presented in Chart 1. Values were rounded up or down to facilitate the allocation of interviewers. This being a panel, the same students and guardians were interviewed in both waves.

Moreover, researchers visited the cities to gauge changes and collect secondary information that might allow them to capture effects derived from the arrival of UCA-Total in the community as a whole, and in terms of local development. However, measuring the program's impact on scholastic performance and in effective learning, given the new pedagogical content disseminated in a digital environment, was not the subject of this study. The study focused much more on how the effects of Project UCA-Total spilled over outside the classroom.

The questionnaire used was divided into seven modules, two of them directed to the students targeted by Project UCA-Total (modules 6 and 7), one to the head of household (guardian)(module 5), and the rest (modules 1, 2, 3, and 4) geared towards collecting information about general aspects of students' homes and families, including information about family members, etc. Before the implementation of the T₀, the questionnaire was tested out using a pilot version on smart phones, so as to check for consistency.

And so, using the samples from each municipality, the survey reached a total of 10,484 students ages 5 and up. Although the minimum age for entering primary school in Brazil is now 6, we opted to work with a more flexible lower limit, considering the possibility that some students might enter at a younger age, depending on their birthdays.

In terms of racial or color distribution, 68% of students declared themselves black or brown, 29% white, and 3% yellow or indigenous, a breakdown similar to the distribution of the Brazilian rural population, according to the latest IBGE Census.

Distribution by age, meanwhile, reveals that the majority of students (42%) are between 11 and 15 years old. Those from ages 5 to 10 make up slightly more than a third (35.6%); and those over age 16 constitute 22.4%. The greater concentration of students between ages 5 and 15 in our analysis may be explained by the fact that the majority of schools in the five affected municipalities only offer primary school.

Graphic 2 presents the distribution by year/grade of the total estimate of students in our analysis. The third and sixth years see the greatest concentration of the students at hand (1,273 and 1,255, respectively). In all, there were 8,552 students in primary school (84.9%) and 1,443 in secondary school (14.3%).

In impact evaluation studies, it is common to seek to divide sample populations into an experimental group (those participating in the policy at hand) and a control group (those not participating in the policy at hand). This methodology seeks to infer causality stemming from an intervention, isolating its effect among many others that may have transformed a given situation, at two or more points of observation over time. In addition to seeking to identify the cause of a transformation or a stable situation, associating it to the intervention at hand (a public policy), all impact evaluations take a comparative approach: either comparing the same population before and after an intervention, or establishing a control group with the same characteristics as the experimental group and analyzing how they evolve in parallel.

In order to conclude that a phenomenon is the effect of a given cause, all the other possible causes of that effect must be discarded (STUART MILL apud ROSSI et al., 2004). Of course, causality is inferred (probabilistic) and not definitively proved (deterministic): the presence of a cause increases the probability of obtaining an effect, but does not guarantee it.

Project UCA-Total is an all-encompassing program; thus, within a given municipality, it is not possible to group students into experimental and control groups once the policy is set in motion. In order to conduct a comparative study, the idea was to carry out the first interview (T₀) before the laptops were distributed to students, and the second interview (T₁) after the intervention. This would allow for an analysis of individual differences in outcomes of interest in order to evaluate changes that may have been triggered by the intervention, in this case comparing the same population before and after the intervention in question.

However, in some of the municipalities in the pilot, laptops were given to students even before students and administrators had been fully trained in using the new equipment in the classroom - before T_0, that is. Moreover, given inevitable delays, some students still had not received their laptops by T_1. This actually benefited the survey, making it possible to compare two groups: the experimental group, with the students who did receive their laptops on time, and the control, with those who still had not received equipment.

USE OF THE DIFFERENCE-IN-DIFFERENCES ESTIMATOR

This methodology, widely used in analyses such as these, basically compares the changes observed in a given variable (y) in the control group before and after the implementation of the public policy with the changes of this variable in the experimental group. First of all, the simplest case of this methodology is considered, with just two points in time - T₀ and T₁ - and two groups - a control and an experimental group. For this case, one would use the following model:

y_ij = β₀ + δ₀ D_T + β₁ D_Uca + δ₁ D_T D_Uca + e_ij

Here, a dummy point in time for T₁ captures the longitudinal effect, post-intervention, on the factors that are equally related to y in each group; the dummy indicates the experimental group that reveals differences between the two before the intervention; the difference-in-differences estimator is the effect, which expresses the impact of the project on the variable of interest, y, expressing the average effect on the experimental side.

This approach is normally used with pseudo-panel data where the unities within the groups are not the same; in general, moreover, the ordinary least squares (OLS) method is applied, without looking to the possible correlation typically present in panel data. However, here OLS was applied to the data from the longitudinal survey with analysis conducted on an individual basis, and, in most cases, with students acting as their own controls. Furthermore, we used stronger estimation methods than OLS, which control for the temporal correlation for each unit of analysis, producing more robust standard errors for the estimated coefficients.

One important point to be observed, and a differential, was that we considered two possible experimental groups: students who received the UCA laptop but did not have permission to take it home; and students who received the laptop and brought it home with them. The control group remained the same - the students who did not receive the UCA laptop. This significant distinction between beneficiaries was observed in exploratory analyses carried out before this modeling exercise. We observed that, despite teachers' allegations, the laptop as a personal device (that is, allowing students to use the laptops at home) had a much greater impact than when use was restricted to schools - which, in many cases, did not even register a significant impact (on par with the control group, which had not yet received the UCA laptop).

SOME RESULTS OF THE IMPACT EVALUATION

The first true affirmation from this evaluation is that there is an impact when students can bring their UCA laptops back from school and use them at home. Using the laptop exclusively at school, without being able to make use of it as a personal device for individual and home use, is a model of intervention whose impact appears equivalent to simply not possessing a laptop. Despite the opinions of many teachers hesitant to authorize home use of UCA laptops, authorizing true possession of the equipment - and allowing students to use it freely - is the best way to foment digital inclusion and spark interest in learning.

Firstly, we investigated the program's impact on a number of the characteristics of the students affected. We evaluated Project UCA's impact on areas such as: liking Portuguese, history, and geography classes; liking mathematics, computer lab, and researching on the internet. Also investigated was UCA's impact on the availability of functioning computer labs in students' schools, attendance in classes offered in the computer lab and the use of these labs outside classes.

The UCA experiment appears to galvanize use of computer labs, as may be seen in Chart 2. We know that in the majority of schools were they were implemented, these labs were underutilized. Hence, the good news is that UCA had a positive impact in terms of more intensive use of computer labs, even as far as their utilization for class time. These impacts were observed both in the group that limited its UCA laptop use to school and in the group that was allowed to take the laptops home.

The impact on the use of computer labs outside class time, meanwhile, was not significant for those who were allowed to take their laptops home, and negative for those who did not take their laptops home. This result is very probably the product of the wifi network covering the entire school, leading students to stay in the building or nearby in order to catch the wireless signal and work online, rather than using the PCs in the computer lab.

In terms of other impacts, we saw that UCA had a positive impact in terms of increasing the number of books read, an effect that decreased in magnitude amongst the students who took their laptops home. In terms of an impact on students completing their homework directly on the computer, this was only significant in the case of the students who took their laptops home; thus, there was no significant difference between those who did not receive the laptop and those who were simply not allowed to take them home.

Assuming that these students are poor children and many do not have access to a computer besides the UCA laptop, an impact on Internet use as a research source in homework completion should only be observed for those with access to a computer. This was verified in the existence of a positive impact on Internet use as a research source for those who took their laptops home and a negative impact for those who, although provided with UCA laptops, were not authorized to bring them back from school.

In a descriptive and explanatory analysis with premodeling, it was seen that without UCA, students living in poor or extremely poor families (with per capita familial monthly income, RDPC, below R$140) would normally have no computer, nor regular access to broadband at home. More than 90% of students with per capita family income equal to or below the poverty line reported that they had access to just one computer at home - the UCA laptop, of course. Individual possession of a laptop is certainly an opportunity created by this policy for the most needy, but also for the half of students whose families are not poor. All benefit, in this case.

For the poorest, however, the panel research revealed that this was the only way for them to enter into the information age and the digital world. For example, a significant impact concentrated solely in the subgroup of the poorest occurred in the following cases: having classes in the computer lab and using computers to do homework (see Chart 3). In the cases of the students from non-poor families, the program's impact came in terms of two variables: reading books and studying at home. For the poorest students, using the UCA laptop, even without taking it home, had a significant and positive impact on liking having computer classes and researching on the Internet. For the less poor students, UCA had a positive impact on liking having computer classes, but only for those students who were authorized to take the laptops home.

The guardians for the students targeted by UCA appeared considerably less sensitive to the direct effects of the program. No impact was seen that might provoke a change in their pattern of digital inclusion, except for a slight uptick in their computer use. Panel research thus confirms that UCA's impact is primarily focused on students - children and youths - and not their relatives.

However, the study also shows that Project UCA made for an effective consolidation of the computer-related supply chain in the municipalities served by the program, as well as driving the dissemination of such equipment and accessories in the homes of those students consulted (Chart 4). These, then would be effects promoting the advance of ICT in remote and less developed areas, less prone to innovation.

Breaking this analysis down into two subgroups - i.e., the poor, those with PCFIs less than or equal to R$140, and the non-poor, those with PCFI greater than R$140 - one sees that, in general, UCA-Total has a more striking effect on the group above the poverty line (see Chart 5). Equally significant effects were not noted within each subgroup, whether in terms of services used or equipment owned. However, the program's impact on book possession - while on a smaller scale - is also significant, and greater (more negative) for the poorest group, which also saw a greater impact on their electricity expenses.

These two observations suggest the necessity for a number of complementary interventions to work against two worrisome impacts, particularly in terms of the groups with less purchasing power - those who ought to be favored the most heavily by digital education, Internet access and mastery of ICT, in terms of reducing socioeconomic and cultural gaps. While Internet access does seem to undercut the reading of traditional, paper books for new generations, and while tablets have definitively opened up alternatives in the form of e-books, online acquisition and access to virtual libraries, it would be wise to develop immediately some form of reading program for UCA-Total's target audience to incentivize the download of works and foment the discovery of great literature, so that Internet access is not limited to the consumption of fragmented reports, exchanges on social networks, and more generic content.

Therefore, in terms of household characteristics, one sees that the opportunities generated by possession of the UCA laptop are not the same for all. Other programs must be formulated as well, promoting opportunities more equally distributed across the target population, so that no specific income bracket is favored in detriment of those less able to take full advantage of technological innovations.

We also confirmed that UCA-Total had some impact on a number of characteristics of individuals living in affected households, but expressed as household characteristics (see Chart 6). To this end, we tallied the following elements within households: children ages 5-9 able to read, and people using the computer, the Internet, or both. Having access to the UCA-Total laptop was expressed in terms of a positive impact on the literacy of the children living in the same household as the benefited student. Hence, the program also affected the very students benefited by UCA in the initial stages of learning letters and numbers.

Proceeding to a subdivision of the target population by income brackets, we see that the UCA laptop's effect on the literacy of children ages 5 to 9 is significant and much more intense in the group with per capita monthly familial income above R$140. This results confirms the previous observation, that income determines a more effective appropriation of the potential use and learning brought by the UCA laptop, and that this differential - unfavorable for the poorest - ought to be corrected via complementary interventions designed to that end.

Motivated by this discovery, we came to study the influence of the UCA laptop on very young children's ability to read and write (see Chart 7). Initially, we sought to capture the UCA laptop's impact on 6- and 7-year-olds'

This is a striking result. Giving a laptop to a child of 6, just as he or she begins learning to read and write, has a highly positive impact, increasing his or her propensity to become literate at this age. The data from the first wave showed that 48.1% of 6-year-old students reported not knowing how to read or write a simple note, a percentage that tends to fall with greater and better interventions on the part of UCA.

FINAL REMARKS

In general, Project UCA-Total had a direct impact in terms of making better use of the technological infrastructure already available in schools. The impact assessment showed that where the UCA laptops were distributed, computer labs - inevitably underutilized, when not completely abandoned - saw their use increase. It is hoped that with this development, digital education may take on a more central role in schools and teaching in general, multiplying access points to the web and making more digital resources available, in a variety of formats.

However, the impact evaluation demonstrated that, despite providing universal access and apparently providing equal opportunities for all students, the non-poor benefit the most from this intervention. We confirmed that the beneficial effect from Project UCA was greater for non-poor students than for those living below the poverty line. Thus, one may assume that, if complementary initiatives are not created to monitor how ICT is appropriated by the neediest children, it is likely that opportunity gaps will remain - the same gaps that a program such as UCA ought to eliminate or at least attenuate.

Another important result confirmed is that Internet access remains extremely limited, and that only some students (21.7% in wave 2, as opposed to 17.6% in T₀) living in households with increased purchasing power enjoy paid broadband in their homes. The rest saw no positive variation between the two waves in terms of residential connectivity. Poor signal quality penalizes the children living in poorer homes who cannot afford to pay for service from private providers. The Brazilian National Broadband Program - if it does come to fruition - may contribute significantly to attenuating these restrictions and effectively generalizing the full use of ICT, creating opportunities and expanding horizons. But the service's minimum cost still indicates a high monthly fee.

Finally, one might note that the panel - although revealing valid and extremely interesting results - took place over a short period, with just six months between the two surveys. New waves, if conducted at regular intervals of six months over the course of two years, might be able to capture impacts from the program that demand a greater period of maturation and, in this initial experiment, were not seen to be significant. Similarly, if the network infrastructure had been satisfactory and the Digital Cities Program

Without a doubt, there was a learning process as to what ICT is and how to process digital inclusion in schools. One sees, however, that costs remain elevated and effects still fall short of expectations. The network infrastructure installed in schools and cities does not meet the standards of the Project, and, although this became obvious immediately, there were no adjustments on the part of the general administration to promote greater efficiency and coverage in terms of connectivity.

A lack of coordination persisted throughout the process, leading the difficulties inherent in taking action at the municipal level to paralyze the implementation process, leading it to advance at very different speeds from site to site. The lack of communication between overall coordinators and local agents, especially after January 2011, when the presidency changed hands, led to discontinuity and inefficiencies that might otherwise have been avoided, as the administration change was taken into account in the Project's planning.

The pedagogical use of ICT in the classroom is still halting, compared with the discoveries that children and youths make outside the school walls. For UCA to truly have an impact on the digital inclusion of children and youth in primary and secondary school, they should be able to take the UCA laptop home with them and make use of it there. When the UCA laptop's use is restricted to schools and the equipment is stored there, the intervention has no effect. Each student should have the rights to personal use and possession of his or her UCA laptop. With this in mind, the law will have to be modified that currently classifies the laptop as a capital good, not a consumer good. Another interesting result was the realization that the UCA laptop has quite positive effects at the very start of the literacy process, increasing 6-year-olds' propensity to learn to read and write.

Needless to say, schools continue serving as vectors for the transformation of local society, and for access to new technologies - as confirmed in this analysis. They lack, however, the means and resources to carry out this role more effectively. It is worth recalling that classroom use of the UCA laptops remains incipient, and the existing applications with new digital educational content have been widely neglected by students and professors, albeit unwittingly. Teachers as well as students should be given the chance to make the most of the opportunity that UCA represents. This study has shown that while students' self-esteem was raised with the acquisition of a computer, many teachers were shaken, some feeling diminished in the face of this novelty which not all had managed to master. In this vein, they need to be targeted by specific and permanent interventions moving toward the progressive expansion of digital skills, ones that take into account the realities in which they act and the fundamental role they fulfill in the lives of their students.

To paraphrase Steve Jobs, if what matters is making innovation lend wings to imagination, then it is very possible that a program such as UCA-Total may only reveal its full transformative potential farther down the road, when its target population - despite a number of lacunas in the program's implementation - has acquired enough background to turn problems into footholds and grow through innovation.

REFERENCES

• GOODWIN, Bryan. One-to-One Laptop Programs are no silver bullet. Teaching Screenagers, v. 68, n. 5, p. 78-79, 2011. Disponível em: <www.ascd.org/publications/educational_leadership/feb11/vol68/num05/One-to-One_Laptop_Programs_Are_No_Silver_Bullet.aspx>. Acesso em: set. 2012.
• GREAVES, Thomas; HAYES, Jeanne; WILSON, Leslie et al. Project RED key findings Shelton, CT: MDR. One-to-One Institute, 2010. Disponível em: <www.one-to-oneinstitute.org/NewsDetail.aspx?id=85>. Acesso em: set. 2012.
• HARGROVE, Erwin C. The missing link Washington. DC: The Urban Institute, 1975.
• HILL, Michael. The policy process in the modern State 3^th ed. London: Pearson, Prentice Hall, 2004.
• LAVINAS, Lena (Coord.). Avaliação de impacto do Projeto Uca-Total (um computador por aluno): relatório final da pesquisa. Rio de Janeiro, nov. 2011. 202 p.
• LAVINAS, Lena; COBO, Barbara. Políticas sociais universais e incondicionais: há chances reais de sua adoção na América Latina? In: CONGRESSO INTERNACIONAL DA REDE MUNDIAL DE RENDA BÁSICA, 13., 30 jun.-2 jul. 2010, FEA/USP, São Paulo. Mimeo.
• LINDBLOM, Charles Edward. Still muddling, not yet through. Public Administration Review, n. 39, p. 517-525, 1979.
• MARTÍNEZ, Ana Laura; ALONSO, Serrana; DÍAS, Diego. Monitoreo y evaluación de impacto social del Plan Ceibal: metodología y primeros resultados a nível nacional, 2009.
• ROSSI, Peter H.; LIPSEY, Mark W.; FREEMAN, Howard E. Evaluation: a systemic approch. 27^th ed. London: Sage, 2004.
• SELLTIZ, Claire et al. Métodos de pesquisa nas relações sociais São Paulo: Herder/Edusp, 1967.
• WOOLDRIDGE, Jeffrey M. Econometric analysis of cross section and panel data Cambridge, MA: MIT Press, 2002.

Brazil's one laptop per child program: impact evaluation and implementation assessment

Lena Lavinas^I; Alinne Veiga^II

¹

While lacking strong evidence on the effects and impacts of technology in the classroom, they supported its use in American schools to facilitate learning on any subject and in any area. In this committee's understanding, beyond combatting digital illiteracy, this was an effort to introduce computers and connectivity into the classroom so as to significantly improve the quality of the country's education and prepare its youths to act in a global, integrated, and highly competitive economy over the course of the 21

^st century. One sentence from the report passed on to the president indicates the thrust of the changes foreseen with these new technologies: "The student assumes a central role as the active architect of his or her own knowledge and skills, rather than passively absorbing information proffered by the teacher."

²

did not see an improvement in their national rankings and their average grades remained stagnant, confirming underuse of this equipment and a reduced scope for these networks, given their extraordinary potential. Some say that the impact of information technology in the classroom may only be correctly evaluated via the creation of new methodologies, seeing as the standardized score tests are not able to gauge students' sociability, their technological development, their command of digital language, etc.

³

This visionary project by Nicholas Negroponte and Yves Behar sparked hopes that it might reduce truancy and dropout rates, promising to advance the universalization of primary and secondary education in a number of countries, particularly emerging nations, for which correcting and overcoming the flaws in the educational system is a

sine qua non for a sustained path of economic development with social inclusion and more equity in terms of opportunities.

⁴

One of the first effects of this decision was the creation of the first GT-UCA at the Universidade de São Paulo (USP) in 2006, bringing together a number of scholars who began designing the Project, its goals, and how to go about implementing it.

⁵

This group was comprised of Tiradentes (Minas Gerais in the Southeast), Santa Cecília do Pavão (Paraná, in Southern Brazil), Terenos (Mato Grosso do Sul, in the Center-West), Barra dos Coqueiros (Sergipe, in the Northeast), São João da Ponta (Pará in the Northern), and Caetés (Pernambuco, also in the Northeast),

⁶

for a total of just over 10,000 primary- and secondary-school students.

⁷

a lack of standardization, and a lack of clarity as to who was responsible for acquiring the equipment, each city solved the problem differently. In Terenos, for example, the municipal Secretary of Education chose to adapt existing cabinets, while the state secretary opted to buy new cabinets. In Tiradentes, it was the opposite: all the municipal schools received new cabinets for computers, while the only state school in the municipality could not manage to acquire the equipment. In Santa Cecília do Pavão and São João da Ponta, no school obtained adapted cabinets, and the task of recharging the laptops was left to students, who had to do this at home - the responsibility, that is, was shifted over to students' families. In Barra dos Coqueiros, all the state schools received new cabinets, purchased by the state Secretary of Education, but no municipal school was benefited. It is important to note that the storage of these laptops has a direct impact on the conservation and useful life of the equipment.

⁸

given to all the professors in the public schools affected by UCA, in order to gauge their real knowledge and use of ICT. This survey was rolled out in two waves, the first before the initial phase of training and receipt of the UCA laptop, and the second after the initial training and use of the laptop.

⁹

Hence, experiments in public schools provide excellent representativeness in terms of color and avoid bias on this count.

¹⁰

On the other hand, UCA-Total had a negative impact on the already-low number of books available in each household, which continued to decline. However, a new form of access to knowledge and information made itself present: having the UCA laptop in students’ homes had a positive impact on the use of the Internet as the daily source of information most consulted by family members, thus increasing electrical costs.

¹¹

tendency to learn to read and write, thus investigating the dynamic of this effect. A positive and dynamic effect was seen on the tendency to be able to read, a greater impact in the case of the less poor students. Even when the age range is increased, from ages 6 to 9, one could observe that UCA displayed a highly significant impact on the tendency to know how to read - as expected, an impact that changes over time, increasing even further at T

₁ and for the richest.

¹²

¹³

had worked and provided quality broadband, it would have been possible to detect other impacts or register those already identified on a larger scale.

Publication Dates

• Publication in this collection
  22 Nov 2013
• Date of issue
  Aug 2013

History

• Received
  May 2013
• Accepted
  June 2013