Better Incentives, Better Marks: A Synthetic Control Evaluation of the Educational Policies in Ceará, Brazil

Abstract

This article evaluates the effects of two educational policies implemented in the Brazilian state of Ceará. The first was a tax incentive (TI) for mayors to improve municipal education. Under this policy, municipal tax transfers were conditioned on educational achievement. The second was a program to offer educational technical assistance (TA) to municipalities. The impact of these policies was estimated by employing the synthetic control method to create a synthetic Ceará not affected by TI and TA. When the two policies were combined, the results were consistent with a 12 percent increase in Portuguese test scores in primary education and a 6.5 percent increase in lower secondary education. There were similar increases in mathematics test scores; however, these were not statistically significant. This study also investigates the impact of educational interventions on upper secondary schools, which, despite not being directly affected by the new policies, received better-prepared students from lower secondary schools. The findings show no effect on this level of education, highlighting the need for debate on how to extend the benefits of educational policies to upper secondary schools, as well as to other Brazilian states. This research is the first to analyze the impacts of the policies in Ceará on primary, lower secondary, and upper secondary schools using data from 1995 to 2019.

Educational policy; incentives; synthetic control method; student performance

Inequality in education produces adverse social and economic outcomes (HECKMAN, 2011). The Brazilian educational system mirrors the profound inequalities of the country. Half of 15-year-old Brazilians do not achieve the minimum level of reading proficiency expected by the end of secondary education, according to the Organization for Economic Cooperation and Development (2019). While most private schools offer a reasonable level of education, only around 10 percent of students can afford them (INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS ANÍSIO TEIXEIRA, 2020b). Most Brazilians thus rely on the public education system, which faces several difficulties in providing high quality schooling.

Over the course of the last decade, youths in Ceará, a state in northeastern Brazil, have experienced substantial improvements in mathematics and Portuguese test scores, according to the ‘Sistema de Avaliação da Educação Básica’ (Saeb), the national evaluation system for primary and secondary schools. Ceará’s educational performance has stood out in comparison with neighboring states, and even in comparison with the wealthier states of Brazil’s southern half. During the period between 2007 and 2019, the state achieved the nation’s largest increase in the educational quality index, Ideb (LOUREIRO et al., 2020).

Ceará took advantage of the autonomy granted to states by the Brazilian Federal Constitution to implement two new educational policies in primary and lower secondary schools. The first consists of a tax incentive (TI) to reward municipalities that achieve good results in education. Under this policy, the better the school district’s performance, the larger the share of tax revenues the municipality receives. The second consists of the ‘Programa Alfabetização na Idade Certa’ (PAIC), a new model of management and collaboration between the state government and municipalities to ensure high standards of literacy and educational quality. The program offered technical and financial assistance to municipalities, oversaw the training of bureaucratic and pedagogical staff, and engaged in the provision of teaching materials (BONAMINO et al., 2019; SUMIYA et al., 2017). Henceforth, PAIC and technical assistance (TA) will be used interchangeably.

The approach adopted by Ceará is in accordance with the literature emphasizing the relationship between incentives for the actors involved in the educational process and student achievement (BISHOP and WOESSMANN, 2004; BRANDÃO, 2014; CARNEIRO and IRFFI, 2018; LAVY, 2009; MBITI et al., 2019; MURALIDHARAN and SUNDARARAMAN, 2011). Moreover, there is empirical evidence that the actions promoted by PAIC are associated with improvements in learning (ANGRIST and LAVY, 2001; BRESSOUX et al., 2009; FREDRIKSEN et al., 2015; McEWAN, 2015; FUJE and TANDON, 2015).

Considering this literature into account, I assume that the policies implemented in Ceará increased the scores of local students in mathematics and Portuguese tests. Furthermore, I assume that, although not directly affected by the new policies, upper secondary schools also experienced improvements because they received better-prepared students from lower levels schools.

To gauge the effect of the interventions in Ceará, I employ the synthetic control method (henceforth referred to as SCM). In the context of this investigation, SCM is an algorithm that selects a set of Brazilian states not affected by the educational interventions to create a control unit. Each selected control state contributes to the synthetic control unit according to a specific weight. Simply put, SCM estimates a synthetic Ceará whose performance in education is a weighted average of the performance of a set of chosen

control states. This method provides transparency and a data-driven tool to select an adequate control unit (ABADIE, 2021).

When TI is combined with TA, the findings are consistent with increases of 12 and 6.5 percent in Portuguese test scores in primary and lower secondary schools, respectively. Regarding mathematics, the effects were similar, but not statistically significant. There was no evidence of impact of the new policies on upper secondary education in Ceará.

These findings are in line with the literature highlighting the positive impact of technical assistance and incentives on educational outcomes (ANGRIST and LAVY, 2001; BRANDÃO, 2014; BRESSOUX et al., 2009; CARNEIRO and IFFI, 2018; FREDRIKSEN et al., 2015; FUJE and TANDON, 2015; LAUTHARTE et al., 2021; McEWAN, 2015). They also seem to support the model of educational production proposed by Bishop and Woessmann (2004), which links higher political priority for education with better student performance.

This article is organized as follows. First, I discuss the interventions implemented in Ceará and analyze the existing literature on education. Then, I present my methodology, along with information on data sources and summary statistics. In the following section, I present, analyze, and contextualize my results. In the final section, I highlight my main conclusions.

The new educational policies devised in Ceará

Ceará is located in the northeastern region of Brazil and had an estimated population of approximately 9.2 million inhabitants in 2020. Its area is 148,895 km², which is slightly larger than Greece. In 2019, the state had a monthly ‘per capita’ income of US$ 233.80, below the national average of US$ 356.93 (INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA, 2021).¹ Ceará’s Human Development Index (HDI) was also lower than the national average in 2017.² Despite these conditions, the state has experienced substantial improvements in education following the implementation of two new policies. Figure 01 shows that students’ performance in Ceará has improved more than the average of other states. The dashed lines show when the two interventions – tax incentives (TI) and technical assistance (TA) – were put into place at each level of education.³

Figure 01
. Scores of students in primary and lower secondary schools from 1995 to 2019.

TI consisted of a performance-based financing scheme under which tax revenues were redistributed to municipalities according to their achievements in the fields of education, health, and environment. TA was implemented with the Programa Alfabetização na Idade Certa (PAIC), the ‘Literacy at the Right Age’ program, which established a new model of management and collaboration between the state government and municipalities to ensure high standards of literacy and educational quality (BONAMINO et al. 2019; LAUTHARTE et al. 2021; SUMIYA et al. 2017).

The laws establishing these two new policies were passed in December 2007, but municipalities in Ceará started to receive TI only in 2009. Because mayors knew that their tax revenues would depend on educational outcomes since the passing of the law, however, I consider 2008 to be the starting year of the policy for the purposes of policy evaluation.⁴

The TA program was established in 2007, comprising initially only the first two years of primary school. Bonamino et al. (2019) argue that the positive effects of PAIC motivated its gradual expansion. In 2011, PAIC+5 extended the program to cover all of primary education (1^st to 5^th grade). In 2015, the program became MAIS PAIC, which included both primary and lower secondary schools (1^st to 9^th grade). Because student performance in mathematics and Portuguese is only measured by Saeb in the 5th and 9th grades, 2011 and 2015 were considered to be the starting years of TA in primary and lower secondary education, respectively, as shown in Figure 01.

In this article, TI and TA are regarded as incentives to increase educational quality. TI is clearly an incentive, since mayors receive higher revenues as a reward for improving education. On the other hand, PAIC’s nature as an incentive is less obvious. Bonamino et al. (2019) see PAIC as a complex arrangement with a high capacity to articulate the cooperation between the state government and municipalities. In a way, TA offers a set of incentives for municipal governments to improve learning quality. Training for teachers and civil servants working in school management can be understood as incentives for improving their teaching and management skills.

Moreover, the Escola Nota Dez prize, an initiative related to PAIC, awards the best schools in learning achievement. These schools are granted financial resources, but only receive the complete prize if they offer support to a lower-performing school (CRUZ et al., 2020; SUMIYA et al., 2017). Thus, Escola Nota Dez is an incentive for schools to achieve better results and to cooperate with other schools.

The following sections explore in detail each of the new educational policies implemented in Ceará.

Programa Alfabetização na Idade Certa (TA)

In Brazil, federated states and municipalities collaboratively organize their educational systems (BRASIL, 1996). As shown in Table 01, both municipalities and states are responsible for providing primary and lower secondary education. This leads to an overlap of responsibilities and ambiguity in the role of each sphere of the federation.

Table 01
Responsibility for education in Brazil according to the national educational guidelines law.

Grade	Level	Responsibility
01-05	Primary Education	Municipality/State
06-09	Lower Secondary Education	Municipality/State
10-12	Upper Secondary Education	State
-	Higher Education	Union (Federal Level)

Source:

Unlike other Brazilian states, Ceará started to address this problem decades ago. In 1995, the state passed the municipalization law. Its objective was to transfer the responsibility for primary and lower secondary schools to municipalities, thus clarifying the role of each sphere of the federation (SEGATTO, 2015). Moreover, Ceará established a forum on education and a permanent program for assistance to municipalities. These initiatives aimed to promote the democratization of access to education, cooperation, and the municipalization of primary and lower secondary education (NASPOLINI, 2001).

Ceará’s new policies produced remarkable results. Between 1995 and 2000, enrollment in primary and lower secondary schools increased by 35.4 percent in the state, whereas the increase in Brazil as a whole was of only 9.3 percent (NASPOLINI, 2001). Regarding municipalization, in 2018, 99.3 percent of the primary schools in Ceará were under municipal administration, compared to 83.5 percent in all of Brazil. In the same year, 96 percent of lower secondary schools were managed by municipalities in Ceará, compared to only 50.5 percent in Brazil (LOUREIRO et al., 2020).

With clear-cut competencies for municipalities and for the state, the collaboration between the two spheres became smoother. By not providing primary and lower secondary education, the state could focus on offering TA to the municipalities. The collaboration between the state and municipalities was institutionalized by Law 14.026, from 2007, which established PAIC (SEGATTO, 2015).

PAIC resulted from the articulation of several organizations and actors (SEGATTO and ABRUCIO, 2018). In 2004, the state parliament of Ceará created the committee for the elimination of illiteracy in Ceará, with the aim of investigating the quality of education in the state. This initiative received support from the United Nations International Children's Emergency Fund (UNICEF), the association of mayors of Ceará, the union of municipal educational leaders (Undime), the state and federal universities of Ceará, private universities, and specialized civil servants working in education (BONAMINO et al., 2019; SEGATTO, 2015).

The committee’s investigation showed that only 40 percent of the students in the analytical sample were literate. To change this reality, PAIC was implemented as a state program starting in 2007 (SEGATTO, 2015). Although the program was optional, it was adopted by all of the state’s 184 municipalities since its beginning (SUMIYA et al., 2017).

PAIC established technical and instrumental standards that defined the responsibilities of each stakeholder in the educational process (CRUZ et al., 2020). The main actions of the program were: the training of teachers focused on classroom practice; the provision of literacy materials to schools; the promotion of workshops to disseminate best practices; the strengthening of the state system of evaluation of primary and lower secondary education (SPAECE⁵); and the training of municipal civil servants with a focus on the management of school systems (LAUTHARTE et al., 2021). These activities were carried out through agreements between municipalities and the State Secretariat of Education (SEDUC).

To facilitate the cooperation between the state government and municipalities, Ceará established the ‘Coordenadoria de Cooperação com os Municípios’ (COPEM), the ‘Coordination for Cooperation with Municipalities’. Figure 02 provides an overview of COPEM. At the state level, experts were hired to train ‘specialist teachers’, the name given to teachers responsible for disseminating skills and good practices in their municipalities. Each municipality had one local manager and several specialist teachers. The local manager was responsible for managing the actions and establishing communication with the SEDUC. Both local managers and specialist teachers could apply for financial support to improve their qualifications and skills (CRUZ et al.,2020).

Figure 02
Collaborative arrangements between the State government and municipalities

The tax incentive (TI)

The Brazilian Constitution states that revenues from the ‘Imposto sobre Circulação de Mercadorias e Serviços’ (ICMS), the state consumption tax, shall be divided between states (75 percent) and municipalities (25 percent). Furthermore, from the 25 percent of revenues reserved for municipalities, 75 percent should be distributed based on the added benefit criteria, which means that municipalities producing and selling more will receive more resources. The Constitution grants the federated states discretion to define how to distribute the remaining 25 percent among their municipalities.⁶

State Law 14.023, passed in Ceará in 2007, established that this share would depend on municipalities’ outcomes in education (18 percent), health (05 percent), and environment (02 percent) (BRANDÃO, 2014).⁷ Although dependent on educational performance, these resources did not have to be invested in education, which represented an additional incentive (LOUREIRO et al., 2020). The share of ICMS referred to above is identified as 𝑄𝑢𝑜𝑡𝑎𝑚^{𝐼𝐶𝑀𝑆}_m, that is, the quota of tax, which each municipality m is entitled to receive:

$\text{Quota}{}_m^{\text{lCMS}}=0.18EQ{I}_m+0.05{HQI}_m+0.02E{Q}_m$ (01)

E𝑄𝐼_𝑚 is the education quality index in municipality m; 𝐻𝑄𝐼_𝑚 is the health quality index in municipality m; and 𝐸𝑄_𝑚 is a dummy indicating whether municipality m has an operational solid waste management system (LOUREIRO et al., 2020).

The methodology to calculate 𝐸𝑄𝐼_𝑚 is shown in Equation 02.⁸ It was reformulated in 2011 to focus on the lower tail of the distribution of performance, that is, municipalities that improve the outcomes for students lagging behind are benefited more than others (LAUTHARTE et al., 2021).

$EQ{I}_m=0.5LQ{I}_m+0.45FQ{I}_m+0.05\lfloor \frac{{\overline{P}}_m}{\sum _m{\overline{P}}_m}\rfloor$ (2)

𝐿𝑄𝐼_𝑚 is the Literacy Quality Index for municipality m. 𝐹𝑄𝐼_𝑚 is the index that measures the quality of so-called ‘fundamental schools’ in municipality m – in Brazil, fundamental schools comprise primary and lower secondary schools. Finally, ${\overline{P}}_m$ is the average passing rate in primary school for municipality m.^{Appendix A} presents how each component of Equation 02 is calculated. One caveat of the EQI computation is that it considers current and past educational performance. Thus, in the first years of his or her term, a given mayor’s incentives are dependent on the past administration. They will only receive incentives integrally dependent on their own performance near the end of their term.

The impact of incentives and technical assistance on Education

Incentives for the actors involved in the educational process

A growing literature explores the impact of incentives on the quality of education. Incentives might target several actors involved in the educational process. Regarding incentives for teachers, results are mixed since effectiveness relies on appropriate incentive design (IMBERMAN, 2015). In a randomized controlled experiment in Tanzania, teacher salary bonuses dependent on student performance increased schooling quality (MBITI et al., 2019). Similar incentives implemented in schools in India and Israel also appear to have positively impacted student performance (LAVY, 2009; MURALIDHARAN and SUNDARARAMAN, 2011). However, not all studies found statistically significant impacts; some even found negative effects of financial incentives for teachers (FRYER, 2013; FRYER et al., 2012).

Incentives for students have also been a topic of research. Although some interventions had a positive effect on student attendance, the effects on performance are less clear (BARRERA-OSORIO et al., 2011; GALIANI and McEWAN, 2013). One intervention in the United States provided cash transfers for students who successfully completed standardized tests. Results showed improvements in mathematics, but no impact on reading and science scores (BETTINGER, 2011).

Studies have also shown that central exams incentivize students to increase their performance since scores might be seen by future employers or by educational institutions (BISHOP, 1997; WOESSMANN, 2018). However, there is also evidence that excessive focus on central exam contents might negatively impact student achievement (COLLIER, 2012).

With regard to incentives for city mayors, LAUTHARTE et al. (2021) studied the interventions in Ceará, employing a regression with year and city fixed-effects. They restricted their sample to schools located at the border between Ceará and neighboring states to make control and treatment groups more similar. The findings showed that TI combined with TA improved student scores in mathematics and Portuguese tests.

Carneiro and Irffi (2018) employ a difference-in-differences model to investigate the impact of TI in Ceará between 2007 and 2009. The findings are consistent with an increase of approximately 04 percent in mathematics and Portuguese test scores in primary education. Brandão (2014) and Petterini and Irffi (2013) employed the same methodology to analyze the policy and found positive impacts in mathematics and Portuguese scores.

In a more theoretical approach, Bishop and Woessmann (2004) devised a basic model of educational production. According to their model, giving political priority to education has positive effects on student achievement.

Technical assistance in education

Several works have examined the effect of TA on academic achievement. In a randomized control trial in Mongolia, researchers found that the provision of textbooks increased student scores, and that this improvement was intensified when textbooks were combined with teacher training (FUJE and TANDON, 2015). Angrist and Lavy (2001) found positive effects of teachers’ in-service training on student scores in reading and mathematics tests. In line with these findings are those of Bressoux et al. (2009), who studied the effects of teacher training in French schools. Their estimates showed an increase of 0.25 standard deviations in mathematics scores, but no improvement in reading.

There is also evidence that the provision of textbooks improves student achievement (FREDRIKSEN et al., 2015; McEWAN, 2015). However, a randomized trial in Kenya showed that textbooks had a positive impact only for the best-performing students (GLEWWE et al., 2009). Textbook choice also appears to have an impact on student scores. More engaging and demanding textbooks seem to increase scores more than less challenging ones (HADAR, 2017; HAM, 2018).

Methods and analytical sample

The synthetic control method

The fundamental problem of causal inference is that once a policy intervention is implemented in a particular space and time, one cannot no longer assess how the outcome of interest would have developed in the absence of that intervention. SCM is employed in this investigation to overcome this limitation.

SCM is a causal inference method that has gained popularity over the last two decades. It has been called “arguably the most important innovation in the evaluation literature in the last fifteen years” (ATHEY and IMBENS, 2017, p. 09). This method was developed to estimate causal effects when there are few aggregate units, with one unit being treated while the others are not. In this context, a combination of non-treated units provides a better control than any single non-treated unit (ABADIE, 2021).

To understand how SCM is estimated, let us consider that we have data for J + 1 units and j = 1, 2, ..., J + 1. In this research, j varies from 01 to 27, since Brazil has 27 federative units, 26 of which are states and the other of which is the Federal District. j = 1 is the treated unit, Ceará. The non-treated units constitute the donor pool, that is, all the candidate control states, j = 2, ..., J + 1. For each time t and unit j, we observe the outcome of interest, student performance. Considering that we have T periods and that T₀ refers to pre-intervention periods, the effect of the reforms in Ceará when t > T₀ is given by:

${\tau }_{\text{Ceará,}t}=Y_{\text{Ceará,}t}^I-Y_{\text{Ceará},t}^N$ (03)

Where 𝜏_{𝐶𝑒𝑎𝑟á,𝑡} is the effect of the reforms in Ceará at time t. 𝑌^I_{𝐶𝑒𝑎𝑟á,𝑡} is the outcome of interest for the treated Ceará, and it is observable. 𝑌^N_{𝐶𝑒𝑎𝑟á,𝑡} is the counterfactual, that is, the outcome of interest in Ceará if there had been no treatment. This counterfactual is what SCM aims to estimate. Equation 03 allows for the analysis of effects over time. This feature is critical in settings where effects take longer to appear or where they dissipate over time (ABADIE, 2021).

Usually, comparative case studies would estimate 𝑌^𝑁_{𝐶𝑒𝑎𝑟á,𝑡} by choosing the outcome of interest of a non-treated unit j with similar characteristics to the treated unit. One drawback of this approach is that it relies heavily on informal statements of the comparability between treated and non-treated units (ABADIE, 2021). SCM proposes that 𝑌^𝑁_{𝐶𝑒𝑎𝑟á,𝑡} could be estimated by a weighted average of the outcomes of a set of units in the donor pool.

The method is concerned with how to optimize the choice of these weights. Suppose $W=(w_2,\dots ,w_{J+1})$is a vector of weights assigned to each donor pool unit and ‘V’ is a vector of weights assigned to each predictor k. ‘W’ is defined, dependent on ‘V’, so that the mean squared prediction error (MSPE) between the treated unit and its synthetic version is minimized in the pre-intervention period⁹.

The weights are chosen so that the synthetic control most closely resembles Ceará’s outcome before intervention. The ability of SCM to estimate the counterfactual depends on how well it predicts the outcome of interest of the treated unit before the intervention. Importantly, SCM provides a transparent and data-driven methodology for choosing the control unit while avoiding specification search (ABADIE, 2021). In a different context, this method was employed to estimate the effects of homicide prevention measures in the Brazilian state of São Paulo (FREIRE, 2018).

Analytical sample: summary and descriptive statistics

Saeb is the main source of data for this research. It consists of Portuguese and mathematics exams, and assesses students every two years: at the end of primary school (5^th grade), lower secondary school (9^th grade), and upper secondary school (12^th grade). The exams are carried out by the ‘Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira’ (INEP), an institute linked to Brazil’s Ministry of Education. Saeb was created in 1990 and, in 1995, started to employ the Item Response Theory (IRT) to allow comparisons across time. Therefore, this study uses data from 1995 until 2019 (INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS ANÍSIO TEIXEIRA, 2020b)¹⁰.

During this period, a random sample of the population was assessed (INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS ANÍSIO TEIXEIRA, 2006). From 2007 onwards, all public schools with at least 30 students were tested (LAUTHARTE et al., 2021).

Only 0.8 percent of basic education is managed by Brazil’s federal government, most of which consists of upper secondary schools (INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS ANÍSIO TEIXEIRA, 2020a). Since these schools receive greater financial and technical support, and since their teachers are usually better qualified than those working in state and municipal schools, they were not included in the sample. For the same reasons, private schools were also excluded. The sample thus only consists of schools run by states and municipalities.

Each observation in the data set contains the average score of a state in a particular year, grade, and subject. I also created a dataset to control for Brazilian states’ social, demographic, and economic characteristics. These indicators come from different sources. The population estimates are from the Ministry of Health¹¹. Data on investment in education and industrial electricity consumption¹² are provided by the ‘Instituto de Pesquisa Econômica Aplicada’ (IPEA)¹³. Real values for investment in education and culture were determined by deflating nominal values against the consumer price index, using 2020 as the base year¹⁴. The real values were then divided by the population to generate a figure for investment per capita for each state. Homicides per 100,000 inhabitants were compiled with data from IPEA and Brazil’s Ministry of Health.

Finally, the unemployment rate was compiled with data from IPEA and the ‘Instituto Brasileiro de Geografia e Estatística’ (IBGE). Linear interpolation was used to impute two missing values in electricity consumption and investment in education and culture. Table 02 presents the analytical sample statistics with standard deviations in parentheses.

Table 02
Summary statistics of the analytical sample

	Pre-Intervention (1995 - 2007)			Post-Intervention (2008 - 2019)		Complete sample
	Ceará (N=7)	Other States (N=182)		Ceará (N=6)	Other States (N=156)	(N = 351)
Primary School
Score in mathematics	163 (10.7)		175 (13.0)	211 (17.5)	207 (16.9)	190 (22.0)
Score in Portuguese	158 (12.5)		167 (13.0)	199 (20.1)	193 (17.4)	179 (20.3)
Lower Secondary School
Score in mathematics	223 (4.32)		234 (12.2)	247 (12.8)	244 (11.9)	239 (13.1)
Score in Portuguese	220 (6.25)		229 (12.5)	247 (14.3)	242 (11.6)	235 (13.8)
Upper Secondary School
Score in mathematics	257 (5.23)		262 (15.5)	257 (5.96)	260 (12.5)	261 (14.2)
Score in Portuguese	251 (7.71)		256 (16.1)	256 (9.21)	259 (12.0)	257 (14.5)
State’s Characteristics
Investment in education and culture per capita in Brazilian Reais	368 (108)		583 (374)	500 (175)	781 (453)	665 (419)
Homicides per 100,000 inhabitants	17.7 (3.75)		25.1 (13.5)	40.3 (14.4)	31.8 (12.2)	28.2 (13.4)
Natural logarithm of the population	15.8 (0.078)		15.1 (1.12)	16.0 (0.028)	15.3 (1.05)	15.2 (1.08)
Unemployment (%)	7.72 (1.14)		9.25 (2.93)	8.96 (2.39)	9.72 (3.25)	9.42 (3.06)
Industrial Electricity Consumption in TWh	1.71 (0.251)		5.15 (9.20)	2.35 (0.151)	6.54 (10.9)	5.65 (9.89)

Source: Created by the author, based on data provided by IBGE, INEP, IPEA and SUS.

Before synthetic control findings are presented, exploratory plots will provide an overview of the analytical sample. Figure 03 presents the distribution of scores in mathematics and Portuguese tests before and after the intervention. Other states were more likely to have the best scores before the intervention. However, Ceará had better scores in the post-intervention period for both primary and lower secondary education. The same improvements were not observed in upper secondary education.

Figure 03
Density plots of the scores in mathematics and Portuguese

Figure 04 presents the change in test scores versus the average per capita investment in education and culture between 2007 and 2019. In primary and lower secondary education, Ceará achieves the highest score increase with a relatively low level of investment per capita. The plot suggests that Ceará was more efficient than other states. Once again, upper secondary education does not exhibit the same positive results.

Figure 04
Score change by average investment between 2007 and 2019

Finally, Figure 05 shows the map of Brazil before and after the intervention. Darker green colors represent higher scores in mathematics and Portuguese in primary education. The whole country improved between 2007 and 2019, but Ceará, highlighted with white contours, appears to have improved more than other states. Spatial plots for lower secondary education are presented in ^{Appendix B} and look quite similar. However, in upper secondary education, as also shown in Annexes, the maps do not suggest improvements.

Figure 05
Maps of Brazil showing mathematics and Portuguese test scores in primary education before and after the intervention

Note: Ceará is highlighted with white contours.

The synthetic control findings

In this section, SCM findings will be analyzed to further investigate the indications provided by the exploratory plots. I estimated¹⁵ one synthetic control for each level of education and subject, resulting in six models. The donor pool comprised all Brazilian states, except for Ceará. The predictors are the state’s characteristics, as presented in Table 02.

Table 03 shows the ‘W’ vector of each synthetic control. It indicates the four states, which contributed to the models: Bahia, Pernambuco, Piauí, and Rio Grande do Sul. The first three states are situated in the same region and share economic, social, and historical characteristics with Ceará. Therefore, it seems reasonable that these states contributed the most to the models. Rio Grande do Sul has distinct socioeconomic characteristics, but it only contributed substantially to the models for upper secondary education, where no statistically significant effects were found.

Table 03
Vector W showing the contribution of each state to each synthetic control

	Primary education		Lower Secondary Education		Upper Secondary Education
State	Mathematics	Portuguese	Mathematics	Portuguese	Mathematics	Portuguese
Bahia	0.356	0.419	0.359	0.082	0.354	0.132
Pernambuco	0.164	0.161	0.174	0.414	0.136	0.254
Piauí	0.479	0.417	0.466	0.417	0.386	0.393
Rio Grande do Sul	0.00	0.00	0.00	0.086	0.117	0.222

Source: Created by the author, based on data provided by IBGE, INEP, IPEA and SUS.

Table 04 shows the ‘V’ vector. It indicates to what extent each predictor contributed to defining the synthetic controls. Industrial electricity consumption, homicides, and population seem to be the predictors with the strongest influence in the models.

Table 04
Vector V showing the contribution of each predictor to each synthetic control

Predictor	Primary Education		Lower Secondary Education		Upper Secondary Education
	Mathematics	Portuguese	Mathematics	Portuguese	Mathematics	Portuguese
Homicides per 100,000 inhabitants	0.34	0.21	0.13	0.00	0.47	0.01
Industrial Electricity Consumption in TWh	0.18	0.32	0.39	0.80	0.16	0.37
Unemployment (%)	0.35	0.04	0.20	0.04	0.05	0.32
Natural logarithm of the population	0.13	0.42	0.29	0.15	0.30	0.29
Investment per capita in education and culture in Brazilian Reais	0.00	0.01	0.00	0.00	0.02	0.01

Source: Created by the author, based on data provided by IBGE, INEP, IPEA and SUS.

Table 05 shows how close the predictors of each synthetic control are to the ones of Ceará before intervention. All synthetic controls are better than the sample mean. However, the ability to emulate Ceará varies across the different synthetic controls and predictors.

Table 05
Comparison of characteristics of Ceará and synthetic Ceará before the interventions

Predictor	Sample Mean	Ceará	Synthetic Ceará
			Primary Education		Lower Secondary Education		Upper Secondary Education
			Mathematics	Portuguese	Mathematics	Portuguese	Mathematics	Portuguese
Homicides per 100,000 inhabitants	25.14	17.72	17.79	18.09	18.21	27.21	18.77	22.43
Industrial Electricity Consumption in TWh	5.15	1.71	3.28	3.78	3.33	2.23	4.18	3.39
Unemployment (%)	9.25	7.72	7.75	8.03	7.82	8.10	7.94	7.80
Natural logarithm of the population	15.08	15.82	15.58	15.67	15.59	15.52	15.71	15.62
Investment in education and culture per capita in Brazilian Reais	582.82	368.33	327.39	326.43	326.11	308.81	351.94	353.28

Source: Created by the author, based on data provided by IBGE, INEP, IPEA and SUS.

In Figure 06, mathematics and Portuguese scores for Ceará and synthetic Ceará are presented. Before the intervention (left of the dashed lines), the synthetic control can emulate the performance of Ceará quite well. For all models, in the post-intervention period, a gap between Ceará and its synthetic version becomes progressively larger, indicating that the intervention had an effect. The yellow lines indicate the performance Ceará would have had in the absence of the reforms, while the green lines indicate the actual performance attained by the state.

Figure 06
Performance of Ceará vs. synthetic Ceará in primary and lower secondary education.

The graphs in Figure 07 show the difference between the score of Ceará and its respective synthetic version. Table 06 shows the results of TI alone and TI combined with TA. Even in lower secondary education, where TI had more time to develop its effect, the effects only increased substantially when TI was combined with TA. In primary education, over the period between 2011 and 2019, scores increased 16.9 and 18.5 points on average in mathematics and Portuguese, respectively. In lower secondary education, over the period from 2015 to 2019, scores increased 12.7 and 14.4 points on average in mathematics and Portuguese, respectively.

Figure 07
Gap between Ceará and synthetic Ceará in primary and lower secondary education

Table 06
Results by policy, level, and subject

Level	Policy	Period	Subject	Effect
Primary education	TI	2008-2010	Mathematics	1.8
Primary education	TI	2008-2010	Portuguese	3.8
Primary education	TI+TA	2011-2019	Mathematics	16.9
Primary education	TI+TA	2011-2019	Portuguese	18.5
Lower secondary education	TI	2008-2014	Mathematics	3.9
Lower secondary education	TI	2008-2014	Portuguese	6.6
Lower secondary education	TI+TA	2015-2019	Mathematics	12.7
Lower secondary education	TI+TA	2015-2019	Portuguese	14.4

Source: Created by the author, based on data provided by IBGE, INEP, IPEA and SUS.

Finally, Figure 08 and Figure 09 show the same graphs for upper secondary education. SCM does not indicate a clear effect of the reforms on this level of education. The first students partially affected by TI took upper secondary exams in 2011¹⁶. The first students partially affected by TA took upper secondary exams in 2019¹⁷.

Figure 08
Performance of Ceará vs. synthetic Ceará in upper secondary education

Figure 09
Gap between Ceará and synthetic Ceará in upper secondary education

Robustness checks

The in-time placebo synthetic control tests the robustness of the findings provided by SCM (Abadie 2021). In this test, an intervention starting in 1999 is artificially created to estimate whether SCM still shows any effect from this ‘false’ intervention. If this were the case, the validity of the results could be put into question. Figure 10 shows no significant difference between the trends after the artificial intervention. Moreover, even when the intervention is artificially backdated by nine years, the effects appear shortly after 2008 with very similar magnitudes as the ones presented in Figure 06.

Figure 10
In-time placebo test with artificial intervention in 1999

A second recommended test is a leave-one-out re-analysis to test whether the results are sensitive to any units selected to create the synthetic control (ABADIE, 2021; ABADIE et al., 2015). Table 03 showed that SCM selected four Brazilian states: Bahia, Pernambuco, Piauí, and Rio Grande do Sul. To check whether eliminating one of these states affects my results, I estimate four synthetic controls by selecting one of the contributing states from the sample one at a time. In Figure 11, the leave-one-out synthetic controls are shown in gray. They are very similar to the synthetic control estimated using the complete donor pool. All of them point to a positive effect of the reforms; in some cases, they point to even larger effects.

Figure 11
Leave-one-out Test

Finally, the generalized synthetic control method (GSCM) was employed to test the robustness of the findings. This method unifies the SCM with linear fixed effects models to improve efficiency and interpretability. It also avoids specification searches and provides p-values for inference (XU, 2017). The results are presented in ^{Appendix C}. Average effects provided by traditional SCM are within the confidence interval of effects provided by GSCM in primary and lower secondary education. However, point estimates and the statistical significance of GSCM findings in primary education are sensitive to the choice of predictors. Moreover, the method suggests statistically significant effects in mathematics, which are further analyzed in the results discussion. The method does not provide statistically significant improvements in upper secondary education.

Inference for the SCM findings

Abadie (2021) proposes a mode of inference based on permutation methods to assess inferential aspects of synthetic control estimates. In this mode, a permutation distribution is obtained by reassigning treatment to each unit in the donor pool one at a time. Each of these estimated synthetic controls produces a ‘placebo effect’. All the placebo effects can then be compared to the effect estimated for the truly treated unit. The effect is only considered significant if it is extreme relative to the permutation distribution (ABADIE, 2021).

In Figure 12, the effects obtained with the treatment artificially reassigned to each donor pool unit are presented. The effect in Ceará is highlighted in green, while the effect in control states is shown in gray. The effects for Ceará are always positive and continuously increase between 2008 and 2015. By contrast, most of the other models move randomly and show smaller effects compared to the ones observed in Ceará.

Figure 12
Score gaps in Ceará and placebo gaps in the 26 control states

Many of the synthetic controls do not fit the pre-intervention data as well as the control estimated for Ceará. Therefore, to compare my synthetic control only with the ones that had similar pre-intervention mean squared prediction errors (MSPE), I excluded cases in which the MSPE was more than twice the MSPE of the synthetic control for Ceará. The results of this procedure can be seen in Figure 13. For Portuguese scores, the findings are unusually large compared to the estimations for other states. For mathematics scores, however, the rarity of the effects remains unclear.

Figure 13
Score gaps comparison with MSPE up to twice the MSPE for Ceará

To further investigate the significance of my findings for mathematics, I carried out a post/pre-intervention MSPE test suggested by Abadie et al. (2010). In this test, the post/pre-MSPE ratio distribution is plotted for all placebo gaps. This approach eliminates the need to choose an MSPE cut-off for evaluation. The idea is that a good synthetic control has a low error before the intervention because it closely fits the data. On the other hand, for the treated unit, the error is large after the intervention because there is an intervention effect. Therefore, I expect my estimated synthetic control post/pre ratio to be unusually high compared to the controls of non-treated states.

As Figure 14 shows, mathematics ratios are not sufficiently rare in the distributions. If one were to assign the intervention randomly in this data, the probability of obtaining a post/pre-intervention MSPE as large as the one for Ceará in primary education – mathematics - would be 8/27, or 0.3. For mathematics in lower secondary school, this probability is 5/27 = 0.19. Thus, the results are not statistically significant for mathematics. For Portuguese, the probability is 0.037 for both levels of education, and the estimates are statistically significant.

Figure 14
Post/pre-intervention MSPE ratio

Discussion of the results and their limitations

Results in Portuguese tests

In primary education, SCM suggests that TI alone increased Portuguese scores by about 04 points. When TI is combined with TA, over the period between 2011 and 2019, the scores are increased by 18.5 points on average, which represents an approximate 12 percent increase compared to the baseline shown in Table 02. In lower secondary education, TI had an average impact of 6.6 points during the 2008-2014 period. TI combined with TA increased the impact to 14.4 points, representing a 6.5 percent increase. To provide an idea of how large the impacts of the combined policies are: an 18.5-points increase represents around 12.4 months of effective schooling, while a 14.4-points increase represents approximately 9.7 months of schooling¹⁸.

These findings are in line with research by Lautharte et al. (2021). The authors report increases in Portuguese scores of around 12 and 15 points in primary and lower secondary education, respectively, during the period between 2008 and 2017.

In primary education, TI was implemented alone for a short period, and perhaps its effect would increase even without TA. However, in lower secondary education, TI had more time to develop its effects (2008-2014), but still did not reach half of the effect observed when TI is combined with TA. This suggests that TA is a significant driver of the effects and that the policies produce better results when implemented together.

Results in mathematics

SCM indicates that improvements in mathematics scores are not statistically significant, as highlighted in Figure 14. On the other hand, robustness checks (^{Appendix C}) and previous investigation suggests statistically significant effects also in mathematics (LAUTHARTE et al., 2021). It is important to highlight, however, that both the robustness check and Lautharte et al. (2021) employ different methodologies. Additionally, Lautharte et al. (2021) employ a reduced geographic and time span (2007 to 2017) compared to this study. One explanation for this divergence is that SCM fails to precisely fit Ceará’s performance in mathematics before the intervention, which leads to a lower post/pre-intervention MSPE that is not sufficiently unlikely in the placebo post/pre-intervention MSPE distribution (Figure 14)¹⁹.

Results in upper secondary schools

The absence of statistically significant impacts on upper secondary education is plausible because, while TI functions as an incentive for mayors, upper secondary education is under the state governor’s responsibility. Moreover, the first students affected by TA only reached upper secondary examinations in 2019 and were only partially affected by TA. It is also reasonable that the effects dissipate over the 03 years of upper secondary education. Further research is needed to evaluate the long-term effects of the interventions on upper secondary education, as well as on the later life of students.

Further limitations

One concern in this study is a pre-intervention upward trend in scores between 2005 and 2007 (Figure 01). This trend could indicate that a third factor, originating in 2005, could be impacting education in Ceará. However, this upward trend is also seen in the control states. A possible explanation for it is the rise in investment in education which occurred between 2005 and 2011 in all of Brazil. This should not bias the results since factors that impact both treated and untread units are already accounted for in the synthetic control. Moreover, investment in education is included as a predictor in the models.

An additional issue is that TI might have led mayors to exert pressure over teachers to train students specifically to perform well in Saeb tests. If this happened, the effects described here would not indicate real improvement in schooling. However, I argue that this is not the case because this study employed Saeb scores (national assessment) to estimate the causal effects. In contrast, the government of Ceará employs SPAECE (a state assessment) to grant TI.

The mechanisms driving performance increase

TI and TA appear to provide better results when implemented together. Impacts of TI more than doubled when it was implemented combined with TA. This suggests TA drives a substantial share of the observed effects. Possible mechanisms behind TA are the trainings for teachers and school civil servants, the collaboration between schools, and the provision of textbooks. Further research is needed to confirm to what extent each of TA’s actions contributed to the effects.

Regarding TI, it is plausible that the program’s incentives increased the level of political priority accorded to education because mayors wanted to maximize their tax revenues. Local politicians might thus have felt more encouraged to improve school infrastructure, including libraries, science labs, and sports facilities. These amenities, in turn, led to better performance.

SPAECE, the state system of evaluation of primary and secondary education, understood as an annual central exam, could also drive part of the effects by increasing students’ reward for studying and by strengthening the monitoring of schools (BISHOP, 1997; WOESSMANN, 2018). Finally, Figure 04 shows that Ceará is not among the states with the highest average spending on education, suggesting that higher spending is not a critical channel driving these effects.

Conclusion

This study provides evidence that incentives and technical assistance can effectively improve educational outcomes. It has been shown that TI and TA have led Ceará to experience substantial and robust improvements in test scores for Portuguese. Compared to baseline scores, there was an increase of around 12 and 6.5 percent in Portuguese scores in primary and lower secondary education, respectively.

These findings present a promising alternative for other Brazilian states pursuing a better quality of education. The new educational policies combined provided a performance gain equivalent to approximately 12.4 and 9.7 months of effective schooling in primary and lower secondary education, respectively. These improvements were achieved without increasing public spending on education relative to other Brazilian states.

This study contributes to the literature on the connection between technical assistance, notably the provision of textbooks and teacher training, and the quality of education. Furthermore, it provided empirical evidence of the effect of political priority for education on the level of schooling offered to students.

Recently, Brazil’s Congress passed a constitutional amendment²⁰, demanding that all Brazilian states condition tax transfers on educational outcomes. The results presented here appear to support this constitutional change. However, the lower municipalization rate observed in other Brazilian states might jeopardize the positive effects of TI, since fewer schools are under municipal administration than in Ceará. Another challenge is to overcome the resistance of mayors who might be wary of losing municipal revenues.

From a policymaking perspective, this study raises a relevant issue regarding the absence of improvements at the level of upper secondary schools. Even though students had a better quality of schooling in primary and lower secondary schools, they did not experience improvements in upper secondary schools. Policymakers should debate incentives directed to governors, the authorities responsible for managing upper secondary education in Brazil. The distribution of a federal tax could be conditioned to the educational outcomes of states in the same way that the distribution of the state tax, ICMS, is conditioned on the educational outcomes of municipalities.

Regarding the national system of education²¹, technical assistance and collaboration between the federal government, states, and municipalities could be tools to replicate the successful experience of Ceará. It is important to note, however, that most Brazilian states do not have a history of collaboration between the state government and municipalities comparable to that of Ceará.

This analysis sheds light on evidence-based alternatives to improve the quality of education in Brazil. Furthermore, the recent changes in the Federal Constitution offer an auspicious opportunity to introduce the strategies highlighted in this research in other Brazilian states. It is expected that the findings obtained in this study and in future research can provide relevant recommendations for the regulation of education in Brazil and in other countries.

References

• ABADIE, Alberto (2021), Using synthetic controls: feasibility, data requirements, and methodological aspects. Journal of Economic Literature. Vol. 59, Nº 02, pp. 391-425.
• ABADIE, Alberto; DIAMOND, Alexis and, HAINMUELLER, Jens (2015), Comparative politics and the synthetic control method. American Journal of Political Science. Vol. 59, Nº 02, pp. 495-510.
• ABADIE, Alberto; DIAMOND, Alexis and, HAINMUELLER, Jens (2011), Synth: an R package for synthetic control methods in comparative case studies. Journal of Statistical Software. Vol. 42, Nº 13, pp. 01-17.
• ABADIE, Alberto; DIAMOND, Alexis and, HAINMUELLER, Jens (2010), Synthetic control methods for comparative case studies: estimating the effect of California’s Tobacco Control Program. Journal of the American Statistical Association. Vol. 105, Nº 490, pp. 493-505.
• ANGRIST, Joshua D. and LAVY, Victor (2001), Does teacher training affect pupil learning? Evidence from matched comparisons in Jerusalem public schools. Journal of Labor Economics. Vol. 19, Nº 02, pp. 343–369.
• ARORA, Vipin and LIESKOVSKY, Josef (2014), Electricity use as an indicator of U.S. economic activity. Working Paper. Washington: U.S. Energy Information Administration.
• ATHEY, Susan and IMBENS, Guido W. (2017), The state of applied econometrics: causality and policy evaluation. Journal of Economic Perspectives. Vol. 31, Nº 02, pp. 03-32.
• BARRERA-OSORIO, Felipe; BERTRAND, Marianne; LINDEN, Leigh L., and PEREZ-CALLE, Francisco (2011), Improving the design of conditional transfer programs: evidence from a randomized education experiment in Colombia. American Economic Journal: Applied Economics. Vol. 03, Nº 02, pp. 167-195.
• BETTINGER, Eric P. (2011), Paying to learn: the effect of financial incentives on elementary school test scores. The Review of Economics and Statistics. Vol. 94, Nº 03, pp. 686-698.
• BISHOP, John H. (1997), The effect of national standards and curriculum-based exams on achievement. The American Economic Review. Vol. 87, Nº 02, pp. 260–264.
• BISHOP, John H. and WOESSMANN, Ludger (2004), Institutional effects in a simple model of educational production. Education Economics. Vol. 12, Nº 01, pp. 17–38.
• BONAMINO, Alicia; MOTA, , Maria Océlia; RAMOS, Maria Elizabete Neves and, CORREA, Erisson Viana (2019), Arranjo institucional de implementação do PAIC e burocratas de médio escalão. In: Teoria e análises sobre implantação de políticas públicas no Brasil. Edited by LOTTA, Gabriela. Brasília: ENAP. pp. 193-224.
• BRANDÃO, Júlia Barbosa (2014) O rateio de ICMS por desempenho de municípios no Ceará e seu impacto em indicadores do sistema de avaliação da educação. Master’s dissertation. Escola Brasileira de Administração Pública e de Empresas. Fundação Getúlio Vargas.
• BRASIL (1996), Lei 9.394, Diário Oficial da União, Nº 248, 23 de dezembro de 1996: p. 27833. Available at ˂http://www.planalto.gov.br/ccivil_03/leis/l9394.htm ˃. Accessed on March, 15, 2021.
  » http://www.planalto.gov.br/ccivil_03/leis/l9394.htm
• BRESSOUX, Pascal; KRAMARZ, Francis and, PROST, Corinne (2009), Teachers’ training, class size and students’ outcomes: learning from administrative forecasting mistakes. Economic Journal. Vol. 119, Nº 536, pp. 540–561.
• CARNEIRO, Diego and IRFFI, Guilherme (2018), Políticas de incentivo à educação no Ceará: análise comparativa das leis de distribuição da cota-parte do ICMS. In: Políticas públicas: avaliando mais de meio trilhão de reais em gastos públicos. Edited by SACHSIDA, Adolfo. Brasília: IPEA. pp. 300-350.
• COLLIER, Trevor C. (2012), The impact of institutional arrangements on educational efficiency. The Open Economics Journal. Vol. 05, pp. 01-14.
• CRUZ, Maria do Carmo Meirelles Toledo; FARAH, Marta Ferreira Santos and, RIBEIRO, Vanda Mendes (2020), Estratégias de gestão da educação e equidade: o caso do Programa Aprendizagem na Idade Certa (mais PAIC). Revista on line de Política e Gestão Educacional. Vol. 24, Nº 03, pp. 1286-1311.
• FREDRIKSEN, Birger; BRAR, Sukhdeep and TRUCANO, Michael (2015), Getting textbooks to every child in Sub-Saharan Africa: strategies for addressing the high cost and low availability problem. Series Directions in development: human development. Washington: World Bank. 127 pp..
• FREIRE, Danilo (2018), Evaluating the effect of homicide prevention strategies in São Paulo, Brazil: a synthetic control approach. Latin American Research Review. Vol. 53, Nº 02, pp. 231–249.
• FRYER JR., Roland G. (2013), Teacher incentives and student achievement: evidence from New York City Public Schools. Journal of Labor Economics. Vol. 31, Nº 02, pp. 373-427.
• FRYER JR., Roland G.; LEVITT, Steven D.; LIST, John and, SADOFF, Sally (2012), Enhancing the efficacy of teacher incentives through loss aversion: a field experiment. Working Paper 18237. Cambridge: National Bureau of Economic Research.
• FUJE, Habtamu and TANDON, Prateek (2015), When do in-service teacher training and books improve student achievement? Experimental evidence from Mongolia. Paper Working 7485. Washington: World Bank.
• GALIANI, Sebastian and McEWAN, Patrick J. (2013), The heterogeneous impact of conditional cash transfers. Journal of Public Economics. Vol. 103, Issue C, pp. 85-96.
• GLEWWE, Paul; KREMER, Michael and, MOULIN, Sylvie Moulin (2009), Many children left behind? Textbooks and test scores in Kenya. American Economic Journal: Applied Economics. Vol. 01, Nº 01, pp. 112-135.
• HADAR, Linor L. (2017), Opportunities to learn: Mathematics textbooks and students’ achievements. Studies in Educational Evaluation. Vol. 55, pp. 153–166.
• HAM, Ann-Katrin van den and HEINZE, Aiso (2018), Does the textbook matter? Longitudinal effects of textbook choice on primary school students’ achievement in mathematics. Studies in Educational Evaluation. Vol. 59, pp. 133–140.
• HECKMAN, James J. (2011), The economics of inequality: the value of early childhood education. American Educator. Vol. 35, pp. 31-47.
• IMBERMAN, Scott A. (2015), How effective are financial incentives for teachers? IZA World of Labor. Nº 158, pp. 01-10.
• INSTITUTO BRASILEIRO DE GEOGRAFIA E ESTATÍSTICA- IBGE (2021), Panorama – Ceará. Available at ˂https://cidades.ibge.gov.br/brasil/ce/panorama ˃. Accessed on March, 03, 2021.
  » https://cidades.ibge.gov.br/brasil/ce/panorama
• INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS ANÍSIO TEIXEIRA (2020a), Censo da Educação Básica 2019: notas estatísticas. Brasília: INEP. 32 pp..
• INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS ANÍSIO TEIXEIRA (2020b), SAEB–Histórico. Available at ˂https://www.gov.br/inep/pt-br/areas-de-atuacao/avaliacao-e-exames-educacionais/saeb/historico ˃. Accessed on March, 08, 2021.
  » https://www.gov.br/inep/pt-br/areas-de-atuacao/avaliacao-e-exames-educacionais/saeb/historico
• INSTITUTO NACIONAL DE ESTUDOS E PESQUISAS EDUCACIONAIS ANÍSIO TEIXEIRA (2006), Manual do Usuário - Microdados SAEB 1995. Brasília: INEP. 186 pp..
• LAUTHARTE, Ildo; OLIVEIRA, Victor Hugo de and, LOUREIRO, Andre (2021), Incentives for mayors to improve learning: evidence from State Reforms in Ceará, Brazil. Working Paper 9509. Washington: World Bank.
• LAVY, Victor (2009), Performance pay and teachers’ effort, productivity, and grading ethics. The American Economic Review. Vol. 99, Nº 05, pp. 1979–2011.
• LOUREIRO, Andre; CRUZ, Louisee; LAUTHARTE, Ildo and, EVANS, David K. (2020), The State of Ceara in Brazil is a role model for reducing learning poverty. Washington: World Bank. Available at ˂https://openknowledge.worldbank.org/handle/10986/34156 ˃. Accessed on March, 15, 2021.
  » https://openknowledge.worldbank.org/handle/10986/34156
• MBITI, Isaac; MURALIDHARAN, Karthik; ROMERO, Mauricio; SHIPPER, Youdi; MANDA, Constantine and, RAJANI, Rakesh (2019), Inputs, incentives, and complementarities in education: experimental evidence from Tanzania. The Quarterly Journal of Economics. Vol. 134, Nº 03, pp. 1627-1673.
• McEWAN, Patrick J. (2015), Improving learning in Primary Schools of developing countries: a meta-analysis of randomized experiments. Review of Educational Research. Vol. 85, Nº 03, pp. 353–394
• MURALIDHARAN, Karthik and SUNDARARAMAN, Venkatesh (2011) Teacher performance pay: experimental evidence from India. Journal of Political Economy. Vol. 119, Nº 01, pp. 39–77.
• NASPOLINI, Antenor (2001), A Reforma da educação básica no Ceará. Estudos Avançados. Vol. 15, Nº 42, pp. 169-186.
• ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT (2019), PISA 2018 Results. Vol. I: What students know and can do. Paris: OECD Publishing. 354 pp..
• PETTERINI, Francis Carlo and IRFFI, Guilherme Diniz (2013), Evaluating the impact of a change in the ICMS tax law in the state of Ceará in municipal education and health indicators. EconomiA. Vol. 14, Nº 03/04, pp. 171-184.
• SEDUC (2021), História do Programa Aprendizagem na Idade Certa - MAIS PAIC. Available at ˂https://www.seduc.ce.gov.br/mais-paic/ ˃. Accessed March, 15, 2021.
  » https://www.seduc.ce.gov.br/mais-paic/
• SEGATTO, Catarina Ianni (2015), O papel dos governos estaduais nas políticas municipais de educação: uma análise dos modelos de cooperação intergovernamental. Doctoral thesis. Escola de Administração Pública e Governo. Fundação Getúlio Vargas.
• SEGATTO, Catarina Ianni and ABRUCIO, Fernando Luiz Abrucio (2018), Os múltiplos papéis dos governos estaduais na política educacional brasileira: os casos do Ceará, Mato Grosso Do Sul, São Paulo e Pará. Revista De Administração Pública. Vol. 52, Nº 06, pp. 1179-1193.
• SUMIYA, Lilia Asuca; ARAUJO, Maria Arlete Duarte de and, SANO, Hironobu (2017), A hora da alfabetização no Ceará: o PAIC e suas múltiplas dinâmicas. Arquivos Analíticos de Políticas Educativas. Vol. 25, Nº 36, pp. 01.-30.
• WOESSMANN, Ludger (2018), Central exit exams improve student outcomes. IZA World of Labor 419. Available at ˂https://doi.org/10.15185/izawol.419 ˃. Accessed on March, 15, 2021.
  » https://doi.org/10.15185/izawol.419
• XU, Yiqing (2017), Generalized synthetic control method: causal inference with interactive fixed effects models. Political Analysis. Vol. 25, Nº 01, pp. 57-76.

• 1
  Considering the exchange rate provided by the Brazilian Central Bank on 31.12.2019.
• 2
  HDI data is provided by ‘Atlas do Desenvolvimento Humano no Brasil’ available at ˂http://www.atlasbrasil.org.br˃.
• 3
  For reproducibility purposes, all code and data employed in this study can be accessed in the following GitHub repository: ˂https://github.com/bruno-ponne/Better-Incentives-Better-Marks˃.
• 4
  In the synthetic control method, it is good practice to backdate the intervention when there are signs of anticipation. Moreover, backdating the intervention does not bias the estimator. Please refer to Abadie (2021, p. 409) for further details.
• 5
  SPAECE stands for Sistema Permanente de Avaliação da Educação Básica do Ceará.
• 6
  In 2020, Constitutional Amendment 108, from 2020, increased the proportion of ICMS over which states have discretion from 25 percent to 35 percent. The Amendment also requires that states condition at least 10 percentage points to performance in education.
• 7
  After the passing of Constitutional Amendment 108, from 2020, Ceará approved the following new criteria: education (18 percent), health (15 percent), and environment (02 percent). This change does not affect the period studied in this investigation.
• 8
  For details about HQI_m and EQ_m, consult LOUREIRO et al. (2020).
• 9
  Please refer to Abadie (2021) for a comprehensive formal presentation of the synthetic control method.
• 10
  The data are available at ˂https://www.gov.br/inep/pt-br/areas-de-atuacao/avaliacao-e-exames-educacionais/saeb/resultados˃.
• 11
  Data publicly available at ˂https://datasus.saude.gov.br˃.
• 12
  Industrial electricity consumption is used as a proxy for economic activity (ARORA and LIESKOVSKY, 2014).
• 13
  Data publicly available at ˂http://www.ipeadata.gov.br˃.
• 14
  The deflator is provided by the ‘Instituto Brasileiro de Geografia e Estatística’ (IBGE) at ˂https://ibge.gov.br˃.
• 15
  SCM was estimated with the R library ‘Synth’ (ABADIE et al., 2011).
• 16
  They were in the 9 grade in 2008 (TI start) and reached the 12 grade (Saeb exam) in 2011, assuming they did not fail any grade.
• 17
  They were in the 8 grade in 2015 (TA starts in lower secondary education) and reached the 12 grade in 2019.
• 18
  To make this comparison, the score increase of students between the 5 and 9 grades was calculated. Only students in states other than Ceará were considered. It was assumed that the students in 5 grade in 2007 reached the 9 grade in 2011. The same procedure was performed for 2011 and 2015; 2015 and 2019. These increases were then averaged and resulted in a 14.9 increase on average per year. Since one year of schooling in Brazil consists of 10 months (200 days) of effective schooling, a 14.9 increase was associated with 10 months of effective schooling.
• 19
  Data on education performance in Brazil is available only since the beginning of the 1990s. SCM might not perform well with a small number of pre-intervention periods. Classic applications of SCM, like the estimation of the effect of California’s tobacco control program (ABADIE et al., 2010), employ around 18 pre-intervention periods while this study had only seven periods, from 1995 to 2007, every two years. For a comprehensive discussion about this issue, please consult Abadie (2021).
• 20
  Constitutional Amendment 108, 2020, available at ˂http://www.planalto.gov.br/ccivil_03/constituicao/emendas/emc/emc108.htm˃.
• 21
  PLP 235, 2019 is a bill currently being discussed in the Brazilian Congress. It aims to establish a national system of education to improve governance and collaboration between the federal government, states and municipalities.

Appendix A

An Education Quality Index (EQI) Equation

This section presents the equation to calculate the Education Quality Index (EQI). Each of its components is calculated as follows:

$EQ{I}_m=0.5LQ{I}_m+0.45FQ{I}_m+0.05\lfloor \frac{{\overline{P}}_m}{\sum _m{\overline{P}}_m}\rfloor$

L𝑄𝐼_𝑚 is the Literacy Quality Index for municipality m. 𝐹𝑄𝐼_𝑚 is the index that measures the quality of the fundamental school for municipality m – in Brazil, fundamental school comprises primary and lower secondary schools. Finally, ${\overline{P}}_m$ is the average passing rate in the primary school for municipality m. The explanation of each component of the equation is as follows:

$LQ{I}_m=0.75\lfloor \frac{L_m}{\sum _m{L}_m}\rfloor +0.25\lfloor \frac{\Delta L_m^N}{\sum _m\Delta L_m^N}\rfloor$

Where L_m is the standardized test score in the literacy evaluation in municipality m, which can be calculated by:

$L_m=\frac{L{L}_m-L{L}_{MIN}}{L{L}_{MAX}-L{L}_{MIN}}$

LL_m is the score in the literacy evaluation (before standardization) for municipality m. 𝐿𝐿_𝑀𝐴𝑋 and 𝐿𝐿_MIN are, respectively, the highest and lowest scores recorded in a given year.

$L{L}_m={\text{average}}_m\cdot \frac{N_{Am}}{N_{Em}}\cdot U{I}_m$

Where average_m is the average score of second graders in the SPAECE-Alfa exam. This exam assesses students’ ability to read at a second grade level. $\frac{N_{Am}}{N_{Em}}$ is the number of students assessed in the SPAECE-Alfa divided by the number of students enrolled in a given municipality. 𝑈𝐼_𝑚 is the index of universalization and is obtained by:

$U{I}_m={(1-{\text{alfa}}_{1m})}^3\cdot {(1-\text{alfa}{a}_{2m})}^1\cdot {(1+\text{alfa}{a}_{3m})}^2$

Where 𝑎𝑙𝑓𝑎_1𝑚, 𝑎𝑙𝑓𝑎_2𝑚, 𝑎𝑙𝑓𝑎_3𝑚 are, respectively, the percentages of students classified as illiterate, with incomplete literacy, and with a desirable level of literacy.

The second term of the addition, which results in the literacy quality index involves △L^N_m, which is the standardized variation of the literacy evaluation for municipality m:

$\Delta L_m^N=\frac{\Delta L_m-\Delta L_{MIN}}{\Delta L_{MAX}-\Delta L_{MIN}}$

Where $\Delta L_m=L_{m\left(t\right)}-L_{m(t-1)}$ with t being the year of the evaluation. △L_MAX and △L_MIN are, respectively, the highest and lowest variations in the literacy evaluation considering all the municipalities in Ceará.

Now we turn to the Fundamental School Quality Index (FQI), given by:

$FQ{I}_m=0.5PQ{I}_m+0.5MQ{I}_m$

Where PQI_m measures the quality index for Portuguese and MQI_m the quality index for mathematics. The following expression can calculate PQI_m:

$PQ{I}_m=0.5\lfloor \frac{P_m}{\sum _m{P}_m}\rfloor +0.5\lfloor \frac{\Delta P_m^N}{\sum _m\Delta P_m^N}\rfloor$

Where P_m is the standardized test result in the Portuguese evaluation for the 5^th grade in municipality m:

$P_m=\frac{P{P}_m-P{P}_{MIN}}{P{P}_{MAX}-P{P}_{MIN}}$

Where PP_m is the result in Portuguese for the 5^th grade in municipality m before standardization, and is given by:

$P{P}_m=\text{average\_port}m\cdot \frac{N_{Am}}{N_{Em}}\cdot AI{P}_m$

Where 𝑎𝑣𝑒𝑟𝑎𝑔𝑒_𝑝𝑜𝑟𝑡_𝑚 is the mean score in Portuguese tests for the 5^th grade in municipality $m\cdot \frac{N_{Am}}{N_{Em}}$ is the number of students assessed in the SPAECE – 5^th grade divided by the number of students enrolled in a given municipality m. AIP_m is an adjustment index for the Portuguese exam calculated by:

$AI{P}_m={(1-p_{rof}{f}_{1LPm})}^2.{(1+p_{rof}{f}_{2LPm})}^2$

Where 𝑝𝑟𝑜𝑓_{1𝐿𝑃𝑚} and 𝑝𝑟𝑜𝑓_{2𝐿𝑃𝑚} refer, respectively, to the percentages of students whose performance is classified as ‘very unsatisfactory’ and as ‘adequate’.

△P^N_m, the standardized variation of the Portuguese evaluation in municipality m, is given by:

$\Delta P_m^N=\frac{\Delta P_m-\Delta P_{MIN}}{\Delta P_{MAX}-\Delta P_{MIN}}$

Where $\Delta L_m=L_{m\left(t\right)}-L_{m(t-1)}$ with t being the year of the evaluation. △P_MAX and △P_MIN are, respectively, the highest and lowest variations in the Portuguese evaluation considering all the municipalities in Ceará.

MQI_m is calculated similarly but with the grades obtained in mathematics tests for the 5^th grade in each municipality m.

Appendix B

Additional exploratory spatial plots

Figure A01
Maps of Brazil showing Portuguese and mathematics scores in lower secondary education before and after the intervention. Ceará is highlighted with white contours.

Figure A02
Maps of Brazil showing Portuguese and mathematics scores in upper secondary education before and after the intervention. Ceará is highlighted with white contours.

Appendix C

Additional robustness checks: the generalized synthetic control method

In this section, an additional robustness check of the findings is carried out, employing the generalized synthetic control method (GSCM). This method unifies the synthetic control method with linear fixed effects models to improve efficiency and interpretability. It also avoids specification searches and provides a p-value for inference (Xu 2017).

Tables A01 and A02 show the estimates provided by the GSCM. Model 01 includes only the variable ‘D’, indicating TI and TA. Model 02 includes three additional predictors: homicides per 100,000 inhabitants, unemployment, and industrial electricity consumption. Model 03 adds all the available predictors. These three models were chosen to evaluate the sensitivity of the results to different combinations of predictors. Since investment in education was not chosen by the traditional synthetic control method, it was not included in Model 02. Population was also not included in Model 02 because it was correlated with electricity consumption and number of homicides.

All results provided by traditional SCM are within the confidence interval of results provided by GSCM. However, point estimates and the statistical significance of the GSCM findings in primary education are sensitive to the choice of predictors. Model 02 was the only one suggesting statistically significant improvements in both Portuguese and mathematics scores. The variables population and investment in education in Model 03 might be inflating variance in the model and causing the increase in the standard errors.

Effects on lower secondary education are consistent across models and similar to the ones provided by traditional SCM.

In both levels of education, GSCM provided statistically significant effects on mathematics scores, in contrast with the traditional synthetic control. This divergence between the two methods might be due to different estimation strategies and different inference modes. Finally, in upper secondary education, models 01, 02, and 03 did not provide any statistically significant estimates.

Table A01
Average treatment effect on the treatment group in primary education (GSCM)

	Model 01		Model 02		Model 03
	Math	Portuguese	Math	Portuguese	Math	Portuguese
D (TI + TA)	29.23**	14.32	29.08**	22.0*	69.67	7.80
	(11.32)	(32.64)	(10.45)	(11.2)	(53.69)	(23.53)
Homicides per 100,000 inhabitants			0.064	0.015	-0.043	0.024
			(0.082)	(0.059)	(0.069)	(0.063)
Unemployment (%)			0.571	0.576	0.058	0.452
			(0.237)	(0.212)	(0.261)	(0.214)
Industrial Electricity Consumption			-0.409	-0.263	-0.567	-0.210
			(0.412)	(0.357)	(0.344)	(0.376)
Natural logarithm of the population					-11.317	-6.190
					(15.042)	(13.825)
Investment in education					-0.0027	-0.0044
					(0.003)	(0.002)

Source: Elaborated by the author. Notes: Standard errors are presented in parenthesis. *p<0.05; **p<0.01; ***p<0.001.

Table A02
Average treatment effect on the treatment group in lower secondary education (GSCM)

	Model 01		Model 02			Model 03
	Math	Portuguese	Math	Portuguese		Math	Portuguese
D (TI + TA)	13.97*	15.07*	13.58*		14.69*	13.48*	14.58*
	(5.535)	(5.88)	(5.802)		(6.32)	(5.90)	(6.34)
Homicides per 100,000 inhabitants			0.005		0.004	0.003	0.002
			(0.054)		(0.059)	(0.05)	(0.054)
Unemployment (%)			0.139		0.082	0.130	0.073
			(0.223)		(0.251)	(0.200)	(0.222)
Industrial Electricity Consumption			-0.268		-0.356	-0.350	-0.453
			(0.326)		(0.36)	(0.301)	(0.330)
Natural logarithm of the population						-9.70	-11.373
						(8.63)	(9.885)
Investment in education						0.005	0.0064
						(0.002)	(0.002)

Source: Elaborated by the author. Notes: Standard errors are presented in parenthesis. *p<0.05; **p<0.01; ***p<0.001.

Edited by

Revised by Thomás Abers Lourenço

Publication Dates

• Publication in this collection
  28 Apr 2023
• Date of issue
  Apr 2023

History

• Received
  24 Apr 2022
• Accepted
  21 Nov 2022