Factors Associated with Mathematics Performance in Brazilian Basic Education

Several studies have indicated that student performance is strongly influenced by school context. Within this frame of reference, the present study sought to identify contextual variables that affect proficiency in mathematics taking the multilevel structure of schools into consideration. The information of 4,338,885 students who participated in the Brazilian national assessment of educational progress of 2009 was used. Multilevel analysis showed an intra-class correlation of .27 and .19 for the 5th grade of elementary school and 9th grade of middle school, the last year of middle school. In both models, more than 50% of school level variance and more than 11% of the variance on student level was explained. The final model is composed of control variables, 13 student level variables, 9 school level variables, and random and interaction effects. The results contribute to a better comprehension of factors that affect school performance, which permits the orientation of public policies.

The current national situation demands education improvement actions and in order for them to become effective, education policies should be supported by academic research results.Large-scale educational assessments are therefore critical for identifying good practices, schools in need of intervention, and factors affecting the learning process.
External assessments were created in Brazil to gather information about performance and results of the educational system (Pestana, 1999).The main objective of the first Brazilian basic education evaluation system (SAEB -Sistema de Avaliação da Educação Básica) was to subsidize the monitoring of public policies aiming the universalization of education access and the improvement of quality, equity, and efficiency in education (Inep, 2007).SAEB currently monitors formal education from the 3rd grade of elementary school to the 3rd year of high school through performance assessment tests and contextual questionnaires.
Several educational studies suggest that contextual factors impact the learning process significantly (Alves & Soares, 2007;Carvallo-Pontón, 2010;Fletcher, 1998;Reynolds, Sammons, Fraine, Townsend, & Damme, 2011;Rutter & Maughan, 2002).Soares (2005) points out that these factors can be divided into three basic categories: school structure, family, and students.Thus, it is considered critical that these assessments cover information other than academic achievement, such as the students' personal and family characteristics, and the context where learning takes place.
Quantitative studies analyzing educational assessment data usually employ hierarchical linear models.Hierarchical models, also known as a multilevel regression analysis, should be used whenever data are collected from individuals belonging to groups or contexts.Lee (2008) and Hox (2010) highlight that in addition to being more appropriate, this approach is also more useful for studying contextual effects, especially in the educational setting.
Students belonging to the same class or school share experiences, have the same teachers, use the same materials and equipment and usually live in the same neighborhood.In addition, schools in the same system employ similar methodologies, have teachers with similar profiles, similar structures, and so on.Requiring the regression coefficients to be the same in all schools would be too restrictive in academic achievement studies (De Leeuw & Meijer, 2008).Multilevel models make it possible to analyze variable relationships within each school, allowing variation of regression coefficients among schools and the study of cross-level variable interactions.
These models present certain advantages, which are absent from models that do not consider individuals nested within groups.The use of multilevel models provides correct standard errors, thus allowing for accurate confidence intervals and hypothesis tests (Goldstein, 2010).By utilizing these models, inter-and intra-school variability can be broken down; consequently, the school's impact on student proficiency can be assessed.
Over the last few decades, several papers have been published utilizing data from large-scale educational assessments in Brazil, many of which have used SAEB data.These studies indicate that Brazilian schools are responsible for about 30% of the variation in student scores (Laros et al., 2012;Laros, Marciano & Andrade, 2010;Andrade & Laros, 2007;Jesus & Laros, 2004).This percentage is called the "school effect".However, the raw school effect still needs to be corrected for socioeconomic composition of the schools (Fletcher, 1998).Fletcher argues that without this correction, variance related to students and their families would be wrongfully attributed to the school (Laros et al., 2012).After considering socioeconomic characteristics, Brazilian schools would then be responsible for about 15% of the variability in student performance, demonstrating that schools still affect student performance to a considerable degree.
Several studies show that school socioeconomic status and parents' education level are the control variables most related to school performance (Jesus & Laros, 2004;Andrade & Laros, 2007;Laros, Marciano e Andrade, 2012).Some student-related variables that affect academic performance are: students not 'on track', students who work, parent support, and the habit of doing homework (Jesus & Laros, 2004;Andrade & Laros, 2007).School-level variables that affect performance are: technical and pedagogical resources, teachers assigning homework, school infrastructure, teacher commitment, teachers setting high expectations for students, students motivated by their parents, and students who work (Jesus & Laros, 2004).
Most studies use assessment data of students in 9th grade, the last year of middle school, and in 3rd year of high school.Despite data availability, there are fewer studies using SAEB data focusing on the performance of Grade 5 elementary school students.
Examples include Alves (2008) and Ferrão, Beltrão and Santos (2002).The lack of studies utilizing Grade 5 data and the lack of comparative analyses between school grades motivated the choice of the population of the present study.Sammons, Hillman and Mortimore (1995) conducted a literature review on school efficiency in the United Kingdom, United States, and the Netherlands and pointed out several student value-adding characteristics, such as: principal's leadership, clear objectives in classroom practices, emphasis on teaching and learning, high quality learning environment, shared vision and goals, positive reinforcement and high expectations of the teachers, monitoring progress, and home-school partnerships.Many of these factors are similar to factors that have been studied in the Brazilian context.However, Brazilian research is generally limited by the information collected in large-scale assessments.
Although research has progressed, Konstantopoulos (2006) suggests that the results of the studies are still inconclusive.In Brazilian studies, there is no consensus on which student and school factors are related to efficient schools.The only consensus that has been reached so far is the need for controlling for socioeconomic context to better assess the school's effect.A larger number of studies is needed to obtain more reliable and valid evidence on school efficiency.
The goal of analyzing the school context is to identify factors that allow defining improvement actions.Taking into account the importance of contextual factors and the hierarchical structure of the school system, this study aimed to identify the variables that interfere on the performance in mathematics of Grade 5 and Grade 9 students with a multilevel hierarchical analysis.

Participants
The Brazilian national assessment of educational progress databases of 2009 (ANRESC -Avaliação Nacional do Rendimento Escolar) is administered by Inep.ANRESC is an external assessment developed in 2005 with a census evaluation technique to determine academic achievement in all public Elementary Schools of Brazil with at least 20 students.This assessment is applied to all state, municipal, and federal public schools both in rural and urban areas.The reference population of ANRESC corresponds to Grade 5 and Grade 9 students.This assessment measures academic achievement in mathematics and reading, in addition to school and contextual variables.
The population of participants in the present study consisted of Grade 5 and Grade 9 students who took the mathematics test, which corresponds to 2,431,333 students from 35,799 schools and 1,907,552 students from 31,994 schools, respectively.On average, Grade 5 schools had 71.0 students (SD = 52.0)and Grade 9 schools had 59.2 students (SD = 44.1).The average age of Grade 5 grade students was 10.8 years old, 49.1% of these students were male, 90.4% studied in urban areas, and 71.2% attended the municipal school system.Grade 9 students had a mean age of 15.4 years old, 45.6% of these students were male, 92.2% studied in urban areas, and 63.4% attended the state school system.The geographic distribution of participating students was as follows: 42% were from the Southeast; 26% from the Northeast; 15% from the South; 10% from the North; and 7% from the Central-West.

Instruments
All instruments used in this study were developed by Inep for the 2009 Brazilian national assessment of educational progress, as follows: -Mathematics test: Cognitive test to assess student performance in mathematics consisting of 77 multiple-choice items divided into 7 blocks with 11 items, each student answering a test booklet with 2 blocks.The allocation of the items into blocks is made in such way that there is always a block of items connecting the test booklets.This allows placing all items on the same scale using item response theory, so that the scores of the different test booklets became comparable.
-Student, teacher, principal, and school contextual questionnaires.At the student level, the questions are related to family characteristics and social, economic, and cultural aspects.At school level, the questionnaires focus on teachers and principals qualification; pedagogic practices; resources and school infrastructure.
-School infrastructure indicator.This tool was developed by Soares Neto, de Jesus, Karino and Andrade (2013) to measure the level of Brazilian school infrastructure.The indicator was developed based on the Brazilian basic education census, administered by Inep.

Data Analysis
Multilevel analysis was applied using variables and factors extracted from the contextual questionnaires.Factors were chosen based on factor analysis realized by Karino, Vinha and Laros (2015) using Grade 9 questionnaires.In the present study, the same criteria were used to identify factors from Grade 5 data.Table 1 shows a description of variables and factors used.
After defining the factors, missing data were replaced using linear trend at point method, according to the participants' other answers within each factor.After the imputation, student scores were calculated for each factor.The remaining variables were treated differently: participants with missing data were excluded from the analysis.
MLwiN 2.30, developed by Goldstein et al. (1998), was used for multilevel regression analysis with student performance in mathematics as the dependent variable.The model was defined using the method proposed by Hox (2010), in a five-step procedure: 2. Introduction of the lower-level (student) explanatory variables fixed effects.
3. Introduction of the higher-level (school) explanatory variables.
4. Level 1 explanatory variable inclination coefficients that possess a significant inter-school variance component are verified (random effects).
5. Cross-level interactions effects, i.e.: school-level and student-level explanatory variable relations are assessed.
In addition, as pointed out by studies utilizing multilevel analysis in the educational context (Fletcher, 1998;Soares, 2004), socioeconomic level-related characteristics and parental education level were controlled.For this reason, a stage was added between steps 1 and 2 suggested by Hox (2010) to include control variables.
Finally, a standardized variable was only included if the estimated regression coefficient was greater than 1.Standardized variables were chosen because variables in different scales make it more difficult to determine coefficient relative importance (Fletcher, 1998).The criterion standardized regression coefficient greater than 1 was used because statistical significance tests (χ 2 test and t ratio) with large samples usually yield significant effects, even if the observed impact is very small.

Results and Discussion
The null model (Model 0), with no explanatory variables, was used as a benchmark to compare subsequent models.It is worth to highlight the observed intercept values (197.9 and 239.4), which represent the average proficiency in mathematics for Grade 5 and Grade 9 students (Table 2).
Another important value calculated by the null model is the intra-class correlation coefficient (ICC).This coefficient may vary from 0 to 1 and it indicates the need for using a multilevel analysis.A value close to 0 indicates that schools are homogeneous and that students' performance does not correlate to the school in which the student is enrolled (Ferrão, 2003).On the other hand, in the presence of a non-trivial value (greater than 10%), a multilevel method should be considered (Lee, 2008).
As shown in Table 2, in this study the intra-class correlation coefficients were .27and .19 for Grade 5 and Grade 9, respectively.This means that 27% and 19% of the Grade 5 and Grade 9 student performance variance is related to the school level, which justifies using the multilevel method.
Lastly, the deviance estimates of 24,992,424.6 and 19,662,922.5 were found for the null models.Deviance calculation estimates the lack of model adjustment and should be compared to subsequent models (Hox, 2010).With the addition of explanatory variables, model adjustment is expected to improve and as a result the deviance will decrease.In the comparison of models, the model with lower deviance will be considered the best one (Kreft & De Leeuw, 1998).
In the following step, control variables were introduced (Model 1).Mother's education and SES were originally considered as control variables, both in student and school levels (aggregate variables).However, aggregated mother's education showed negative effects in the model of Grade 9.This might be a result of multicolinearity, since there was a strong correlation between aggregated mother's education and SES.Aggregated mother's education was positively related to proficiency in   mathematics when separately analyzed.This variable was removed from the analysis, as its effect was negative for Grade 9 and it became negative for Grade 5 when other variables were added.
Table 2 shows Model 1 results.In general, students with higher socioeconomic status and students whose mothers have more formal education usually showed better performance.Aggregated SES showed higher coefficients than not-aggregated SES, the observed impact was 13.79 and 9.30 points on student performance for Grade 5 and Grade 9 models, respectively.Therefore, in order to predict the school's average performance, the average of students' socioeconomic level is more important than their individual socioeconomic level (Soares, 2004).These results are in line with previous studies (Soares & Alves, 2003;Soares, 2004;Jesus & Laros, 2004;Andrade & Laros, 2007;Laros, Marciano & Andrade, 2010).These studies highlight the importance of including control variables to avoid attributing to schools an effect that comes from the economic and cultural capital of the families.
Control variables contribute significantly to the model adjustment, statistics presented extremely high values (576,293.7 and 43,443.9)(Table 2).Adopting alpha = 5%, the improvement can be considered statistically significant.However, this test is very sensitive to high number of observations.Due to that, it will not be taken into account in the next steps.
Control variables explained about 1.2% and 1.8% of level 1 (student level) variance of the analyzed data.For the school level (level 2), 46.9% and 43.6% variance was explained by the models.Percentages related to school level reinforce how economic and cultural family-related capital (SES level and mother's education) affects student performance.
However, the explained variance at school level is lower in the present study than in other research papers.Jesus and Laros (2004) show that 79% of school level variance is explained by SES level, parents' formal education level, and ethnicity; Laros, Marciano and Andrade (2010) show that 77.4% of school level variability is explained by control variables (SES level, mother's education, and ethnicity); Laros, Marciano and Andrade (2012) show that SES level, mother's education, and ethnicity used as control variables are responsible for 74.6% of school level variance.The lower percentage of explained variance is related to the fact that schools in this study are more homogeneous.In previous studies, the SAEB database was used, which contained a sample of private and public schools.The present study used data from the Brazilian national assessment of educational progress (Prova Brasil), which only uses data of public schools.Table 2 also shows a corrected ICC of .17 and .12,indicating that the amount of variance in mathematics performance attributed to the school is still significant -even after adding control variables.
In the next step, variables of student level were added to the model (Model 2).Student level effects were estimated first, due to a higher number of observations (Hox, 2010).Subsequently, school level variables were introduced (Model 3).Models 2 and 3 are presented in Table 3.
The following variables showed an effect (regression coefficient) greater than 1 for Grade 5 data: gender, correct age, first school inscription, chores, work, grade repetition, homework, encouragement, books, TV and number of people.The dropout presented a coefficient lower than 1, but was kept in the model because it showed significant a random effect in Model 4.
For Grade 9 students, variables that presented an effect greater than 1 were: gender, corrected age, first school inscription, grade repetition, homework, encouragement, ethnicity, books, TV and number of people.In this case, the dropout variable was not taken into account due to its positive coefficient in the model, possibly as a result of multicollinearity.
Variables related to school career in Model 2 (Table 3) should be highlighted.Grade repetition showed a negative effect on school performance.If a Grade 5 or Grade 9 student had previously failed a school grade, their mathematics scores reduced on average 5.99 and 5.53 points, respectively.The first school inscription and correct age presented positive effects (3.05 and 2.35 points for the Grade 5 and 4.54 and 1.00 points for the Grade 9).Similar results have been shown by other researchers.Albernaz, Ferreira and Franco (2002) have also observed how failing a school grade presents negative impact; Laros and Andrade (2007) and Ferrão, Beltrão and Santos (2002) also point out how grade repetition can negatively influence student performance.Grade repetition is the opposite of the correct age variable used in this study.The correct age variable was chosen due to an initial descriptive analysis pointing out that students placed ahead or before their actual school grade usually presented lower average scores in mathematics.
Male students tend to have higher average scores in mathematics, especially Grade 9 students with a 5.70 coefficient.Similar results are demonstrated by Barbosa and Fernandes (2001) and Albernaz et al. (2002).Homework also influences performance in mathematics, since a 4.43 and 5.88 effect was observed for the grades analyzed.Lastly, students who spent more time watching TV, surfing Internet, or playing electronic games presented superior performance.This was one of the highest effects observed.Coefficients for Grade 5 and Grade 9 models were 1.87 and 3.90, respectively.It is worth pointing out that this variable could not be accurately interpreted.The question is related to hours spend on TV, Internet, and electronic games jointly.However, the positive effect could be related to the information provided by TV and Internet that are useful to learning process.
When comparing the results of the two models, the ethnicity effect is shown to be higher among Grade 9 students, doing chores and work have higher impact on Grade 5 students' performance, and encouragement was shown to be positive for Grade 5 and negative for Grade 9. Ethnicity having a greater effect on Grade 9 model is quite peculiar.This fact may be related to other effects studied in psychology, such as discrimination and selffulfilling prophecies, which are strongly influenced by contextual factors.Doing chores probably impacts Grade 5 students greatly because it is less common for children to do chores at that age.A negative effect of the encouragement was also observed by Laros, Marciano and Andrade (2010).According to these authors, the negative effect does not mean that encouragement will reduce student scores.It is most likely that bad performance be the cause of greater parent encouragement and demand on older students.
Model 2 explains 11.5% of level 1 variance and 58.3% of level 2 variance for Grade 5 students: for Grade 9 students the percentages are 11.9% and 49.9%.These values are shown in brackets in the tables.When the model with control variables is considered, 10.4% and 10.3% of level 1 variance and 21.5% and 11.1% of level 2 variance were explained.
The following step consisted of the introduction of level 2 (school) variables.Variables presenting a higher than 1 effect for Grade 5 data were: equipment, depredation, security, dropout rate, shortage of teachers, pedagogical equipment, pedagogic practices, and number of students.For Grade 9 data: depredation, security, shortage of teachers, number of students, and school infrastructure.It was not possible to obtain a factor related to pedagogical equipment from Grade 9 questionnaire, thus it was not considered in this model.
At the school level, dropout rate showed greater effect in Grade 5 model, with a -3.65 point coefficient.
Grade 9 students were more impacted by infrastructure (2.44) and the number of students in the school (-2.33).
In Model 3, variables related to school equipment also impacted the performance of the Grade 5 students.Equipment and pedagogical equipment presented effects of 2.01 and 1.39 points on the performance.Similar positive impacts were also observed in the research conducted by Andrade and Laros (2007) and Jesus and Laros (2004).Variables concerning security and depredation presented negative impact for both models, which indicates that students perform worse in schools with bad installations and where they feel less safe.
With the addition of level 2 variables, 65.4% and 52.6% of level 2 variance was explained in Grade 5 and Grade 9 models, respectively.Comparing to the control variables model (Model 1), 34.8% and 15.9% of level 2 variance were explained.
In the next step, random effects of student level variables were verified in order to analyze if the variables behave differently between schools (Model 4).This study only considered random effects for variables closely related to school context in which the school could have a more active influence.School policies, for instance, may interfere reducing the repetition negative effect, yielding interpretable random effects.Thus, the random effect was not tested for the following level 1 variables: gender, ethnicity, books, TV and number of people.The random effect was considered significant when the relative effect (square root of the variance over the fixed effect) was greater than .40.
Table 4 shows that the grade repetition and correct age presented the highest random effects, 14.54 and 6.75 for Grade 5 and 12.63 and 3.90 for Grade 9.These level 1 variables had considerably different impacts between schools.Laros et al. (2010) and Jesus and Laros (2004) also highlight the random effect of school delay.
The cross-level interaction effects were verified in the last step (Model 5 -Table 5).Interaction effects greater than .40were taken into account.Three interaction effects were verified from Grade 5 data: first school inscription and equipment; grade repetition and equipment; homework and equipment.Equipment interacts positively with first school inscription and homework.Therefore, the better the school equipment is, the stronger the positive impact of first school inscription and homework on the performance.The negative interaction between grade repetition and equipment indicates the difference in performance between students who have failed a school year and students who have not  a school year had greater negative impact in schools with better equipment; being a student who works was more detrimental to performance in schools with a higher number of students; and doing homework had higher positive effects in schools with good equipment and more students.
There have been few studies evaluating the effects of cross-level interaction, and none of them support the results found in this study.Laros et al. (2010) found interaction effects between an aggregated grade repetition and school delay, and between an aggregated cultural resources and school delay.Soares (2004) presents some interaction results among variables related to teachers and schools concerning equity issues.The author shows that schools and teachers with better characteristics increase differences between groups such as gender and ethnicity, socioeconomic level and school delay (high performance students take more advantage of school improvements, increasing inequality).
This paper chose to analyze few interaction effects, emphasizing school variables that can generate important information to guide improvement actions.School composition variables cannot be changed by school administration, thus interaction effects related to student profile were not taken into account.

Conclusion
The results of the present study complement the Brazilian research on educational assessment.One of the greatest benefits of this study is the possibility of identifying variables with different effects on Grade 5 and Grade 9.This comparison was hindered in previous studies because different criteria and variables were employed.Additionally, databases chosen in previous studies contained samples of reference population, with thousands of students.This study used a census database with information on millions of students, which required analysis process adjustments.
The results reinforce the influence of some level 1 variables on school performance, such as grade repetition, correct age, and homework (Albernaz, Ferreira & Franco, 2002;Laros & Jesus, 2004;Laros & Andrade, 2007).On the other hand, results point out that other variables are also relevant, such as number of people living at home and watching TV.
At level 2 the variable effects are less consistent between Grade 5 and Grade 9. Grade 5 variables that presented higher effects were equipment, security, and dropout rate; Grade 9 variables that presented higher effects were number of students and school infrastructure.These results indicate that specific studies need to be conducted for each educational level, to better understand the influence of school variables.
Additionally, this research contributed to the advance of random and interaction effects analysis.These effects bring more complexity into the model, and few current studies have endeavored similar analysis.In this sense, this research is a pioneer study.
Our objective was to assess school effects focusing on performance in mathematics.Therefore, we suggest that future studies accomplish comparative analysis between school grades, using language as a dependent variable.Some school characteristics are expected to impact certain knowledge areas more than others, generating different results.Lastly, we expect to have contributed to supporting public interventions and policies that could improve Brazilian education.
homework is corrected by the teacher.Factor extracted from student survey.Encouragement Encouragement by parents.Factor extracted from student survey.Ethnicity 0 = brown, black, and indigenous; 1 = Asian and white Books Amount of books at home: 0 = none; 1 = 1 to 20; 2 = 21 to 100; 3 = Over 100 books TV Amount of hours per day watching TV, surfing Internet or playing electronic games during the week.Grade 5: 1 = 1 hour or less; 2 = two hours or less Grade 9: 1 = 1 hour or less; 2 = two hours; 3 = 3 hours or less Number of people Number of people living at home with you: 1 = none or one person 2 = two people 3 = three people 4 = four to five people 5 = six to eight people 6 = more than eight people (Continued)

Table 3
Explanatory Variables Models for Grade 5 and Grade 9 Note.The explained variance shown in brackets refers to the control variables model.

Table 4
Model with Random Effects for Grade 5 and Grade 9

Table 5
Model with Random and Interactions Effects for Grade 5 and Grade 9