Motivation Assessment Scale for Learning in Higher Education ( EMAPRE-U ) : Validity Evidence

As part of the psychological evaluation, it is essential to use measuring instruments appropriate to the specific areas of interest and with good psychometric properties. In this perspective, this study was intended to seek validity evidence by analyzing the internal structure for the Motivation Assessment Scale for Learning (EMAPRE) in higher education, using confirmatory factor analysis. The EMAPRE is a Likert scale with 28 items, with the options “agree”, “disagree” and “do not know”. The study included 815 students, of whom 58.9% were male, with age spread from 18 to 63 years, in several majors in private institutions. Three models were tested and the results indicated best fits indices with the three-factor model, but with the need to eliminate two items from the EMAPRE. Adequate reliability levels were found, indicating that the instrument is useful in assessing learning motivation in university students.

Motivation in the school context is very complex, and in the specific context of the classroom, the student needs to be motivated to perform tasks of cognitive nature, that involve the processing and integration of information, thought, attention, problem solving (Bzuneck, 2004).Thus, the motivation to learn consists of the student's engagement in school activities, applying effort and persistence in the learning process, which is frequently associated with a successful learning (Bzuneck, 2004;Barkur, Govindan, & Kamath, 2013), academic performance, student involvement and the use of learning strategies (Cardoso & Bzuneck, 2004;Zenorini & Santos, 2004;Zenorini, Santos, & Monteiro, 2011), being considered one of the main factors that contribute to student learning and the understanding of what leads them to engage in academic activities (Santos, Alcará, & Zenorini, 2013;Zenorini & Santos, 2010a).
While reviewing the Brazilian studies developed from 1990 on motivation to learn, Boruchovitch and Bzuneck (2010) verified a gradual increase in the number of researches on the matter of the school context, under various educational approaches and different educational levels.On a further review of Santos, Alcará and Monteiro (2012), the results showed that the theories on Self-Determination and Achievement Goals were the most commonly used to support the studies in the period between 2001 and 2010.The authors stressed that, though both theories are a part of a macro system, which is the motivation to learn, and their main goal is to understand the reasons that lead a student to perform the academic tasks, they address specific phenomena.
The Theory of Self-Determination, developed circa 1970 by Deci and Ryan, has the goal of emphasizing the natural tendencies of the human beings toward growth, making necessary the supply of basic psychological needs of autonomy, competence and establishing links with other people (Deci & Ryan, 2000).On the other hand, the Theory of Achievement Goals derives from the cognitive approach of the Achievement Motivation theory from authors like McClelland and Atkinson circa 1950, and has brought back a new form of understanding the internal processes, giving less importance to more stable aspects such as personality traits and admitting that the cognitions underlying the goals are influenced by the classroom context (Bzuneck, 2004).
The Theory of Achievement Goals will be in focus in this work, as it gives theoretical support to the instrument used and has been taken as a theoretical framework of many contemporary studies (Harackiewicz, Barron, Pintrich, Elliot, & Thrash, 2002).The achievement goals that name the theory refer to purposes that lead people into reaching their objectives and that affect the way they approach the tasks (Bzuneck, 2004).The initial studies focused their attention into two main goals, the learning goal and the performance goal, to describe the student behaviors occurring in the school context, as described below.
Students oriented to a learning goal seek an intellectual development and believe that the positive results on the tasks come from their own effort.When they are successful in a certain task they are gripped by feelings of accomplishment, while mistakes are interpreted as reasons for learning, serving as a guiding light to the development of more effective strategies to reach their objectives and, consequently, more positive educational outcomes (Ames, 1992;Senko, Hulleman, & Harackiewicz, 2011).Studies have shown that the learning goal is associated with more persistence and effort during challenging tasks (Ames, 1992), with the use of in-depth cognitive strategies (Cardoso & Bzuneck, 2004); with searching for help when having difficulties, as well as with academic success (Kaplan, Lichtinger, & Goradetsky, 2009), and, specifically in higher education, with success expectations, value of task, career intentions and academic aspirations (Plante, O'Klefe, & Théorêt, 2013).
It is assumed that students guided by the performance goal are concerned about external rewards facing pressure and social demands, as well as concerned about showing the results of their learning (Elliot & Church, 1997).In this sense, they value public recognition and try to show themselves as more capable than the others.However, sometimes it may occur that they are less involved in the academic activities and avoid challenges, in fear that they'll be exposed and face negative consequences (Santos et al., 2012).Given this possible ambivalence, authors have been discussing that this goal may be divided into two components, namely, the performance-approximation goal and the performance-avoidance goal (Bzuneck, 2004;Harackiewicz et al., 2002;Middleton & Midgley, 1997;Zenorini & Santos, 2010a).
The performance-approximation goal is characterized by the student's search for proving himself capable and intelligent, ranking at the top, in comparison with his colleagues.Studies have shown different results regarding the performance-approximation goal, since some have been finding beneficial implications, such as goal setting and prediction of good academic outcome (Elliot & Church, 1997;Harackiewicz et al., 2002).On the other hand, in the study by Elliot, McGregor and Gable (1999), while the performance-approximation goal did positively correlate to good outcomes in exams and with the learning goal, it also correlated with the performance-avoidance goal, and with superficial learning strategies.Bandalos, Finney end Geske (2003) saw this goal as correlated with disorganization in the use of strategies and with high levels of anxiety, and in a recent study Kim, Hur end Park (2014) found that performanceapproximation and avoidance goals were positively correlated, both positively associated with superficial learning strategies, time management and looking for help in the face of academic difficulties.
In turn, the performance-avoidance goal is characterized by the student's fear in showing incompetency to his peers, not wanting to seem inferior to them.According to Senko, Hulleman and Harackiewicz (2011), the recurring use of this goal may be detrimental to academic results.It is observed that there is congruence between the results of researches concerning this goal, as they highlight its relation with unorganized studying, anxiety, low academic self-efficacy and low academic outcomes.(Boruchovitch & Costa, 2004;Elliot & Church, 1997;Kim et al., 2014;Middleton & Midgley, 1997;Zenorini et al., 2011).
To Senko et al. (2011), the division between performance-approximation and performance-avoidance goals has helped clarify inconsistencies in the definition of the performance goal.However, as more literature on Performance Goals surface, speculation has arisen over whether these two goals should be studied together as one, since studies have found positive correlations in moderate magnitude between performance-approximation and performance-avoidance goals (Hulleman, Schrager, Bodmann, & Harackiewicz, 2010).Facing this question, authors such as Muruyama, Elliot andYamagata (2011) andLinnenbrink-Garcia et al. (2012), have conducted studies evaluating the possibility of keeping performance goal as just one factor or subdividing it, and have obtained results that reinforce the notion that the goals should be separated.
The targets established by the Theory of Achievement Goals have been investigated empirically through instruments and measurements.In Brazil, according to a survey conducted by Santos, Alcará and Monteiro (2012), just two instruments were retrieved which were used for evaluation of motivation in college students, under the perspective of the Theory of Achievement Goals.One of those referred to the adaptation of the Scale of Sensitivity to the different Achievement Goals by Midgley et al. in 1998, and the other to the Scale of Achievement Goals developed by Accorsi, Bzuneck and Guimarães in 2007.Given the scarcity of instruments that evaluate the students in this schooling stage, studies began on the Scale for Learning Motivation (EMAPRE-U), using it in college students.This instrument was developed in Brazil by Zenorini and Santos (2010b), initially to evaluate high school students, and comprises the three goals congruent with the theory of Achievement Goals (learning, performance-approximation and performance-avoidance goals).
Using the scale developed by Zenorini and Santos (2010b), two studies were conducted with college students.Santos, Mognon, Alcará and Lemos (2011a) sought to analyze the convergent validity, relating the EMAPRE-U with the University Students Learning Motivation Scale (EMA-U), whose theoretical base is the Theory of Self-Determination.The results indicated statistically significant correlations of moderate magnitude between the learning goal and the intrinsic motivation, and weak magnitude negative correlation with the extrinsic motivation.A positive correlation of weak magnitude was also identified between the performance-avoidance goal and the intrinsic motivation, as well as a negative correlation with extrinsic motivation.
On the other hand, Santos, Mognon, Lima and Cunha (2011b) verified the relation between motivation to learn and adaptation to academic life.Positive and significant correlations of weak magnitude were detected, between the learning goal and the student's skill factor, between the performance-approximation goal and the conditions to study factor, and the performanceavoidance goal and the factors: commitment to the course, conditions to study and academic outcome.
Even though the largest portion of the results found in the researches by Santos et al. (2011a) and Santos et al. (2011b) using the EMAPRE-U are in agreement with what is presented in the literature, it was verified in those studies that the performance-avoidance goal showed unexpected results, with statistically significant relations with academic achievement, commitment to the course, intrinsic motivation and behavioral engagement, which instigated the developing on new studies, starting with verifying the psychometric qualities of the EMAPRE-U with a specific sample of college students.This way, aiming to verify the internal structure of the scale, Santos, Alcará and Zenorini (2013) applied it in 429 students.The results indicated, through Exploratory Factor Analysis (EFA), the three expected factors, which explained 39,41% of the total variance.The items were allocated to the same factors as the original scale and the coefficients obtained through Cronbach's alpha were adequately presented, with learning goal's α of 0.72, performance-avoidance goal's α of 0.83 and performanceapproximation goal's α of 0.82.It was detected that there was a positive moderate correlation between performance-avoidance and performance-avoidance (r = 0.41; p < 0.001), and only the performance-approximation goal showed positive significant correlation with the learning goal, with weak magnitude (r = 0.13; p < 0.006).
The EMAPRE-U was studied in Portugal by Frade and Veiga (2014), who sought validity evidences for the instrument.The participants were 149 graduates of the Officer Training Course, aged 25-36 years, 92.6% male.The results indicated the same factor structure as in Brazil, with the items allocated to the same factors, the performance-approximation goal with factor loadings (0.42 -0.81; α = 0.89), performance-avoidance goal (0.68 -0.89; α = 0.92) and learning goal (0.54 -0.84; α = 0.89).
It is noteworthy that item seven had to be excluded because it didn't allocate to any of the factors.Moreover, the three EMAPRE-U factors showed statistically significant and positive correlations with cognitive engagement, while the learning goal and the performanceapproximation goal related with affective commitment, dedication and vigor, and the performance-avoidance goal with behavioral engagement.
Even though the results of the EFA showed that the EMAPRE-U is adequate to be used for evaluating college students, the verification of the psychometric qualities of the instrument need to be broadened, adding new validity evidences.In this perspective the objective of this study is to continue to verify the validity evidences on the internal structure of the EMAPRE-U items with college students, using confirmatory factor analysis, assessing the plausibility of its factor structure.

Participants
Participants were 815 college students of a private institution in the state of Sergipe (Brazilian Northeast).The students were in different majors from the areas of humanities, formal sciences and health, ranging from the first to the tenth semester, taking night classes.The age of the students varied between 18-63 years (M = 21.35;SD = 7.93), most of them male (58.9%).The institution and courses were not randomly selected.

Scale for Evaluation of Motivation for Learning in
College Students -EMAPRE-U (Zenorini & Santos, 2010b).
As previously stated, the EMAPRE was initially designed for high schools and consisted of 50 items, 20 of those concerning to learning goal, 20 to performance-approximation and 20 to performance-avoidance, grouped in a Likert scale with 3 option choices -agree (3 points), don't know (2 points) and disagree (1 point).The exploratory factor analysis indicated a final version with 28 items, with the three proposed factors and adequate reliability indices assessed by Cronbach's alpha (α), with the learning goal totaling 12 items and α of 0.80; performance-approximation goal with seven items and α of 0.76; and avoidance-performance with nine items and α of 0.73.The relations between factors indicated correlations that are considered weak, between performance-approximation goal and avoidance (r = 0.137), as well as between learning goal and performance-approximation (r = 0.133) and negative between learning goal and performance-avoidance (r = -0.231).
For use with college students, the authors performed some adaptations to the original instrument, such as the type of language used in some items, for example, substituting 'homework' for 'academic tasks'.The evidences of validity regarding the internal structure, verified with the EFA, have already been presented and were deemed adequate.As a result, the final instrument had 28 items allocated to the same three factors as the EMAPRE.To illustrate, some items from the EMAPRE are mentioned -performance-avoidance goal ("I don't participate in debates because I don't want my colleagues to laugh at me"), performance-approximation goal (It's important for me to do my tasks better than the others") and learning goal ("I do my tasks because I'm interested in them").

Procedure
Data Collection.The heads at the university, who authorized the research in their campus, indicated the courses, classrooms and teacher who would possibly accept that the data collection took place during the period of their classes.With that, sessions were scheduled with the teachers, and in the scheduled days all the students that were in the classroom were invited to take part in the research.The objectives of the research were explained to them, the voluntary nature of participating, and the confidentiality of their responses and identities.To those who accepted to participate, the instruments were administered; all the data collection took place collectively.The completion time was about 20 minutes.
Data Analysis.To properly answer the objectives of the research, we proceeded with the exploring of the data in the Statistical Package for the Social Science (version 20), confirming the non-existence of omissions.With regard to the normality of data, the values of skewness ranged from -1,114 to 2,898 and the kurtosis were meaningful, ranging from -1,716 to 7,906.
Given the non-normal distribution and the options for categorical answers of the EMAPRE, the estimation method Means and Variance Adjusted Weighted Least Square (WLSMV) was used for the confirmatory factorial analysis (CFA) in the Mplus software (version 7.2).The WLSMV estimator is the most suitable to be used in items with answers that can either be categorical or ordinal, such as the items in the EMAPRE.The used adjustment indices were RMSEA (Root Mean Square Error of Approximation -expected value: lower than 0,07, with an expected confidence interval of 90%); CFI (Comparative Fit Index -expected value: greater than 0.92); TLI (Tucker-Lewis Index -expected value: greater than 0.92) and WRMR (Weighted Root Mean Residualexpected value lower than 0.80) (Hair, Black, Babin, Anderson, & Tatham, 2009).
The internal consistency was assessed by a composite reliability, given that Cronbach's alpha is sensitive to normality deviations (Maroco & Garcia-Marques, 2006).The composite reliability varies from 0 to 1, being evaluated as satisfactory when ≥ 0.70.After the most appropriate model was established, a descriptive analysis of the data was made through the measures of mean, standard deviation and Pearson correlations.

Ethical Considerations
Initially the university's authorization was requested for data collection, and after approval from the Research Ethics Committee (information removed by the Journal), contact was made with the coordinators of each course and days and times for the data collection were agreed upon with the teachers.The students that took part in the research signed an Informed Consent Term.

Results
Given the discussions in the literature on the need to not separate the performance-approximation and performance-avoidance goals, in Model 1 the CFA was tested with two factors.Thus, just the learning and the performance goals were considered.The adjustment indices all proved unsatisfactory, as seen in Table 1.With that, the CFA was employed admitting that the item saturation would occur on the three factors, given the three-factor structure found in the Exploratory Factorial Analysis conducted by Santos et al. (2013), but the index results on Model 2 were also not suitable.
When comparing the models, it was visible that the three-factor model showed indices more suitable, though not yet satisfactory.The results showed a reduced χ 2 and significant value (p < 0.001), CFI and TLI indices were below indicated (< 0.90), although the RMSEA (< 0.08) met the adequacy standards.According to Hair, Black, Babin, Anderson and Tatham (2009), values that are, in their majority, below acceptable in these indicators suggest the need to respecify the model.
Given these results, an analysis of residues and modification indices was made.When analyzing the matrix of residual covariance, that gives information on how well the variance and covariance was reproduced by the estimated standards of the model, it was noticeable that the results were very close to zero, meaning that the values were normal.The results in the modification indices indicated the need to establish covariance between the mistakes in the items 3 and 4 of the performance-approximation goal and items 14 and 25 of the learning goal.
A quality analysis was made in the items and it was verified that they had very similar content -item 03 (to me, it is important to do things better than the others) and 04 (it is important to me that I do tasks better than my colleagues), the same with items 14 (the harder the class, the more I like to try to understand it) and 25 (I like difficult and challenging tasks).Based on this analysis, it was decided that one of each of the covariance items should be eliminated and the model would be re-specified.With that, Model 3 involved the elimination of items 3 and 14.The low proportion of variance was key to their elimination in their factors.With this re-specification of the Model, the final version of the EMAPRE is presented in Figure 1.
Figure 1 shows the estimated parameters for the EMAPRE indicated no correlation that reached 1 or -1 and no negative factors on the items, the results ranging from 0.399 to 0.912.As to the relation between factors, the results indicated that both the correlation and covariance between Factor 1 -performance-avoidance goal and Factor 2 -performance-approximation goal was positive, of moderate magnitude, while these two factors showed negative correlation with Factor 3learning goal, with a weak magnitude to Factor 2 and moderate magnitude to Factor 1, based in the statistic assumptions by Dancey and Reidy (2006).With that, it was verified that there are no overlapping factors and, thus, there is no need to define performance-approximation and avoidance as one single factor.Indices of item-factor correlation were also calculated, which can be considered very satisfactory presented in the performance-avoidance goal with variation (r = 0.63 -r = 0.77), performance-approximation (r = 0.49 -r = 0.71), learning goal (r = 0,44 -r = 0,63).The internal consistency of the factors was considered adequate, given that they all presented composite reliability (CR > 0.70), being 0.91 for Factor 1, 0.86 for Factor 2, 0.93 for factor 3 and 0.90 for general Factor.The means found for each of the 26 items, individually studied, were located between 1.15 and 2.51 (M = 1.90); with Standard Deviations (SD) between 0.43 and 0.88.In addition, the means in each factor were also verified.The total value in found in each factor was divided by the number of items and the results indicated that the largest mean in the studied sample was in Factor 2 -performance-approximation (M=15.56;SD=3.75), followed by Factor 1 -performance-avoidance goal (M=14.29;SD=3.59) and lastly the learning goal (M=13.97;SD=3.45).

Discussion
The objective of this study was to verify the adequacy of the factorial model for EMAPRE-U that had Figura 1. Final model proposed for the EMAPRE.already been found in the Exploratory Factor Analysis (EFA) conducted by Santos et al. (2013), with college students.The establishing of three factors for EMA-PRE had already been done in other studies (Frade & Veiga, 2014;Zenorini & Santos, 2010b).For that reason, a new EFA was not presented in this study, however it was performed as an exploratory tool and the results indicated the same configuration obtained by Santos et al. (2013), as to the number of items in the instrument, the number of items allocated to each factor and the explained variance.
To test the models, it is noteworthy that though the focus of the study was based on the factor model found in the study by Santos et al. (2013), a two-factor model was also tested.The choice of a bi-factor model was due to the fact that the initial studies of the achievement goal theory showed two main goals, the learning goal and the performance goal (Bzuneck, 2004;Zenorini & Santos, 2010a), making no distinction between the performance-approximation and avoidance goals.Beside that there are discussions in the current literature that analyzed whether this division is indeed theoretically necessary (Hulleman et al., 2010;Linnenbrink-Garcia et al., 2012;Muruyama et al., 2011).However the results found in the present research indicated unsatisfactory levels of adjustment when analyzing the two factors.This way, the two elements of the performance goal can't be considered just one factor, with the differentiation a good contribution to the refinement of this kind of goals, since they are not in the same consolidation stage as the learning goal (Zenorini & Santos, 2004).
In the initial assessment of the adjustment indices of Model 2, based on the three factors (performanceavoidance goal; performance-approximation goal and learning goal) the results were also unsatisfactory.However, in Model 3 two items were eliminated (3 and 14), and the adjustment indices were more adequate, as well as the results of the composite reliability, which showed results varying from 0.86 to 0.93.In the study by Santos et al. (2013) and Zenorini and Santos (2010b), the indices of internal consistency evaluated through Cronbach's alpha on the three factors resulted lower, varying from 0.72 to 0.83, while in the research by Frade and Veiga (2014) the results were closer, ranging from 0.89 to 0.92.Urbina (2007) recommends that the adequacy and satisfactoriness of the model be based on coefficients higher than 0.70, being considered reliable and over 0.80 they can be considered good.Based on the results found, we can assume that the EMAPRE-U is a valid and reliable instrument and can be used with college students.
Regarding the indices of item-factor correlations, they were quite adequate (r = 0.44 to r = 0.77) indicating that the items are in line with what the factors are supposed to evaluate and the values are similar with those found in the study by Frade and Veiga (2014).Both the results of the correlations and the covariances between factors were positive and moderate between the performance-approximation goal and the performance-avoidance goal, while both presented negative relation with the learning goal.Positive correlation indices between the performance goals have been documented in studies (Elliot & Church, 1997;Elliot et al., 1999;Kaplan et al., 2009;Kim et al., 2014;Santos et al., 2013;Senko et al., 2011), while the correlation indices with moderate magnitude between the performance-approximation and performance-avoidance goals are similar to other studies, such as that of Muruyama et al. (2011) andLinnenbrink-Garcia et al. (2012).These authors ponder that though there is moderate correlation between the goals, there is substantial evidence that suggest that differentiation between them is useful and theoretically significant.
It is believed that new investigations are necessary, to continue exploring the association between the performance-approximation and performance-avoidance goals, which in the present study were negative.It is known that some researchers have found positive correlation between the two goals and discussed their beneficial effects, especially when combined (Bzuneck, 2004;Cardoso & Bzuneck, 2004;Harackiewicz, Barron, & Elliot, 1997;Plante et al., 2012), while others question the negative aspects of the performance-avoidance goal for its link with variables such as anxiety and disorganization (Bandalos, Finney end Geske, 2002;Elliot et al., 1999;Kim et al., 2014).This is a contradictory issue that is still open, over which only the accumulation of new findings may clarify.On the other hand, the negative correlation found between the performance-avoidance goal and the learning goal are justifiable, since some researches (Elliot & Church, 1997;Kim et al., 2014;Middleton & Midgley, 1997;Zenorini et al., 2011) have associated it with disorganized study, anxiety, low academic selfefficacy and low academic performance.Authors such as Muruyama et al. (2011) andLinnenbrink-Garcia et al. (2012) hypothesize that the divergent results found in the studies in regard to the goals may occur not just because of individual and context differences between the participants, but also because of the use of various instruments to assess the achievement goals.
As for the descriptive results that were analyzed, it was verified that the greater average was for with the performance-approximation goal.This indicates that the college students declared more concern over the external context than with appreciation of learning.International studies with college students resulted in a tendency towards the learning goal (Barkur et al., 2013;Elliot & Church, 1997;Elliot et al., 1999), while the Brazilian studies with college students using the EMAPRE-U (Santos et al., 2011a;2001b) also found the college students more interested in showing themselves as smart for their peers and teachers, using as a parameter comparison with colleagues.With this in mind it is possible to assume that the educational context may mold the students goals and, as a result, their accomplishment goals (O'Keefe, Ben-Eliyahu, & Linnenbrink-Garcia, 2013) and that, depending on the academic situation, the best student orientation, be it learning goal, be it performance-approximation goal, is key to achieving the desired learning objective (Barkur et al., 2013).
It is hypothesized that the result found in this research can be a reflex of a competitive environment, very common in the college environment (Harackiewicz et al., 2002), but it can also be a consequence of the educational system, which drives the incentive for grades and achievements much more, in some aspects, than learning itself.This goal may result in negative effects such as the students preference for superficial learning strategies and, in case failure occurs, the student may believe that it was due to his lack of capacity, which may lead to feelings of shame, anger and even giving up on certain academic activities or even the whole course, in order to avoid new frustrations (Bzuneck, 2004).
The results obtained with the use of the EMA-PRE-U with college students led to the finding of a model with satisfactory adjustment indices and adequate reliability, covariance and item-factor indices.However, there was the need to eliminate two items.For that reason, new studies are deemed necessary, that can confirm the factorial structure found, as well as others that replicate the studies with samples that have more national representation, since one of the limitations on the study was the fact that the whole sample was from a single Brazilian state, of a single private institution, and with students that go only to night classes.
It is also suggested that, with this current version of the scale, new evidences of validity are investigated with measures that evaluate related constructs positively or negatively (academic self-efficacy and academic stressors, for example), as well as their relation with the participation in the classroom and academic achievement.The concern with the quality of the instruments among researchers must be emphasized, since these measures may help to support the subsidizing of new researches, possible diagnosis and interventions.

Table 1
Values of the Adjustment Indices in the Confirmatory Factorial Analysis (CFA) for the Models