Self-regulation for Reading Comprehension: Psychometric Properties of Two Motivation Scales

Autorregulação para Compreensão de Leitura: Propriedades Psicométricas de Duas Escalas de Motivação

Autorregulación para la Comprensión Lectora: Propiedades Psicométricas de dos Escalas de Motivación

Abstract

Personal achievement goals and self-efficacy make up the motivational dimension of self-regulated learning. This research investigates the initial psychometric properties of the Achievement Goals Scale for Reading Comprehension and the Self-efficacy Scale for Reading Comprehension. The evidence-based test content validity study involved three expert judges and 16 Middle School, who pointed to the theoretical and practical adequacy of the scales. Validity evidence based on the scales’ internal structure was obtained with a sample of 522 students, using factor analysis as statistical resources. We verified the plausibility of the three-factor Achievement Goals Scale model and the one-factor Self-efficacy Scale model, and identified reasonable reliability estimates. Results indicate that the scales can be used in exploratory investigations. We suggest further research to expand its psychometric quality.

Keywords:
goals; self-efficacy; self-management; reading skills

Resumo

As metas de realização e a autoeficácia integram a dimensão motivacional da aprendizagem autorregulada. Este estudo teve por objetivo investigar as propriedades psicométricas iniciais da Escala Metas de Realização para a Compreensão de Leitura e da Escala Autoeficácia para Compreender a Leitura. Na investigação da evidência de validade de conteúdo participaram três juízes especialistas e 16 estudantes do Ensino Fundamental II, que apontaram para a adequação teórica e prática das escalas. A evidência de validade baseada na estrutura interna foi obtida com uma amostra de 522 estudantes, utilizando-se como recursos estatísticos a aplicação de análises fatoriais. Verificou-se a plausibilidade dos modelos da Escala de Metas de Realização, constituída de três fatores, e da Escala de Autoeficácia, unifatorial, bem como a identificação de estimativas adequadas de fidedignidade. Os resultados deste estudo indicam que as escalas podem ser utilizadas em investigações exploratórias. Sugere-se a continuidade das pesquisas para ampliar a sua qualidade psicométrica.

Palavras-chave:
objetivos; autoeficácia; autogestão; habilidades para leitura

Resumen

Las metas de logro y la autoeficacia integran la dimensión de motivación del aprendizaje autorregulado. En este estudio, se investigaron las propiedades psicométricas iniciales de la Escala de Metas de Logro y la Escala de Autoeficacia para la Comprensión Lectora. En la investigación de evidencias de validez de contenido participó tres expertos y 16 alumnos de la educación secundaria, quienes señalaron la adecuación teórica y práctica de las escalas. La evidencia de validez basada en las escalas internas se obtuvo con una muestra de 522 alumnos, utilizando el análisis factorial como recurso estadístico. Se verificó la plausibilidad del modelo de Escala de Metas de Logro, que consta de tres factores, y el factor único del modelo de Escala de Autoeficacia, además se verificaron parámetros de confiabilidad adecuados. Los resultados de este estudio indican que las escalas se pueden utilizar en investigaciones exploratorias. Sugerimos más estudios para expandir sus propiedades psicométricas.

Palabras clave:
objetivos; autoeficácia; autogestión; habilidades para la lectura

Motivation is one of the components of self-regulation for learning (SRL) associated with student’s interest, expectations, determination, and persistence in activities involving reading comprehension (Louick, Leider, Daley, Proctor, & Gardner, 2016; Richey, Bernacki, Belenky, & Nokes-Malach, 2017). Accordingly, the Achievement Goals Scale for Reading Comprehension (Escala Metas de Realização para a Compreensão de Leitura - EMR-CL) and the Self-Efficacy Scale for Reading Comprehension (Escala Autoeficácia para Compreender a Leitura - EA-CL) were constructed to evaluate two key processes of the SRL motivational dimension, namely: the personal achievement goals and self-efficacy (Zimmerman & Risemberg, 1997) of Middle School students to perform tasks involving this cognitive-linguistic skill.

Their construction derives from a larger research project, which identified the lack of instruments to asses achievement goals and self-efficacy for reading comprehension consistent with the specificities of Middle School. The emphasis on reading comprehension is justified because instruments that assess learning goals from a general perspective, such as academic performance (Santos, Moraes, & Lima, 2018) and performance in specific disciplines - Portuguese (Zambon & Rose, 2012), have a certain difficulty to appraise them in specific skills, such as reading comprehension. Such scenario also holds for self-efficacy assessment, since beliefs tend to vary according to the different types of school activities and subjects (Carroll & Fox, 2017; Toland & Usher, 2016).

In EMR-CL, therefore, the Learning Goal items were elaborated considering the valorization of knowledge extracted from the text. The Performance-approach Goal item set focused on achieving recognition for good performance in reading comprehension. In the Performance-avoidance Goal, items referred to concern about impending embarrassing situations due to the lack of reading comprehension skills (Bardach, Oczlon, Pietschnig, & Lüftenegger, 2020; Bzuneck & Boruchovitch, 2016; Klauda & Guthrie, 2014; Richey et al., 2017). Elaboration of EA-CL items emphasized the students’ perceived competence for reading comprehension, considering the complexity of the texts and the intertextuality prescribed in the Middle School curriculum, as well as the use of strategies that contribute to good performance in this cognitive-linguistic skill (Carroll & Fox, 2017; Zimmerman & Risemberg, 1997).

Personal Achievement Goals are characterized by how students approach school routine, which encompasses their attitude towards school activities and their self-perception of interpersonal relationships at school (Bardach et al., 2020; Richey et al., 2017). EMR-CL assess the beliefs manifested in three achievement goals. Learning Goal is present in students who value reading comprehension as a way to learn new things and expand their intellectual knowledge. In the Performance-approach Goal, students understand this cognitive-linguistic skill as a means of achieving recognition from teachers and classmates and thereby evidencing good academic performance. In the Performance-avoidance Goal, students seek to understand what they read, anticipating that by doing so they will minimize the chances of being exposed by a possible failure, should they not complete the activity (Bardach et al., 2020; Bzuneck & Boruchovitch, 2016).

Research conducted by Korpershoek, Kuyper, and van der Werf (2015) and Richey et al. with students from the United States and the Netherlands, enrolled in 7th and 9th grade of Basic Education, report that motivational guidance by the Learning Goal compared to Performance Goals was associated with students’ engagement in reading activities and good performance in reading comprehension. These researchers also identified a link between the Learning Goal with academic self-efficacy, school commitment, more flexible patterns of perceived intelligence/capacity, and social motivation, referring to interpersonal relationships.

In the Brazilian educational context, research observed a higher prevalence of the Learning Goal in Middle School students with better performance in Portuguese classes (Zambon & Rose, 2012). A correlational study found positive relationships for the Learning Goal and reading comprehension, and negative relationships between this skill and the Performance Goals (Santos et al., 2018).

In research with Elementary School students, however, the Learning Goal showed no statistically significant correlation with reading comprehension (r = -.09; p > .05), while statistically significant correlations of positive direction were identified between the Performance-approach and Performance-avoidance Goals and reading comprehension(r = .37 and r = .25, respectively; p < .05) (Ferraz, Cantalice, & Santos, 2019). Results found by Ferraz et al. (2019) revealed that guidance by Performance goals also converges with this cognitive skill, especially in students oriented towards the Performance-approach Goal. The authors hypothesized that contextual variables related to how the school approaches reading in evaluative activities my interfere with the guidance of students’ achievement goals for reading comprehension. A meta-analysis study corroborates Ferraz et al. (2019) arguments regarding the role of the school climate in meeting personal achievement goals (Bardach et al., 2020).

EA-CL, in turn, assesses students’ self-efficacy to perform procedures linked to reading comprehension proficiency, such as the use of cognitive and metacognitive strategies (Zimmerman & Risemberg, 1997). This construct is a predictor of reading comprehension (Louick et al., 2016). In early school years (4th to 6th grade), self-efficacy is associated with word reading and predicts aspects linked to the development of reading comprehension, such as working memory capacity and metalinguistic skills like phonological awareness (Carroll & Fox, 2017). In later school years (7th and 8th grade), self-efficacy functions in a more complex manner, since students’ beliefs are established based on their previous experiences, expressed by how they perceive their level of reading performance, which can affect the valuation of this activity and, consequently, the motivation to read (Klauda & Guthrie, 2014).

Based on the above, this study investigates the initial psychometric properties of the Achievement Goals Scale for Reading Comprehension and Self-Efficacy Scale for Reading Comprehension. In investigating the content validity evidence of EMR-CL and EA-CL, we hypothesized that the aspects assessed in their items were theoretically adequate and representative of the situations experienced by Middle School students in tasks related to reading comprehension (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 2014). When analyzing the validity evidence based on the EMR-CL internal structure, we expected to find a three-factor model relating to the Learning goal, Performance-approach and Performance-avoidance goals (Bardach et al., 2020; Ferraz et al., 2019; Santos et al., 2018; Zambon & Rose, 2012). For the EA-CL, we assumed to identify a one-factor model based on students’ self-efficacy beliefs about their reading comprehension ability, as well as to employ procedures linked to success in this cognitive-linguistic skill (Carroll & Fox, 2017; Klauda & Guthrie, 2014; Louick et al., 2016; Zimmerman & Risemberg, 1997).

Method

Evidence-based content validity study - Part 1: Expert judges

Participants

Three expert judges who work as professors and researchers that articulate the fields of Psychology and Education, in Higher Education Institutions in Brazil and Portugal. Importantly, the judges do not participate in the authorship of this research.

Instrument

Experts Evaluation Protocol. Prepared by the authors based on the Content Validity Coefficient - CVC (Hernández Nieto, 2002), the protocol evaluates the items of the Achievement Goals Scale for Reading Comprehension (EMR-CL) and the Self-Efficacy Scale for Reading Comprehension (EA-CL) by the validation criteria clarity of language (CL), practical relevance (PR), theoretical relevance (TR), answered on a Likert scale ranging from 1 (Scarcely adequate) to 5 (Fully adequate). Its fourth validation criterion - theoretical dimension (TD) - is answered by a dichotomous response - Yes or No. The Protocol also has a field for observations.

Procedure

Data collection. Teachers/researchers who had expertise in the constructs of self-regulation and reading comprehension skills were selected from the authors’ contact network. The research was conducted remotely via Google Forms, with the invitation to participate and access link of the survey being sent by e-mail.

Data analysis. Microsoft Excel software; Kappa Fleiss online calculator (http://justusrandolph.net/kappa/).To calculate the Content Validity Coefficient - CVC (Hernández Nieto, 2002), we considered a value below .80 in the validation criteria CL, PR and TR for rewording and excluding items from the EMR-CL and EA-CL. In TD we considered Kappa Fleiss values, qualified as acceptable between .40 and 1 (Brennan & Prediger, 1981). The experts’ observations were analyzed qualitatively, with special attention to the adequacy of the EMR-CL and EA-CL items to the educational reality of Middle School (American Educational Research Association [AERA] et al., 2014).

Ethical Considerations

The project from which this research report derived was approved by the Research Ethics Committee of the Universidade São Francisco, CAAE no. 10497519,8,0000,5514 (opinion no. 3,263,350). Data collection with the expert judges occurred upon acceptance of the Informed Consent Form (ICF). Student participation required an Informed Consent Form signed by the legal guardian and by the students’ signature to the Consent Agreement Form (CAF). For students who have reached majority, a specific ICF was made available.

Results

As indicated in Table 1 and Table 2, the CVCt for EMR-CL and EA-CL in all three validation criteria was within expectations and the k value, rated as excellent. Item 18 of the EMR-CL was reworded based on the judges’ suggestion, aiming to ascribe teacher’s praise directly to the student. The judges’ observations underpinned the permanence of items 2, 10, 13, 14, 24 and 27 of EMR-CL and item 9 of the EA-CL, despite the CVCc values below 0.80 and/or poorly rated k. Items 6, 7 and 12 of the EA-CL were kept by the researchers, given their adequacy to the Junior High context. This decision aimed submitting these items to target audience assessment.

Table 1
Judges’ Analysis: Achievement Goals Scale for Reading Comprehension

Achievement Goals Scale for Reading Comprehension items	CVCc			k
	CL	PR	TR	TD
1. Understand what *** new things	.96	.96	.96	1
2. I strive to *** my knowledge	.90	.96	.90	1
3. Comprehend what I read *** in my everyday life	.83	.83	.76	1
4. I like to feel *** complicated to read	.83	.90	.90	1
5. Schoolwork *** what is written	.83	.76	.76	1
6. I keep reading *** the text	.90	.90	.90	1
7. I get excited when *** read a text	.83	.90	.83	1
8. Reading comprehension *** school subjects	.96	.96	.96	1
9. When I like a text *** other similar	.83	.90	.90	1
10. Understand a text *** think about things	.83	.83	.76	1
11. I feel challenged *** interpret a text	.90	.83	.90	1
12. Understand what I read *** do well in school	.90	.96	.90	1
13. I strive to understand *** good I am	.90	.90	.90	.33
14. Understand what I read *** classmates	.83	.83	.83	.33
15. I like to be challenged *** classmates	.83	.83	.83	1
16. Understand the text *** school activities	.90	.90	.90	1
17. I like that people *** understand the text	.90	.90	.90	1
18. I am happy when *** I understood the text well	.83	.83	.90	1
19. Reading comprehension *** of school	.96	.96	.96	1
20. I read a lot of books *** think I am smart	.83	.83	.83	1
21. I feel competent *** what I read	.96	.96	.96	1
22. When the teacher *** my classmates	.90	.90	.90	1
23. Reading comprehension *** do poorly in exams	.83	.83	.83	1
24. I avoid reading texts *** understand the text	.90	.76	.96	1
25. I get worried *** texts I read	.90	.96	.96	1
26. I feel uncomfortable *** I have understood of the text	.83	.90	.90	1
27. I feel ashamed *** I understood of the text	.76	.76	.76	.33
28. I give up reading *** reading comprehension	.90	.96	.96	1
29. I feel afraid *** reading comprehension	.83	.83	.83	1
30. I get nervous *** a difficult text	.96	.96	.96	1
31. I hardly read *** teacher asks at school	.76	.90	.90	1
32. I feel unintelligent *** a text	.83	.83	.83	1
33. I get anxious *** interpret a text	.86	.96	.96	1
CVCt	.87	.88	.88
k (full scale)	.88
Inter-judge agreement for the full scale	93.94%
Item rewording	Reworded items		Justifications
Item 5. Schoolwork becomes easier when I understand what is written	Item 5. When I understand what is written it is easier to do the assignment		CVCc < .80
Item 18. I am happy when the teacher tells the whole class that I understood the text well	Item 18. I am happy when the teacher tells me that I am good at reading comprehension		Experts’ suggestions
Item 31. I hardly read things other than the texts the teacher asks for at school	Item 31. I only read the texts the teacher asks for at school		CVCc < .80

• Note. CVCc = Constant Content Validity Coefficient; k = Kappa Fleiss; CL = Clarity of language; PR = Practical relevance; TR = Theoretical relevance; TD = Theoretical dimension; CVCt = Total Content Validity Coefficient. Excerpts changed in the reworded items are underlined.

Table 2
Judges’ Analysis: Self-Efficacy Scale for Reading Comprehension

Self-Efficacy Scale for Reading Comprehension items	CVCc			k
	CL	PR	TR	TD
1. Understand texts *** complicated	.90	.96	.96	1
2. Use good strategies *** the text	.96	.83	.96	1
3. Draft *** the text	.96	.83	.96	1
4. Interpret *** I read	.96	.96	.96	1
5. Find *** to read	.90	.90	.90	1
6. Find *** or other sites	.76	.76	.76	1
7. Make associations *** in the text	.76	.90	.90	1
8. Know what the text is about *** reading	.96	.83	.96	1
9. Classify types *** or a poem	.90	.90	.76	.33
10. Imagine things *** in the text	.90	.76	.90	1
11. Save the information *** text	.90	.90	.90	1
12. Associate information *** daily life	.83	.76	.83	1
13. Assess whether *** the text	.96	.96	.96	1
14. Understand what are *** reading comprehension	.96	.96	.96	1
15. Share my *** classmates	.90	.96	.83	1
16. Read the texts *** by the teacher	.90	.90	.90	1
17. Associate the text *** others I have read	.96	.96	.96	1
18. Ask for help *** hard to understand	.96	.96	.96	1
19. Recognize which classmate *** a text	.90	.96	.96	1
20. Answer questions *** by the teacher	.96	.96	.96	1
21. Understanding *** mathematical problems	.96	.96	.96	1
22. Understanding school subject books *** and science	.96	.96	.96	1
23. Ask a classmate *** to understand the text	.83	.90	.90	1
24. Identify in what era *** I read	.96	.83	.90	1
CVCt	.88	.86	.88
k (full scale)	.83
Inter-judge agreement for the full scale	91.67%
Item rewording	Reworded items		Justifications
Item 7. Make associations between paragraphs in the text	Item 7. Make associations between ideas in the text		CVCc < .80; Experts’ suggestions

• Note. CVCc = Constant Content Validity Coefficient; k = Kappa Fleiss; CL = Clarity of language; PR = Practical relevance; TR = Theoretical relevance; TD = Theoretical dimension; CVCt = Total Content Validity Coefficient. Excerpts changed in the reworded items are underlined.

Method

Content validity evidence study - Part 2: Target audience assessment

Participants

Sixteen Middle School students from a school located in the countryside of São Paulo, aged between 11 and 15 years (M _age = 13 years; SD = 1.31), participated in the study. Each school year was represented by four students, with equal gender distribution.

Instrument

Pilot Study Assessment Protocol. Elaborated by the authors, the protocol aims to assess the intelligibility of the EMR-CL and EA-CL by Middle School students. Part I of the Protocol assesses the wording and responses of the scales, and Part II the items. Objective questions are answered in a dichotomous Yes or No way. The field “observations” is reserved for the students’ suggestions and indication of incomprehensible passages.

Procedure

Data collection. Students without reading difficulties participated in the research. This information was obtained from the school office. Research took place in person during class time, in a room provided by the school. Data collection was carried out in two moments via one-on-one interview, with an interval of one to two weeks between meetings. Students assessed the EMR-CL and EA-CL.

Data analysis. Data analysis was performed using Microsoft Excel. Answers to the objective questions were counted using absolute frequency analysis. Observations were analyzed qualitatively, considering the clarity, degree of difficulty, and relevance of the items to the reality experienced by Middle School students. Results of these analyses informed the rewording and exclusion of items.

Results

Table 3 and Table 4 indicate that students expressed difficulties with Part 1 of the Pilot Study Assessment Protocol, especially on the Achievement Goals Scale for Reading Comprehension (EMR-CL). However, this issue referred to the students’ lack of experience to act as evaluators, mainly due to the fragmentation of the scales in the Pilot Study Assessment Protocol. Therefore, we identified no issues with respect to understanding the statements and response keys on the EMR-CL and the Self-Efficacy Scale for Reading Comprehension (EA-CL).

Table 3
Target Audience Assessment: Achievement Goals Scale for Reading Comprehension

Part 1: Statement and response key labels
1.Understanding the instructions for filling out the scale	AF=14 (87.5%)
2. Identification of confusing passages	AF=7 (43.75%)
3. Identification of unknown words	AF=1(6.25%)
4. Understanding response key labels	AF=14 (87.5%)
Part 2: Items	LG	PAG	PAvG
1. Identification of similar content	AF=9 (56.25%)	AF=10 (62.5%)	AF=9 (56.25%)
2. Difficulty in comprehension	AF=1 (6.25%)	AF=3 (18.75%)	AF=1 (6.25%)
Item rewording	Reworded item
Item 3. Comprehend what I read helps me understand the things that happen in my everyday life	Comprehend what I read helps me to understand things in my day life
Item 4. I like to feel challenged to understand texts more complicated to read	I like the challenge of understanding texts more complicated to read
Item 6. I keep reading until I understand the text	Even when it is difficult, I keep reading until I understand the text
Item 15. I like to be challenged to do better in reading comprehension than my classmates	I like to do better in reading comprehension than my classmates
Item 20. I read a lot of books so that people will think I am smart	I try to read a lot of books so that people will think I am smart
Item 21. I feel competent when I understand what I read	I feel intelligent when I understand what I read
Item 29. I am afraid the teacher will tell the whole class that I have difficulty with reading comprehension	I dislike it when the teacher tells the whole class that I have trouble understanding what I read
Excluded items	Justifications
Item 11. I feel challenged when the teacher asks me to interpret a text	Students chose item 4 due to content similarity "I like to feel challenged to understand texts more complicated to read"
Item 22. When the teacher asks to interpret a text, I want to do better than my classmates	Students chose item 13 due to content similarity "I strive to understand difficult texts so that my teacher and classmates can see how good I am"
Item 26. I feel uncomfortable when I am asked to say what I have understood of the text	The content of items 26, 27 and 33 was represented by item 23 [30], "I get nervous when I have to read a difficult text"
Item 27. I feel ashamed to tell others what I understood of the text
Item 33. I get anxious every time the teacher asks me to interpret a text

• Note. AF = Absolute frequency; LG = Learning Goal; PAG = Performance-Approach Goal; PAvG = Performance-Avoidance Goal.

Table 4
Target Audience Assessment: Self-Efficacy Scale for Reading Comprehension

Part 1: Statement and response key labels
1.Understanding the instructions for filling out the scale	AF=15 (93.75%)
2. Identification of confusing passages	AF=3 (18.75%)
3. Identification of unknown words	AF=0 (0%)
4. Understanding response key labels	AF=16 (100%)
Part 2: Items
1. Identification of similar content	AF=5 (31.25%)
2. Difficulty in comprehension	AF=2 (12.5%)
Item rewording	Reworded item
Item 5. Finding time to read	Always find time to read
Item 8. Know what the text is about after reading it	Know what the text is about, after reading it
Item 13. Assess whether I have understood the text	Know if I understood the text
Item 20. Answer questions about the text asked by the teacher	Answer questions about the text, asked by the teacher
Item 22. Understanding school subject books, such as geography and science	Understanding school subject books, such as history, geography and science
Excluded items	Justifications
Item 4. Interpret what I read	Students chose item 8 due to content similarity "Know what the text is about after reading it"
Item 19. Recognize which classmate can help me better understand a text	Students chose item 23 due to content similarity "Ask a classmate to help me understand the text"

• Note. AF = Absolute frequency.

As a suggestion, two students indicated changing the Likert-type response key of the EMR-CL to a dichotomous Yes or No format. We disregarded such recommendation, as the pilot study verified that students understood the scale labels (not at all true to completely true), as well as for issues related to measuring achievement goals.

In Part II of the Pilot Study Assessment Protocol, students pointed to similar items and passages evaluated as difficult to understand, which led to the exclusion of five items from the EMR-CL (Table 3) and two items from the EA-CL (Table 4).As Table 3 shows, most of the reworded EMR-CL items were based on the students’ notes, which referred to term substitution. The purpose of rewording EA-CL items was to improve their writing. After this study, the EMR-CL had 28 items and the EA-CL 22 items.

Discussion

Results show that EMR-CL and EA-CL have evidence of content validity. Regarding EMR-CL, the experts’ analysis supported proposing three achievement goals directed at reading comprehension activities, as observed in other instruments that assess this construct for more general school situations (Bardach et al., 2020; Ferraz et al., 2019; Santos et al., 2018; Zambon & Rose, 2012). The judges also considered the actions and behaviors of the EA-CL items valid, as aspects integrated into self-regulation for learning. Studies argue that an adequate level of self-efficacy, expressed by self-perceived competence compatible with the reality experienced by students (functional beliefs) acts positively on their motivation to perform reading comprehension activities (Klauda & Guthrie, 2014; Louick et al., 2016).

In the second part of this study, referring to target audience assessment, the students’ suggestion about changing the response key of the EMR-CL was disregarded. A decision informed by the confirmation that students understood the meaning of response keys and their functioning (American Educational Research Association [AERA] et al., 2014; Gehlbach & Artino, 2017). In EMR-CL and EA-CL, we considered that the four response options and their respective labels are necessary to assess SRL, for this format provides indicative of the gradation levels of the constructs assessed (Gehlbach & Artino, 2018), that is, it allows to identify the degree to which students adhere to a certain achievement goal and whether their self-efficacy beliefs for reading comprehension are more positive or negative (Zimmerman & Risemberg, 1997). As for the target audience’s specificities, the four-point scale established is compatible with the cognitive aspects mobilized in the response process of self-report scales. Considering that the students evaluated by EMR-CL and EA-CL are preadolescents and adolescents, this number of response keys matches their metacognitive capacity to answer this type of instrument (Gehlbach & Artino, 2018; Toland & Usher, 2016). Note that in the EA-CL, students recognized that the content of the items is compatible with the procedures adopted in reading comprehension tasks, related to the use of strategies, self-monitoring, time management, among others (Carroll & Fox, 2017; Klauda & Guthrie, 2014; Louick et al., 2016; Zimmerman & Risemberg, 1997).

Method

Participants

The study sample consisted of 522 students from 6th to 9th grade (Middle School) from three schools in the countryside of São Paulo, aged 10 to 18 years (M = 12.72; SD = 1.26), of which 53.64% (n = 280) were girls and 17.24% (n = 90) had a history of retention.

Instruments

Achievement Goals Scale for Reading Comprehension (EMR-CL). Scale comprising 28 items that aim to assess the achievement goals for reading comprehension - learning goal, performance-approach and performance-avoidance goals. Response format is a 4-point Likert type - Not at all true to Completely true.

Self-Efficacy Scale for Reading Comprehension (EA-CL). Scale comprising 22 items that assess the students’ self-efficacy beliefs to perform procedures that presuppose good performance in reading comprehension. Response format is a 4-point Likert type - Not at all capable to Fully capable.

Importantly, both scales make up the motivational dimension of the Multidimensional Battery of Self-Regulation for Reading Comprehension (Bateria Multidimensional da Autorregulação para a Compreensão de Leitura - BAMA-Leitura), construed by the authors of this article. BAMA-Leitura also has seven other scales that assess the following dimensions of self-regulation for reading comprehension: method (reading strategies), time management (planning and managing time for reading), self-perceived behavior (intrapersonal causal attributions for success and failure in reading comprehension, self-monitoring, and positive and negative self-reactions), physical environment (self-selection of physical environment and instructional devices for reading), and social environment (self-selection of good readers as models and selective help-seeking).

Data collection. Sample size was established by sample calculation performed based on the number of students enrolled in Middle School in the state of São Paulo disclosed by the 2017 School Census (N = 2254356) - confidence level of 95% and sampling error of 5%. In each of the schools, application of the EMR-CL and EA-CL took place collectively, during class. Among the 607 protocols collected, only students who had no history of learning disabilities were selected for the research. This information, obtained from each school office, led to the exclusion of 19 protocols. We also excluded 66 protocols that were incorrectly filled out or contained incomplete answers on one or more scales.

Data analysis. Data analysis was performed using Factor (Lorenzo-Seva & Ferrando, 2020), MPlus version 7.11 (Muthén & Muthén, 2012), JASP (Goss-Sampson, 2020) and R Studio Workbench.

Achievement Goals Scale for Reading Comprehension (EMR-CL). EMR-CL data was analyzed using Exploratory Structural Equation Modeling (ESEM). The number of factors was determined by the theoretical conceptualization of 1 x 2 achievement goals, using the Geomin oblique rotation method and Weighted Least Square Mean and Variance Adjusted (WSLMV) estimation method (Muthén & Muthén, 2012). Spearman’s rho correlation analysis (ρ) was applied after the ESEM to verify the correlations between the scale factors. To interpret the magnitude of correlations we established the following values: < .29, small; ≅ .30 and .49, moderate; ≤ .50, high (Goss-Sampson, 2020).

Self-Efficacy Scale for Reading Comprehension (EA-CL). EA-CL data was analyzed using factor retention and Parallel Analysis (PA), whose result was contrasted with the observance of the indicators: Unidimensional Congruence (UniCo) > .95, Explained Common Variance (ECV) > .85, and Mean of Item Residual Absolute Loadings (MIREAL) < .30, to confirm the single-factor structure of EA-CL (Damásio & Dutra, 2017). PA indicated the retention of a single factor with a mean variance for the real data of 42.69%, greater than the mean explained variance for the random data of 10.19%, with a 95% confidence interval. This structure was corroborated by the indicators UniCo = .95, ECV = .85, and MIREAL = .14. Fit of the single-factor model was investigated using Confirmatory Factor Analysis (CFA), with Weighted Least Square Mean and Variance Adjusted (WSLMV) estimation method (Muthén & Muthén, 2012).

Plausibility evaluation of the models generated by ESEM and CFA. Ratio of χ² /gl ≤ 5 and Root Mean Square Error of Approximation (RMSEA) fit indices between ≤ .05 and .10 (confidence interval [CI] 90%), Confirmatory Fit Index (CFI) and Tucker-Lewis Index (TLI) of ≥ .95 and .80 (Marôco, 014). Item exclusion criteria. Factor loadings below .40. In the EMR-CL, items that had cross-factor loadings above .40, on a different factor for which they were constructed, remained under the theoretical and empirical underpinning of personal achievement goals. Reliability estimation was investigated using the composite reliability index (CR), calculated in the Composite Reliability Calculator (http://www.thestatisticalmind.com/calculators/comprel/composite_reliability.htm).CC values > .70 were qualified as acceptable. Item discrimination parameter. Item Response Theory, with the mirt package, one-parameter model - discrimination (a).

Results

To obtain a plausible EMR-CL model, we tested three structures using ESEM. The first two analyses enerated the following indices: ESEM 1 (28 items), χ² /gl = 2.60 ; RMSEA = 0.05 (CI .051 - .060); CFI = .91; TLI = .88 - exclusion of seven items with factor loadings < .40; ESEM 2 (21 items), χ² /gl = 2.51; RMSEA = .05 (CI .046 - .060); CFI = .94; TLI = .91 - exclusion of one item with factor loading < .40. Table 5 presents the results of the third ESEM. Regarding factor structure, the EMR-CL with 20 items distributed into three factors proved to be plausible and presented adequate reliability indices. The scale factors showed to be correlated with each other: Learning Goal and Performance-Approach Goal, ρ = .33 (p < .001); Learning Goal and Performance-Avoidance Goal, ρ = -.27 (p < .001); and Performance-Approach Goal and Performance-Avoidance Goal, ρ = -.23 (p = .03).

Table 5
Internal Structure: Achievement Goals Scale and Self-Efficacy Scale for Reading Comprehension

Achievement Goals Scale for Reading Comprehension items	ESEM 3				Self-Efficacy Scale for Reading Comprehension items	CFA 2
	F1	F2	F3	a		F1	a
1. Understand what *** new things	.56	-.04	-.06	1.14	1. Understand *** more complicated	.63	1.51
2. I avoid reading texts *** understand the text	-.09	-.09	.61	1.23	2. Use good *** text	.65	1.48
3. I like *** my classmates	-.11	.78	.01	1.86	3. Elaborate good *** text	.59	1.29
4. Comprehend what I read *** in my everyday life	.48	.11	.01	1.07	4. Always find *** to read	.56	1.14
5. I enjoy the challenge *** complicated to read	-.01	.45	-.43	F2: .74; F3: -1.03	5. Understand the meaning *** other places	.54	1.11
6. When I understand *** do the lesson	.50	-.13	-.06	.91	6. Make associations *** in the text	.56	1.21
7. Reading comprehension *** do poorly in exams	.58	-.13	.09	.99	7. Know what the text is about *** reading	.69	1.69
8. Even when it is difficult *** the text	.17	.24	-.49	-1.43	8. Classify types *** or a poem	.54	1.06
9. I strive to understand *** good I am	-.03	.76	-.02	2.11	9. Imagine things *** in the text	.47	.92
10. I give up reading *** reading comprehension	-.12	.01	.62	1.50	10. Save the information *** text	.64	1.46
11. I try to read *** will think I am smart	-.03	.70	-.02	1.33	11. Associate information *** daily life	.56	1.17
12. I only read *** at school	-.08	-.16	.41	.96	12. Know if *** text	.58	1.30
13. Understand what I read *** classmates	.09	.62	.18	1.52	13. Understand what are *** reading comprehension	.48	.96
14. Understand the text *** school activities	.55	.12	.02	1.16	14. Share my *** classmates	.49	.93
15. I dislike it when *** I read	.14	.08	.44	.56	15. Read the texts *** by the teacher	.60	1.35
16. Understand a text *** about things	.64	.07	-.06	1.65	16. Associate the text *** I have read	.60	1.31
17. I like that people *** the text	.12	.51	.17	1.12	17. Ask *** hard to understand	.42	.76
18. I get nervous *** a difficult text	.15	-.01	.58	.96	18. Answer questions *** the teacher	.62	1.40
19. Reading comprehension *** school subjects	.77	-.01	-.02	2.23	19. Understand *** mathematical problems	.56	1.14
20. Reading comprehension *** of school	.82	.04	-.01	2.68	20. Understanding school subject books *** science	.60	1.35
Plausibility indices of the ESEM 3 model					21. Identify in what era *** I read	.56	1.19
					Plausibility indices of the CFA 2 model
χ2/gl = 2.20; RMSEA = .05 (CI .041 - .055); CFI = .94; TLI = .93					χ2/gl = 2.15; RMSEA = .05 (CI .041 - .053); CFI = .96; TLI = .96
Reliability Indices					Reliability Index
CR: F1 = .83; F2 = .81; F3 = .71					CR = .91

• Note. F1 = Factor 1, Learning Goal; F2 = Factor 2, Performance-Approach Goal; F3 = Factor 3, Performance-Avoidance Goal; a = discrimination parameter. Values in bold indicate factor loadings above .40. For access to the scales please contact the first author of this article.

Table 5 also shows the discrimination parameter values of the EMR-CL items. In the Learning Goal Factor, item 20 had the highest value, a = 2.68 (Reading comprehension helps me get higher grades in school subjects), while item 6 (When I understand what is written it is easier to understand the lesson) showed the lowest value, a = .91. In the Performance-Approach Goal Factor, item 9 (I strive to understand difficult texts so that my teacher and classmates see how good I am) obtained the highest degree of discrimination, a = 2.11; and item 5 (I enjoy the challenge of understanding texts more complicated to read), held the lowest value, a = .74. Finally, the Performance-Avoidance Goal Factor had a highest degree of discrimination in item 10 (I give up reading the text when it is of difficult reading comprehension), a = 1.50, while item 15 (I dislike when the teacher tells the whole class that I have difficulty understanding what I read) presented the lowest value, a = .56.

For the Self-Efficacy Scale for Reading Comprehension (EA-CL), we performed two confirmatory factor analyses to obtain an adequate model. The first CFA (22 items) showed the following values: χ² /gl = 2.39, RMSEA = .05 (CI .046 - .058), CFI = .95 and TLI = .94. We excluded item “Ask a classmate to help me understand the text” due to factor loading < .40. Table 5 summarizes the results of the second CFA. The χ² /gl ratio values and fit indices denote the plausibility of the model. The one-factor structure also showed an adequate reliability index. Regarding the discrimination parameter of EA-CL, we identified the highest value in item 7 (Know what the text is about after reading), a = 1.69, and the lowest value in item 17 (Ask the teacher for help when the text is difficult to understand), a = .76.

Discussion

Our results suggest that EMR-CL has validity evidence based on internal structure, as well as adequate reliability estimates. These findings corroborate the initial hypothesis of finding a three-factor solution - learning goal, performance-approach goal and performance-avoidance goal (Bardach et al., 2020; Ferraz et al., 2019; Santos et al., 2018; Zambon & Rose, 2012). We recommend that item 5 be counted in both the Performance-Approach Goal Factor (F2) and the Performance-Avoidance Goal Factor (F3). The negative sign in the factor loadings of items 5 and 8 (Table 5) indicate that their scores should be reversed when computing the scale results. Future studies with the EMR-CL should, however, analyze whether this configuration holds up, since the degree of discrimination for this item in the Performance-Approach Goal Factor was lower compared to the others, as well as in relation to the Performance-Avoidance Goal Factor.

It is worth discussing that five EMR-CL items presented factor loadings above .40 in factors other than the ones they were constructed for (items 5, 7, 8, 14, and 20; Table 5), with item 5 showing cross-factor loadings on Factor 2 and Factor 3. Item 5 originally represented the learning goal, but loaded on the factors Performance-Approach goal and Performance-Avoidance goal (F2 and F3). This may be explained by the mention to the challenge of reading more complex texts. Students guided by the performance-approach goal recognize this activity as highly valued, which increases the chances of achieving academic merit. In contrast, students guided by the performance-avoidance goal perceive this situation as aversive, which justifies the negative sign of the item on F3 (Bardach et al., 2020; Bzuneck & Boruchovitch, 2016).

Learning Goal Factor (F1) loaded item 7, built to represent the performance-avoidance goal, and items 14 and 20, designed for the performance-approach goal. These items emphasize students’ recognition of the relationship between reading comprehension and the completion of academic tasks, which includes assessment activities. We consider the disposition of these items in F1 relevant, as no features of the performance-approach goal are explicit in items 14 and 20, referring to competition and academic success as the end product of learning. Similarly, item 7 does not emphasize avoiding reading activities, a behavior identified in the performance-avoidance goal orientation, but rather on the relationship established by the student about the benefits of reading comprehension. We conjecture that the school climate is also associated with the students’ recognition of the demands for grades/school performance, making this type of concern part of the motivation for students guided by the learning goal, and not only restricted to the performance goals (Bardach et al., 2020; Ferraz et al., 2019).

Item 8, designed for the learning goal, received a negative factor load on the Performance-Avoidance Goal Factor (F3). The performance-avoidance goal does not correspond directly to demotivation, but to a potential risk factor for decreased motivation, since it is characterized by less persistence and interest in activities that require mobilizing greater efforts, which is contrary to the operating logic of the learning goal (Ferraz et al., 2019; Korpershoek et al., 2015).

Despite the correlations between the EMR-CL factors, the negative direction and low magnitude indices indicate that the performance-avoidance goal is only weakly related to the learning and performance-approach goals. This result corroborates the conception that these goals act alternately, that is, depending on how the student perceives the specificities of a given task, the orientation toward one performance goal may stand out while the characteristics of another goal are minimized or suppressed (Bzuneck & Boruchovitch, 2016). The positive and moderate magnitude correlation assumes that the convergence between the learning goal and the performance-approach goal can guide the student’s motivation concomitantly, due to their shared aspects (Bzuneck & Boruchovitch, 2016; Ferraz et al., 2019).

As for the EA-CL with 21 items, besides showing indicative validity evidence based on internal structure, also had a good reliability estimate. The scale’s final version assesses students’ level of self-efficacy regarding their reading comprehension and ability to use strategies that enable the success of this cognitive-linguistic skill, which includes time management and help-seeking. Due to the content assessed by the scale, we conjecture that high score are indicative of students’ mastery of the SRL and vice versa. Another point to be investigated in future studies is the relationship between self-efficacy and help-seeking with emphasis on the relationships established between peers, since in the EA-CL only item 17 remained, which referred to students’ self-efficacy to seek help from teachers. We assume that students’ self-efficacy beliefs to seek help are related to the quality of relationships established in the school climate, just as they are related to personal achievement goals (Bardach et al., 2020; Korpershoek et al., 2015; Richey et al., 2017). Accordingly, the low discrimination degree of this item signals the need to develop new studies with EA-CL to investigate its relevance in measuring self-efficacy for reading comprehension in Middle School students.

In conclusion, this study provides researchers and psychologists with two instruments of good initial psychometric properties for assessing Middle School students’ SRL motivational component, focusing on reading comprehension. To extend their validity evidence, future research with the EMR-CL and EA-CL envisions investigating the relations of these scales with reading comprehension performance and intrapersonal causality attributions (prospective motivation).

As for limitations, we highlight the reduced representativeness of the sample, since the investigations focused on the content and internal structure of the EMR-CL and EA-CL encompassed only students from the countryside of São Paulo. Regarding the analysis of the scales’ internal structure, we recognize the restrictions of factor analysis due to the dependence on sample characteristics. These aspects highlight the importance of conducting new studies with the EMR-CL and EA-CL, considering diversifying and expanding the sample and applying methodological resources from the Item Response Theory.

Finally, we reiterate that to achieve self-regulation for reading comprehension one must know the motivational quality of students, and that using measurement instruments with good psychometric properties is a viable way to conduct investigations of this nature. Consequently, the EMR-CL and EA-CL are promising tools for psychological and educational assessment in the Brazilian educational context.

References

• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). The standards for educational and psychological testing. Washington, DC: AERA.
• Bardach, L., Oczlon, S., Pietschnig, J., & Lüftenegger, M. (2020). Has achievement goal theory been right? A meta-analysis of the relation between goal structures and personal achievement goals. Journal of Educational Psychology, 112(6), 1197-1220. doi:10.1037/edu0000419
  » https://doi.org/10.1037/edu0000419
• Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687-699. doi:10.1177/001316448104100307
  » https://doi.org/10.1177/001316448104100307
• Bzuneck, J. A., & Boruchovitch, E. (2016). Motivação e autorregulação da motivação no contexto educativo [Motivation and self-regulation of motivation in the educational context]. Psicologia Ensino & Formação, 7(2), 73-84. doi:10.21826/2179-58002016727584
  » https://doi.org/10.21826/2179-58002016727584
• Carroll, J. M., & Fox, A. C. (2017). Reading self-efficacy predicts word reading but not comprehension in both girls and boys. Frontiers in Psychology, 7(e2056), 1-9. doi:10.3389/fpsyg.2016.02056
  » https://doi.org/10.3389/fpsyg.2016.02056
• Damásio, B. F., & Dutra, D. F. (2017). Análise fatorial exploratória: Um tutorial com o software Factor. [Exploratory Factor Analysis: A tutorial with the Factor software.] In B. F. Damásio & J. C. Borsa (Orgs.), Manual de desenvolvimento de instrumentos psicológicos [Psychological Instrument Development Manual] (pp. 241-266). São Paulo, SP: Vetor.
• Ferraz, A. S., Cantalice, L. M., & Santos, A. A. A. (2019). Motivação para aprender e compreensão de leitura em alunos do Ensino Fundamental I [Learning motivation and reading comprehension in elementary school students]. Estudos Interdisciplinares em Psicologia, 10(1), 173-189. doi:10.5433/2236-6407.2019v10n1p173
  » https://doi.org/10.5433/2236-6407.2019v10n1p173
• Gehlbach, H., & Artino, A. R., Jr. (2018). The survey checklist (manifesto). Academic Medicine, 93(3), 360-366. doi: 10.1097/ACM.0000000000002083
  » https://doi.org/10.1097/ACM.0000000000002083
• Goss-Sampson, M. A. (2020). Statistical analysis in JASP: A guide for students (4th ed.). Retrieved from https://jasp-stats.org/wp-content/uploads/2020/11/Statistical-Analysis-in-JASP-A-Students-Guide-v14-Nov2020.pdf
  » https://jasp-stats.org/wp-content/uploads/2020/11/Statistical-Analysis-in-JASP-A-Students-Guide-v14-Nov2020.pdf
• Hernández Nieto, R. A. (2002). Contributions to statistical analysis: The coefficients of proportional variance, content validity and kappa. Mérida, Venezuela: Universidad de los Andes.
• Klauda, S. L., & Guthrie, J. T. (2014). Comparing relations of motivation, engagement, and achievement among struggling and advanced adolescent readers. Reading and Writing, 28(2), 239-269. doi:10.1007/s11145-014-9523-2
  » https://doi.org/10.1007/s11145-014-9523-2
• Korpershoek, H., Kuyper, H., & van der Werf, M. P. C. (2015). Differences in students’ school motivation: A latent class modelling approach. Social Psychology of Education, 18(1), 137-163. doi:10.1007/s11218-014-9274-6
  » https://doi.org/10.1007/s11218-014-9274-6
• Lorenzo-Seva, U., & Ferrando, P. J. (2020). Factor Retrieved from https://psico.fcep.urv.cat/utilitats/factor/index.html
  » https://psico.fcep.urv.cat/utilitats/factor/index.html
• Louick, R., Leider, C. M., Daley, S. G., Proctor, C. P., & Gardner, G. L. (2016). Motivation for reading among struggling middle school readers: A mixed methods study. Learning and Individual Differences, 49, 260-269. doi:10.1016/j.lindif.2016.06.027
  » https://doi.org/10.1016/j.lindif.2016.06.027
• Marôco, J. (2014). Análise de equações estruturais: Fundamentos teóricos, software & aplicações. [Structural Equation Analysis: Theoretical foundations, software & applications.] Pêro Pinheiro, Portugal: ReportNumber.
• Muthén, L. K., & Muthén, B. O. (2012). Mplus user’s guide: Statistical analysis with latent variables (7th ed.). Los Angeles, CA: Authors.
• Richey, J. E., Bernacki, M. L., Belenky, D. M., & Nokes-Malach, T. J. (2018). Comparing class- and task-level measures of achievement goals. The Journal of Experimental Education, 86(4), 560-578. doi:10.1080/00220973.2017.1386155
  » https://doi.org/10.1080/00220973.2017.1386155
• Santos, A. A. A., Moraes, M. S., & Lima, T. H. (2018). Compreensão de leitura e motivação para aprendizagem de alunos do ensino fundamental [Reading and motivation understanding for learning of fundamental teaching students]. Psicologia Escolar e Educacional, 22(1), 93-101. doi:10.1590/2175-35392018012208
  » https://doi.org/10.1590/2175-35392018012208
• Toland, M. D., & Usher, E. L. (2016). Assessing mathematics self-efficacy: How many categories do we really need? The Journal of Early Adolescence, 36(7), 932-960. doi:10.1177/0272431615588952
  » https://doi.org/10.1177/0272431615588952
• Zambon, M. P., & Rose, T. M. S. (2012). Motivação de alunos do ensino fundamental: Relações entre rendimento acadêmico, autoconceito, atribuições de causalidade e metas de realização [Student motivation in fundamental education: Relations between academic performance, self-image, attributions of causality and goals]. Educação e Pesquisa, 38(4), 965-980. doi:10.1590/S1517-97022012000400012
  » https://doi.org/10.1590/S1517-97022012000400012
• Zimmerman, B. J., & Risemberg, R. (1997). Self-Regulatory dimensions of academic learning and motivation. In G. D. Phye (Ed.), Handbook of academic learning: Construction of knowledege (pp. 105-125). Amsterdam, The Netherlands: Academic Press.

• Support:

  Fundação de Amparo à Pesquisa do Estado de São Paulo (Process nº 2018/19897-4).

Edited by

Associate Editor:

Luciana Mourão Cerqueira e Silva

Publication Dates

• Publication in this collection
  15 Nov 2021
• Date of issue
  2021

History

• Received
  31 Mar 2021
• Reviewed
  24 June 2021
• Reviewed
  22 July 2021
• Accepted
  22 July 2021