Development and validation of a socio-emotional skills assessment instrument for sixth grade of Primary Education in Uruguay

Desenvolvimento e validação de um instrumento para avaliação de habilidades socioemocionais no sexto ano do Ensino Fundamental no Uruguai

Abstract

The development and implementation of instruments with high psychometric standards for the assessment and monitoring of socio-emotional skills is considered an important step in the agenda for their promotion in educational contexts. This article describes the process behind the development and validation of a large-scale assessment instrument of socio-emotional skills – including interpersonal and intrapersonal skills and aspects related to motivation and self-regulation –, for sixth grade primary school-children in Uruguay. We present results regarding factor structure and internal consistency obtained in throughout three studies. We conclude that the final instrument presents proper psychometric qualities, including evidence of convergent validity. Nonetheless, it is recognized that the development of assessment instruments of this type requires continued efforts for its improvement based on a longitudinal and iterative review.

Keywords
Educational measurement; Emotions; Validation studies

Resumo

O desenvolvimento e a implementação de instrumentos de alta qualidade psicométrica para avaliação e monitoramento das competências socioemocionais são considerados um passo importante na agenda para promover as habilidades no contexto educacional. Este artigo descreve o processo de desenvolvimento e validação de um instrumento de avaliação em larga escala das habilidades socioemocionais – incluindo habilidades inter e intrapessoais, bem como aspectos relevantes da motivação e autorregulação – no sexto ciclo do Ensino Básico no Uruguai. Os resultados são apresentados quanto à sua estrutura fatorial e consistência interna obtidas em três estudos. Conclui-se que o instrumento final demonstra ter características psicométricas satisfatórias e validade convergente; no entanto, reconhece-se que o desenvolvimento instrumental deste tipo requer uma abordagem de melhoria contínua após uma revisão longitudinal e iterativa.

Palavras-chave
Avaliação educacional; Emoções; Estudos de validação

In recent years, educational assessment has mainly focused on the standardized evaluation of students’ educational achievement, focused on measuring reading or mathematical skills, based on the assumption that social and emotional aspects belong mainly to the private realm (García Retana, 2012). For some years, there has been an enhancement of the idea that social and emotional aspects not only belong to the educational field but are essential for meaningful learning and for the development of people throughout life (Duong & Bradshaw, 2017; García & Weiss, 2016; Organización para la Cooperación y el Desarrollo Económicos, 2014). There is large academic evidence supporting that Socio-Emotional Skills (SES) affect learning and academic performance, educational development, mental health, individual and collective well-being (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011; Farrington et al., 2012; Morrison-Gutman & Schoon, 2013; Taylor, Oberle, Durlak, & Weissberg, 2017). At the same time, they have the potential to mediate the negative effect of unfavorable socioeconomic conditions on academic performance (Instituto Nacional de Evaluación Educativa [INEEd], 2015; Kautz, Heckman, Diris, TerWeel, & Borghans, 2014; McCoy, Connors, Morris, Yoshikawa, & Friedman-Krauss, 2015).

Socio-Emotional Skills are the result of dynamic processes and can be significantly fostered in the school environment (Durlak et al., 2011; Greenberg et al., 2003; Heckman & Kautz, 2014; Taylor et al., 2017). The development and implementation of high-quality instruments for the evaluation and monitoring of socio-emotional skills is considered an important step with the agenda for the promotion of socio-emotional development in the educational context (Garcia & Weiss, 2016; Thomson et al., 2018; Weissberg, Durlak, Domitrovich, & Gullotta, 2015).

Cultural, contextual and linguistic factors can influence the different types of validity and reliability of assessment instruments, emphasizing the need for adaptation processes and cultural validation, involving different players in the educational system (Davidson et al., 2018; McKown, 2015; Zamarripa & Lerma, 2013).

In Uruguay, although progress has been made in the validation of instruments to evaluate specific socio-emotional competencies (Instituto Nacional de Evaluación Educativa [INEEd], 2018a), there is no instrument matching the recommendations for SES evaluation, according to McKown (2015), implying that it was: (1) based on a theoretical SES model; (2) informed by teachers; (3) and specific enough to evaluate an array of dimensions that allows the comprehensive and relevant evaluation of SES.

Socio-emotional skills assessments within Aristas’ framework

The Instituto Nacional de Evaluación Educativa(INEEd, National Institute of Educational Evaluation) of Uruguay proposed the development of a National Assessment of Educational Achievement system called Aristas, which aims to provide information for the design of evidence-based policies, from a broad conception of educational achievement that transcends the frontier of measuring reading, mathematics or scientific skills. It includes – among other components –, the evaluation of the SES of students in the sixth year of primary education as an indicator of some of the competences that the educational system intends to promote, as well as a potentially explanatory factor for reading and mathematics performance and the climate of coexistence and participation in educational centers.

Currently, there is a great diversity of frameworks to conceptually operationalize SES. Organizations such as the Organization for Economic Cooperation and Development define them as the individual capacities that are manifested in persistent patterns of thoughts, feelings and behaviors and that can be developed from formal and informal learning experiences that significantly influence the social and economic results of the individuals throughout their life (Organization for Economic Cooperation and Development, 2015). They can also be understood as affective, behavioral, cognitive and regulatory tools, which affect one’s emotional state and the relationship with others, and the understanding of the impact of actions on events and others. These are skills that come to play and interact to achieve goals, a key element in learning (Organization for Economic Cooperation and Development, 2015).

In the framework of Aristas, SES are defined as a set of cognitive, emotional and social tools that have the function of adapting the individual to the environment, and that facilitate personal development, social relations, learning and well-being (INEEd, 2018a; Zins & Elias, 2007). They become configured dynamically from the interaction of the individual with the context, through experiences and learning, and in this sense they are considered as the result of development processes, which extend throughout the entire life cycle (Johnson, 2008).

Our broad definition encompasses the wide variety of specific skills that have been addressed in the academic literature and the field of educational evaluation (INEEd, 2018a; Morrison-Gutman & Schoon, 2013). Three selection criteria were applied to determine the skills to be evaluated: their ability to be modified from specific interventions in the school environment (malleability criteria), their incidence on learning and school life, their relevance for the development and well-being of students. The summary of the evidence that supports our selection and grouping is reported in INEEd (2018a). Table 1 shows the taxonomy developed for our evaluation of socio-emotional skills in the sixth year of primary education and the corresponding definitions.

The objective of this paper is to present the validation of the INEEd socio-emotional skills questionnaire for sixth year primary school students. The instrument is operationalized at 3 levels, in accordance with the taxonomy reported in Table 1: the first level covers the SES subscales, the second level groups three scales (Interpersonal Skills, Intrapersonal Skills and Motivation and Self-regulation), and the third level refers to the full SES scale.

Method

Stage 1: Development of Items. Based on the conceptual decisions summarized in Table 1, a list of items was made for each of the selected SES, according to their operational definition. In the first instance, the items used in consolidated instruments were used. They were identified based on a systematic review of the literature, both in English and Spanish, and the items in English were translated into Spanish and back-translated. Seeking to address all the proposed constructs, in a second instance, the elaboration of own new, customized items was used. To obtain inputs for the linguistic and contextual adaptation of the items, the item proposal was submitted to experts, teachers and sixth-year students from primary school. The experts received a preliminary version, together with the operational definitions; they were requested to review the items according to the following criteria: understanding of the item and adequacy in the national context; relationship between the item and the dimension it intends to measure; sufficiency of the operationalization of the dimensions (Escobar-Pérez & Cuervo-Martínez, 2008). The systematization of these data led to a first adjustment of the instrument.

Table 1
Socio-emotional skills assessed by the Instituto Nacional de Evaluación Educativa in sixth year primary education school children

Learning Motivation and Self-Regulation: Skills focused on academic goals
Metacognitive self-regulation	Awareness and control of cognitive activities through planning, monitoring and continuous correction of cognitive activities during the execution of a task (based on Pintrich, Smith, García, & McKeachie, 1991).
Growth mindset	The fundamental belief that our basic qualities, such as intelligence, can be cultivated through effort (based on Dweck, 2010).
Intrinsic motivation	Participation in a task as an end in itself, for reasons such as defiance, curiosity, or mastery of a task or subject (based on Pintrich et al., 1991).
Academic perseverance	Commitment to academic tasks, focus and persistence in the pursuit of academic goals, despite obstacles, difficulties and distractions (based on Farrington et al., 2012).
Interpersonal Skills: Skills for constructive social interaction
Empathy	Ability to understand and share the emotional state of others and respond in a way compatible with it through perspective taking, recognition of emotions and their context (based on Dadds et al., 2008).
Relationship Skills	Socially acceptable behavioral skills that allow effective interaction with others (based on Gresham & Elliot, 2007).
Intrapersonal Skills: Skills for managing one’s emotions and reactions
Emotional Regulation	It refers to cognitive strategies for managing internal emotional information and regulating emotional expression. Strategies refer to resources such as the ability to divert attention, take perspective or reformulate emotional reaction (based on Garnefski & Kraaij, 2007).
Self Control	The ability to control impulsive reactions to both positive and negative situations in order to meet obligations and short-term goals. Therefore, it implies the ability to change responses and avoid undesirable behaviors in a given context (based on Bandy & Moore, 2010).

In a following stage, interviews and focus groups were conducted with 40 teachers from seven regions of the country to verify the linguistic and contextual adequacy of the instrument, as well as to highlight suggestions. Finally, the individual application of the instrument was carried out with 11 sixth-grade students, in public education centers in four departments of the country. Students were required to read each item, outloud voice, and to describe in their own words their understanding and indicate their responses and the rationale for their choice. Based on this experience, the final adjustments of the instrument were made.

Stage 2: Internal Structure and Validation. The application of the instrument was made online using the one computer per child public policy, in the classroom, within the curricular schedule, administered by INEEd applicators. As for ethical requirements, the requirements of the Law on Protection of Personal Data and Action of “Habeas Data” (nº 18.331) and Decree nº 379/008 on research with humans were followed. This research is framed on the National Educational Assessment, established at the Educational Law n° 18.437 and approved by the Ministry of Education and Culture. The application of the socio-emotional skills questionnaire was part of a battery of instruments applied in the same instance, as part of the pilot and final Aristas study (INEEd, 2018b).

Pilot phase

In the pilot phase, two studies were carried out, one in which 4,227 sixth-year students participated; they were distributed in 119 schools in three areas of the country (metropolitan, interior and border), in which Cronbach alphas ranged from, 47 to 0.82 in the total and subscales. After the first pilot, a version with modifications was carried out again to improve some subscales that showed psychometric difficulties (α = 0.47 for growth mindset and α = 0.50 for emotional regulation). The second pilot study was conducted with 307 sixth-year students from 11 schools in the metropolitan area selected for convenience. Improvements in the reliability of the aforementioned subscales were observed, showing acceptable values for the main study (α = 0.73 for growth mind-set and α = 0.65 for emotional regulation).

Main study

The final application of Aristas was performed in October and November 2017 (INEEd, 2018b). It was carried out by means of a national representative sample of sixth year primary school students, implementing a sample design with systematic random selection of class groups taking into account the size of the schools, the socioeconomic context and type of schools (INEEd, 2018b), n = 7662, with an equitable distribution between women and men (50.30% and 49.66% respectively), an equitable distribution by socioeconomic quintile (about 20.00% for each quintile), and 87.24% of students between 11 and 12 years of age.

In order to explore the convergent validity of our socio-emotional skills scales, a series of instruments developed within the Aristas framework were included in the design of this study. Indicators of social wellbeing in the educational center were evaluated through 2 scales developed by the INEEd (Alonso & Mels, 2018): bond with the teacher (7 items) and bond between peers (6 items), with a Likert scale of 1 (never/almost never) to 4 (always/almost always). Its psychometric characteristics in the sample were satisfactory (one-dimensionality, αof 0.85 and 0.87 respectively). Both these scales and the SES scales were calculated based on the Item Response Theory, with an average of 50 points and a deviation of ten points. The distribution of the scales and probability of response to the items are reported in the report prepared by INEEd (INEEd, 2018b).

Data Analysis

Exploratory factor analyses were performed using the Principal Component Analysis method. The construct validity of the instrument as a whole was explored, with Principal Component Analysis with oblimin rotation, retaining the factors with eigenvalues above 1. The cut-off point for acceptable factor loads used was 0.30 (Costello & Osborne, 2005), and the Cronbach’s alpha was calculated for the subscales (level 1), the Interpersonal Skills, Intrapersonal Skills and Motivation and Self-regulation scales (level 2), as well as for the full scale (level 3).

Pearson correlations were calculated for all scales and subscales of the SES instrument. The following theoretical associations of the scales were investigated as an indicator of their convergent validity, using Pearson’s correlations: students’ bond with their teacher, students’ bond with their classmates, performance in mathematics, performance in reading. In all cases, a positive and statistically significant correlation was expected between the SES scales and the indicators of social well-being and academic performance.

Results

Results Principal Component Analysis: The Principal Component Analysis retains 8 components with eigenvalues greater than 1, accounting for 46.02% of the variance in the sample (Table 2). The first factor, corresponding to the Self-Control Scale, explains 16.55% of the variance. The factorial solution reproduces the theoretical structure of the instrument with one exception: the item “If my work doesn’t turn out well, I get upset and stop doing it” has a double load, both on the factor that corresponds to the Self-Control Scale (corresponding to its theoretical structure shows a load of 0.37), as in the Academic Perseverance Scale (negative load of -0.42). The other items are grouped according to the construct for which they were developed, showing good factor loadings ranging from 0.33 to 0.79. The Cronbach alpha values of the subscales (level 1) range satisfactorily between 0.65 and 0.74. At the second level of analysis (scales), alpha values of 0.60 are observed for motivation and self-regulation, 0.77 for interpesonal skills and 0.55 for intrapersonal skills. The SES total scale, meanwhile, shows good internal consistency (α = 0.76) (Table 3).

Table 2
Descriptives, factor loadings and internal consistency of the items

							Factorial load
Level 3	Level 2	Level 1	Item	n	M	SD	F1	F2	F3	F4	F5	F6	F7	F8
Habilidades socioemocionales (α = 0.76)	Motivation and Self-regulation (α = 0.60)	Metacognitive self-regulation (α = 0.74)	1	7029	2.96	0.95	0.58
			2	7029	2.95	0.96	0.51
			3	7029	3.28	0.85	0.57
			4	7029	3.48	0.77	0.68
			5	7029	3.46	0.76	0.66
			6	7029	3.43	0.76	0.58
		Growth mindset (α = 0.65)	7	7027	2.26	0.99		-0.78
			8	7027	2.30	0.96		-0.71
			9	7027	2.14	1.02		-0.79
		Intrinsic motivation (α = 0.65)	10	7034	3.50	0.75			-0.54
			11	7034	2.97	0.92			-0.71
			12	7034	2.69	0.99			-0.68
			13	7034	3.33	0.84			-0.54
			14	7034	3.58	0.69			-0.54
		Academic perseverance (α = 0.71)	15	7039	1.72	0.91				-0.73
			16	7039	1.56	0.82				-0.74
			17	7039	1.72	0.94				-0.65
			18	7039	2.01	0.96				-0.48
			19	7039	1.53	0.86				-0.48
	Interpersonal skills (α = 0.77)	Empathy (α = 0.70)	20	7033	3.38	0.78					-0.77
			21	7033	3.20	0.84					-0.72
			22	7033	3.37	0.84					-0.57
			23	7033	3.33	0.85					-0.41
			24	7033	3.33	0.77					-0.53
		Relationship Skills (α = 0.65)	25	7036	3.27	0.85						-0.66
			26	7036	3.33	0.85						-0.66
			27	7036	2.86	1.00						-0.60
			28	7036	2.89	0.94						-0.49
			29	7036	2.98	1.00						-0.33
			30	7036	2.53	1.12						-0.47
	Intrapersonal skills (α = 0.55)	Emotional Regulation (α = 0.67)	31	7031	2.73	1.01							0.63
			32	7031	2.77	0.98							0.49
			33	7031	2.94	1.06							0.67
			34	7031	2.71	1.00							0.65
			35	7031	2.99	1.02							0.75
		Self-control (α = 0.72)	36	7028	2.62	0.95								0.50
			37	7028	2.22	1.06								0.56
			38	7028	2.38	1.06								0.69
			39	7028	2.03	1.08								0.68
			40	7028	2.35	0.94								0.73
			41	7028	1.6	0.91				-0.42				0.37
		Eigenvalue					6.79	1.45	1.15	1.77	1.19	1.24	2.03	3.26
		% variance					16.55	3.53	2.80	4.33	2.90	3.01	4.96	7.94

• Note: M: Mean; SD: Standard Deviation; F1: Factor 1; F: Factor 2; F3: Factor 3; F4: Factor 4; F5: Factor 5; F6: Factor 6; F7: Factor 7; F8: Factor 8.

Table 3
Items number and statement

Item nº	Item
1	I check my homework to make sure I did it right.
2	When I’m studying, I ask myself questions to know if I’m understanding correctly.
3	When I have to do some work, I get organized (for example: I look for the materials).
4	I read the questions on a test carefully before I start answering.
5	If I don’t understand something, I read it again more carefully until it becomes clear.
6	When I realize that something is going wrong in my work, I correct it.
7	If you are smart you don’t need to work hard to do well in school.
8	Some children are just born smart and do better in school.
9	A good student does not need to make an effort to do well in school.
10	I study to learn.
11	I like to study.
12	I like difficult tasks because they are challenging.
13	If I get excited about a subject, I want to continue learning beyond what the class learns.
14	I like to learn new things.
15	If a task is too difficult, I leave it undone.
16	I abandon tasks before finishing them.
17	If a task is too long, I just do the easy parts.
18	I forget to do my homework.
19	I lose things they lent me.
20	I can tell when a classmate feels sad.
21	I know when a classmate is upset, even when he says nothing.
22	It bothers me when they treat a classmate badly.
23	I am happy when others are happy.
24	I help my colleagues when they have a problem.
25	I make friends easily.
26	I feel part of a group because we do things together.
27	At recess, I invite other children to do things together.
28	When something happens that I can’t solve, I ask for help.
29	When someone gets mad at me, I talk to fix things.
30	I tell my friends my feelings.
31	When I’m angry I think about something else.
32	I can calm down when I’m nervous.
33	When I’m sad about something, I try to think of something happy.
34	When something makes me upset, I try to think it’s not so serious.
35	When I’m sad I try to distract myself.
36	I am easily distracted in class.
37	It’s hard for me to wait my turn.
38	I get upset easily.
39	I do things without thinking (like hitting, insulting) but then I regret it.
40	In some situations, I act without thinking.
41	If my work doesn’t turn out well, I get upset and stop doing it.

Internal Correlations: At the level of the scales (level 2), moderate correlations of the Motivation and Self-regulation scale with the Interpersonal (r = 0.50) and Intrapersonal (r = 0.40) Skill Scales stand out. The latter show a weak correlation of 0.17. Internally, moderate to high correlations are observed between the subscales and their relevant scales, although two exceptions stand out. First, regarding the Intrapersonal Skills Scale, a very high correlation is observed with the Self-control subscale (r = 0.98) and a low correlation with the Emotional Regulation subscale (r = 0.31). Second, the Growth Mindset subscale shows a very weak correlation (r = 0.07) with the Motivation and Self-Regulation Scale, as well as with the other components that make up this scale. It only shows a correlation – albeit a weak one –, with the Academic Perseverance subscale (r = 0.20).

External Correlations: Interpersonal skills show low but considerable correlations, with the scales of bond with the teacher (r = 0.30) and bond between peers (r = 0.41). The Motivation and Self-Regulation Scale is also correlated with the bond with the teacher (r = 0.34). Regarding the correlations of the scales (level 2) with the scores obtained in the math and reading tests, its correlation – although weak –, with the Motivation and Self-Regulation Scale (r = 0.18 in both cases) stands out, being higher compared with the correlations of these variables with the Inter- and Intrapersonal Skills Scales (Table 4).

Table 4
Correlations between scales and indicators of well-being and scores on tests of academic achievement

	1	2	3	4	5	6	7
1. Interpersonal skills	-	17**	50**	30**	41**	09**	13**
2. Intrapersonal skills		-	40**	16**	18**	17**	17**
3. Motivation and self-regulation			-	34**	27**	18**	18**
4. Link with the teacher				-	42**	08**	08**
5. Link between parties					-	16**	14**
6. Math Score						-	64**
7. Reading Score							-

• Note:
• ^**
  p ≤ 0.01.

Discussion

This work tried to capture the process of development and validation a large-scale evaluation instrument of SES of sixth year primary school children in Uruguay. The SES evaluation questionnaire allows a first approach to a series of skills in our educational system, which will lead to awareness and promotion of the relevance of socio-emotional development as outcome and mediators of the learning processes fostered in school (Garcia & Weiss, 2016; Thomson et al., 2018; Weissberg et al., 2015).

The instrumental development process presented in this work led to an instrument with satisfactory psychometric characteristics, in terms of its factor structure and internal consistency of the full scale (α = 0.76), and the subscales (with α values ranging from, 0.65 and 0.74). The scale that groups Interpersonal Skills (α = 0.72) is also considered good and the scale that groups skills related to Learning Motivation and Self-regulation (α = 0.60) is acceptable. The internal consistency of the Intrapersonal Skills scale is considered insufficient (α = 0.55) and is due to the very low correlation between its two subscales. Therefore, the Self-control subscale explains almost all the variance of the Intrapersonal Skills Scale. Based on these data, it is suggested to review the Emotional Regulation subscale and to expand the specificity of its evaluation, which under its current conceptual definition is limited in scope, encompassing only the cognitive strategies of emotional regulation (INEEd, 2018a), which leaves out a series of strategies focused on the background of the emotional response (Gross, 1998), and can explain the relative psychometric weakness of this subscale. In addition, certain indicators of the scales convergent validity were observed, obviously in those that were more reliable, and positive correlations between Interpersonal Skills and indicators of social well-being such as the bond with teachers and with classmates (INEEd, 2018b; Wentzel, 1998). The Motivation and Self-regulation Scale, on the other hand, shows a considerable correlation with the relationship reported by the students with their teacher, which is consistent with the adjustment to the normative imperatives of school culture and the correct interaction according to social norms (INEEd, 2018b; Viscardi & Alonso, 2013). Both scales at the same time demonstrate a weak but positive relationship (ranging from r = 0.17 to 0.18) with the scores obtained on the math and reading tests. Considering the multi-causality of academic performance (Garcia & Weiss, 2016; Richardson, Abraham, & Bond, 2012), the observation of correlations of this size is considered relevant, highlighting, however, the need to deepen further studies through multilevel models and structural equations.

The data found yield relevant data for the development of this type of measurements. First, persistent difficulties encountered for the Growth Mindset ability are highlighted. We consider that these may be due to the fact that this construct, although it has several antecedents for educational intervention, has been rarely evaluated (Sisk, Burgoyne, Sun, Butler, & Macnamara, 2018); we therefore have few inputs for the development of our instrument (Dweck, Chiu, & Hong, 1995). In turn, the item qualitative validation stage revealed the students’ difficulties in understanding the concept of intelligence, being mainly associated with academic behaviors matching the definition of “a good student”.

Second, despite the initial conceptual definition of the constructs to be measured, in their operationalization a certain overlap between skills could not be avoided, resulting, among others, in the double factor load, indicating the relevance of the item “If my work doesn’t turn out well, I get upset and I stop doing it” for the measurement of the concept of Self-control, as well as for Perseverance (even with a negative load, as it corresponds conceptually). Academic perseverance implies putting into play a series of self-control processes for performing academic activities (Gordeeva et al., 2017), for which reason it is expected that this conceptual relationship also manifests itself in the psychometric results of the instrument. However, it highlights one of the challenges presented in the evaluation of SES, which should be – according to McKown (2015) –, specific enough to capture the SES essential for socio-emotional development, while remaining comprehensive.

Finally, throughout the process of elaboration and validation of the scales, difficulties were found with the inclusion of inverse items. These difficulties may be due to different aspects. On the one hand, Suárez-Alvarez et al. (2018) state that the cognitive processes required to answer inverse items, are usually different, benefiting participants with better language skills. On the other hand, there is the principle of acquiescence (the tendency to indicate according to the series of items on the scale) or attention problems to understand the subtlety of the inverse item (van Sonderen, Sanderman, & Coyne, 2013). The scale was not elaborated from the beginning to address these difficulties, for example using a balance between direct and inverse items, as suggested by Ferrando & Lorenzo-Seva (2010). Therefore, and considering this controversy, it was decided to use scales composed of items in a single sense to improve the psychometric quality of the instrument.

Limitations and implications

Despite the suggestions of some authors, it was not decided to use novel evaluation formats, which could eventually mitigate problems of response trends and social desirability. In this instance, the use of Likert scales in the self-report modality was chosen, being the most feasible form of evaluation of SES on a large scale. These characteristics are considered a methodological weakness of our instrument. It is understood that evaluations based on the opinions of multiple informants would be a more recommended evaluation format.

On the other hand, it is acknowledged that opting for a broad and comprehensive evaluation of SES, implied losing specificity and sensitivity in the evaluation of the constructs that compose them. The replicability of our instrument is limited by its target audience – sixth year primary school children in Uruguay – having been developed and validated specifically for this population, and taking into account that the socio-emotional skills are susceptible to the evaluated stage of development, as well as the social and cultural context in which they operate.

It is intended to carry out future studies of similar characteristics every 3 years, being part of a continuous evaluation system of the Uruguayan educational system, including also the evaluation in the third year of secondary education. Although the relevance of the construction of standards to contribute to the interpretation of data and the assessment of changes over time is recognized, this has not been the focus of our work.

Acknowledgments

We appreciate the collaboration of V. ANFITTI, V. FIGUEROA, D. PADULA, L. MORENO, J. SOCA, and C. HARETCHE in the studies presented in this work.

• Support: Instituto Nacional de Evaluación Educativa para la Evaluación Nacional de Logros Educativos (Ley nº 18.437) and Agencia Nacional de Investigación e Innovación y de la Fundación Centro Ceibal para el Estudio de las Tecnologías Digitales en la Educación (Code nº FSED_2_2018_1_150661).

How to cite this article

• Panizza, M. E., Cuevasanta, D., & Mels, C. (2020). Development and validation of a socio-emotional skills assessment instrument for sixth grade of Primary Education in Uruguay. Estudos de Psicologia (Campinas), 37, e190066. http://dx.doi.org/10.1590/1982-0275202037e190066

References

• Alonso, C., & Mels, C. (2018). Hacia la evaluación de convivencia, participación y derechos humanos en educación media. Anais do III Congreso Latinoamericano de Medición y Evaluación Educacional, Montevideo Recuperado el https://drive.google.com/file/d/1uZKf-i_u9VCrMFpF9pqW1Wm9okxkMU2J/view
  » https://drive.google.com/file/d/1uZKf-i_u9VCrMFpF9pqW1Wm9okxkMU2J/view
• Bandy, T., & Moore, K. (2010). Assessing self-regulation: a guide for out-of-school time program practitioners. Child Trends, 23, 1-8.
• Costello, A., & Osborne, J. (2005). Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment, Research and Evaluation,10(7), 1-9.
• Dadds, M. R., Hunter, K., Hawes, D. J., Frost, A. D. J., Vassallo, S., Bunn, P., … Masry, Y. (2008). A measure of cognitive and affective empathy in children using parent ratings. Child Psychiatry and Human Development, 39(2), 111-122.
• Davidson, L., Crowder, M., Gordon, R., Domitrovich, C., Brown, R., & Hayes, B. (2018). A continuous improvement approach to social and emotional competency measurement. Journal of Applied Developmental Psychology, 55, 93-106
• Duong, J., & Bradshaw, C. (2017). Links between contexts and middle to late childhood social-emotional development. American Journal of Community Psychology, 60(3-4), 538-554. http://dx.doi.org/10.1002/ajcp.12201
  » https://doi.org/10.1002/ajcp.12201
• Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: a meta-analysis of school-based universal interventions. Child Development, 82(1), 405-432.
• Dweck, C. (2010). Mind-sets and equitable education. Principal Leadership, 10(5), 26-29.
• Dweck, C. S., Chiu, C., & Hong, Y. (1995). Implicit theories and their role in judgments and reactions: a world from two perspectives. Psychological Inquiry, 6(4), 267-285. http:dx.doi.org/10.1207/s15327965pli0604_1
  » https://doi.org/10.1207/s15327965pli0604_1
• Escobar-Pérez, J., & Cuervo-Martínez, A. (2008). Validez de contenido y juicio de expertos: una aproximación a su utilización. Avancesen Medición, 6, 27-36
• Farrington, C., Roderick, M., Allensworth, E., Nagaoka, E., Keyes, T., Johnson, D., & Beechum, N. O. (2012). Teaching adolescents to become learners: the role of noncognitive factors in shaping school performance: a critical literature review Chicago: Consortium on Chicago School Research.
• Ferrando, P., & Lorenzo-Seva, U. (2010). Acquiescence as a source of bias and model and person misfit: a theoretical and empirical analysis. British Journal of Mathematical and Statistical Psychology, 63(2), 427-448. http://dx.doi.org/10.1348/000711009X470740
  » https://doi.org/10.1348/000711009X470740
• García Retana, J. A. (2012), La educación emocional, su importancia en el proceso de aprendizaje, Revista Educación, 36(1), 1-24.
• García, E., & Weiss, E. (2016). Making whole-child education the norm: how research and policy initiatives can make social and emotional skills a focal point of children’s education Washington: Economic Policy Institute.
• Garnefski, N., & Kraaij, V. (2007). Cognitive emotion regulation. psychometric features and prospective relationships with depression and anxiety in adults. European Journal of Psychological Assessment, 23(3), 141-149.
• Gordeeva, T. O., Osin, E. N., Suchkov, D. D., Ivanova, T. Y., Sychev, O. A., & Bobrov, V. V. (2017). Self-Control as a personal resource: determining its relationships to success, perseverance, and well-being. Russian Education & Society, 59(5-6), 231-255.
• Greenberg, M. T., Weissberg, R. P., O’Brien, M. U., Zins, J. E., Fredericks, L., Resnik, H., & Elias, M. J. (2003). Enhancing school-based prevention and youth development through coordinated social, emotional, and academic learning. American Psychologist, 58(6-7), 466-474. http://dx.doi.org/10.1037/0003-066X.58.6-7.466
  » https://doi.org/10.1037/0003-066X.58.6-7.466
• Gresham, F. M., & Elliot, S. (2007). Social skills rating system manual Mineápolis: American Guidance Service.
• Gross, J. J. (1998). The emerging field of emotion regulation: an integrative review. Review of General Psychology, 2(3), 271-299.
• Heckman, J., & Kautz, T. (2014). Fostering and measuring skills: interventions that improve character and cognition. Technical report Chicago: University of Chicago Press.
• Instituto Nacional de Evaluación Educativa. (2015). Habilidades no-cognitivas y desempeños en matemática entre los estudiantes uruguayos evaluados en PISA 2012 Montevideo: Autor.
• Instituto Nacional de Evaluación Educativa. (2018a). Aristas: marco de habilidades socioemocionales en sexto de educación primaria Montevideo: Autor.
• Instituto Nacional de Evaluación Educativa. (2018b). Aristas 2017: informe de resultados de tercero y sexto de educación primaria Montevideo: Autor.
• Johnson, E. S. (2008). Ecological systems and complexity theory: toward an alternative model of accountability in education. Complicity: An International Journal of Complexity and Education, 5(1), 1-10. http://dx.doi/10.29173/cmplct8777
  » https://doi.org/10.29173/cmplct8777
• Kautz, T., Heckman, J. J., Diris, R., TerWeel, B., & Borghans, L. (2014). Fostering and measuring skills: improving cognitive and non-cognitive skills to promote lifetime success Cambridge: National Bureau of Economic Research.
• McCoy, D. C., Connors, M. C., Morris, P. A., Yoshikawa, H., & Friedman-Krauss, A. H. (2015). Neighborhood economic disadvantage and children’s cognitive and social-emotional development: exploring head start classroom quality as a mediating mechanism. Early Childhood Research Quarterly, 32(3), 150-159. http://dx.doi.org/10.1016/j.ecresq.2015.04.003
  » https://doi.org/10.1016/j.ecresq.2015.04.003
• McKown, C. (2015). Challenges and opportunities in the direct assessment of children’s social and emotional comprehension. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, & T. P. Gulotta (Eds.), Handbook of social and emotional learning: research and practice New York: Guilford Press.
• Morrison-Gutman, L., & Schoon, I. (2013). The impact of non-cognitive skills on outcomes for young people: literature review London: Education Endowment Foundation.
• Organization for Economic Cooperation and Development. (2015). Skills for social progress: the power of social and emotional skills París: Author.
• Organización para la Cooperación y el Desarrollo Económicos. (2014). Skills for social progress: the power of social and emotional skills París: Author.
• Pintrich, P. R., Smith, D., García, T., & McKeachie, W. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ) Ann Arbor: National Center for Research to Improve Postsecondary Teaching and Learning.
• Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students’ academic performance: a systematic review and meta-analysis. Psychological Bulletin, 138(2), 353-387.
• Sisk, V., Burgoyne, A., Sun, J., Butler, J., & Macnamara, B. (2018). To what extent and under which circumstances are growth mind-sets important to academic achievement? two meta-analyses. Psychological Science, 29(4), 549-571.
• Suárez-Alvarez, J., Pedrosa, I., Lozano-Fernández, L. M., García-Cueto, E., Cuesta, M., & Muñiz, J. (2018). Using reversed items in Likert scales: a questionable practice. Psicothema, 30(2), 149-158.
• Taylor, R. D., Oberle, E., Durlak, J. A., & Weissberg, R. P. (2017). Promoting positive youth development through school-based social and emotional learning interventions: a meta-analysis of follow-up effects. Child Development, 88(4), 1156-1171.
• Thomson, K., Oberle, E., Gadermann, A., Guhn, M., Rowcliffe, P., & Schonert-Reichl, K., (2018). Measuring social-emotional development in middle childhood: the middle years development instrument. Journal of Applied Developmental Psychology, 55, 107-118. http://dx.doi.org/10.1016/j.appdev.2017.03.005
  » https://doi.org/10.1016/j.appdev.2017.03.005
• van Sonderen, E., Sanderman, R., & Coyne, J. (2013). Ineffectiveness of reverse wording of questionnaire items: let’s learn from cows in the rain. Plos One, 8(7), e68967. http://dx.doi.org/10.1371/journal.pone.0068967
  » https://doi.org/10.1371/journal.pone.0068967
• Viscardi, N., & Alonso, N. (2013). Gramática (s) de la convivencia: un examen a la cotidianidad escolar y la cultura política en la Educación Primaria y Media en Uruguay Montevideo: Administración Nacional de Educación Pública.
• Weissberg, R., Durlak, J., Domitrovich, C., & Gullotta, T. (2015). Social and emotional learning: past, present, and future. In J. Durlak, C. Domitrovich, R. Weissberg, & T. P. Gullotta (Eds.), Handbook for social and emotional learning New York: Guilford.
• Wentzel, K. (1998). Social relationships and motivation in middle school: the role of parents, teachers, and peers. Journal of Educational Psychology, 90(2), 202-209.
• Zamarripa, M., & Lerma, E. (2013). School-based assessment with Latina/ o children and adolescents. In L. Benuto (Ed.), Guide to psychological assessment with Hispanics New York: Springer.
• Zins, J. E., & Elias, M. J. (2007). Social and emotional learning: promoting the development of all students. Journal of Educational and Psychological Consultation, 17(2-3), 233-255.

Publication Dates

• Publication in this collection
  18 May 2020
• Date of issue
  2020

History

• Received
  20 May 2019
• Reviewed
  07 Nov 2019
• Accepted
  24 Mar 2020