Validity evidence of the Social and Emotional Nationwide Assessment (SENNA 1.0) Inventory ¹

Evidências de Validade do Inventário Social and Emotional Nationwide Assessment (SENNA 1.0)

Evidencias de Validez del Inventario Social and Emotional Nationwide Assessment (SENNA 1.0)

Abstract:

Given the necessity of adequate instruments to measure socio-emotional skills, this study aimed to obtain validity evidence of the Social and Emotional Nationwide Assessment inventory (SENNA 1.0). The instrument was administered to a sample of 634 students (59% females) with a mean age of 16.3 years (SD = 1.21), from eight secondary schools of the Federal District ] of Brazil. Exploratory factor analysis indicated a six factor structure that explained 42.7% of the common variance, while confirmatory factor analysis and exploratory structural equational modeling analysis showed a moderate fit to the data. Reliability coefficients of the factor scores varied between .66 and .89. The coefficients of the convergent validity with the Reduced Scale of the Big Five Personality Factors (ER5FP) had a mean value of .59. In conclusion, the results indicate satisfactory evidence for the score validity of the SENNA 1.0 inventory.

Keywords:
personality traits; test validity; test reliability

Resumo:

Dada a necessidade de instrumentos adequados para a mensuração das competências socioemocionais, o objetivo deste estudo foi obter evidências de validade dos escores do inventário Social and Emotional Nationwide Assessment (SENNA 1.0). O instrumento foi administrado a 634 estudantes (59% mulheres) com uma média de idade de 16,3 anos (DP = 1,21) de oito escolas de ensino médio do Distrito Federal no Brasil. A análise fatorial exploratória indicou uma estrutura de seis fatores explicando 42,7% da variância comum, enquanto análise fatorial confirmatória e modelagem por equações estruturais exploratórias indicaram um ajuste moderado aos dados. Os coeficientes de fidedignidade dos escores fatoriais variaram entre 0,66 e 0,89. Os coeficientes de validade convergente com a Escala Reduzida de Cinco Grandes Fatores de Personalidade (ER5FP) tiveram um valor médio de 0,59. Em conclusão, os resultados sugerem evidências satisfatórias de validade do inventario SENNA 1.0.

Palavras-chave:
traços de personalidade; validade do teste; precisão do teste

Resumen:

Dado la necesidad de instrumentos adecuados para medir competencias socioemocionales, el estudio tuvo el objetivo de obtener evidencias de validez del inventario Social and Emotional Nationwide Assessment (SENNA 1.0). El instrumento fue aplicado a una muestra de 634 estudiantes (59% mujeres) con una edad promedio de 16,3 años (DE = 1,21), de ocho escuelas secundarias del Distrito Federal en Brasil. El análisis factorial exploratorio indicó una estructura de seis factores que explicó el 42,7% de varianza común, mientras que el análisis factorial confirmatorio y el modelaje por ecuaciones estructurales exploratorias mostraron un moderado ajuste a los datos. Los coeficientes de confiabilidad de los puntajes factoriales variaron entre 0,66 a 0,89. Los coeficientes de validez convergente con la Escala Reducida de Cinco Grandes Factores de Personalidad (ER5FP) tuvieron un valor promedio de 0,59. En conclusión, los resultados sugieren evidencias satisfactorias de validez del inventario SENNA 1.0.

Palabras clave:
rasgos de personalidad; validación de test; precisión de test

Twenty-first century challenges require changes in the educational system in order to develop various skills for the academic, professional, and personal success of children and youth. These skills involve the traditionally measured cognitive skills such as literacy and numeracy, which have been largely acknowledged. Nevertheless, there are other non-cognitive skills related to social and emotional learning that are not properly captured by traditional performance evaluations at schools, but have been lauded for their role in enhancing children’s and youth’s success in school and life (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011; Friedman & Kern, 2014; Organisation for Economic Co-operation and Development [OECD], 2015).

Zins and Elias (2006) defined socio-emotional learning as the process of acquiring the necessary knowledge, attitudes, and skills to recognize and manage emotions, care, and concern for others; make responsible decisions; establish positive relationships; and handle challenging situations. Similarly, socio-emotional skills (SEMS) have been defined by De Fruyt, Wille and John (2015) as individual characteristics that originate from the interaction between biological predispositions and environmental factors; which manifest as consistent patterns of thoughts, feelings, and behaviors; that are developed through formal and informal learning experiences; and influence the socioeconomic outcomes of individuals throughout their life.

In economics, psychology and education, several studies have shown meaningful and replicable associations between SEMS and various important life outcomes (Kautz, Heckman, Diris, ter Weel, & Borghans, 2014). A meta-analysis by Poropat (2009) showed that personality dimensions such as Conscientiousness are associated with school achievement. In addition, a more recent study by Duckworth, Tsukayama, and May (2011) provided evidence that supports the causal role of self-control in achievement measured by the GPA of students. Moreover, when compared with cognitive measures (IQ and achievement tests), personality measures are just as or more predictive than achievement tests (Almlund, Duckworth, Heckman, & Kautz, 2011). Specifically, there is growing empirical evidence for the contribution of personality traits including Conscientiousness and Neuroticism, on labor market outcomes, such as job performance and wages (Almlund et al., 2011), as well as health and well-being indicators across a person’s lifespan (Friedman & Kern, 2014).

For these reasons, the educational system can play a pivotal role in developing SEMS. There is now convincing evidence that school-based intervention programs can affect and moderate SEMS development and contribute to cognitive achievement (Durlak et al., 2011). However, including socio-emotional skills learning in schools constitutes a paradigm shift for the educational evaluation system, due to its traditional emphasis on the assessment of cognitive learning outcomes, with little systematic attention for other types of skills. The educational field therefore needs measurement tools to assess SEMS comprehensively and reliably in order to examine students’ individual differences in those skills and evaluate the impact of interventions on SEMS development (Duckworth & Yeager, 2015).

Given this necessity and with the intention of developing an integrative framework that identifies key dimensions of SEMS, the Institute Ayrton Senna conducted a project to develop the Social and Emotional Nationwide Assessment (SENNA 1.0) which was constructed by Primi, Santos, John, and De Fruyt (2016). The SENNA 1.0 inventory was developed from the inspection of the underlying structure of eight robust scales that are frequently used to measure socio-emotional skills in childhood and adolescence: Rosenberg Self-Esteem Scale (Rosenberg, 1979); Strengths and Difficulties Questionnaire-SDQ (Goodman, 1997); Big Five Inventory-BFI (John, Donahue, & Kentle, 1991); Self-Efficacy Questionnaire for Children (Muris, 2001); Big Five for Children- BFC (Barbaranelli, Caprara, Rabasca, & Pastorelli, 2003); Core Self Evaluations-CORE (Judge, Erez, Bono, & Thoresen, 2003); Grit Scale (Duckworth & Quinn, 2009) and the Nowick-Strickland Locus of Control Scale (Nowicki & Strickland, 1973).

The eight measurement scales were selected because they met four requirements (Primi et al., 2016): (a) the measured constructs should have predictive power over at least one educational or social outcome; (b) they should be easy to administer in terms of understandable language, low cost and short time; (c) the measured constructs should be malleable; (d) they should have robust psychometric properties.

Primi and colleagues (2016) examined the overlap and commonality existing across the more than 200 items of the eight scales with the idea of representing their common variance by a more manageable group of socio-emotional skills. The underlying dimensions were identified in a first study with a sample of 3,023 students from primary and secondary schools in Rio de Janeiro. Exploratory factor analyses with Geomin rotation showed that the structure of the instrument could be represented by a group of six dimensions, five of which showed strong parallels and could be easily interpreted from the Big Five personality framework.

The first factor, Conscientiousness, captures characteristics like perseverance, efficiency and efficacy. The second, Neuroticism, includes negative emotions such as anger, frustration, anxiety and sadness. The third is called Extraversion and includes skills like friendliness, sociability, self-confidence, enthusiasm, as well as self- efficacy and core self-evaluation. The fourth dimension, Agreeableness, captures skills like tolerance, modesty, friendliness, being sympathetic and acting prosocial. The fifth dimension, Openness to Experience includes creativity and curiosity skills, artistic and unconventional interests and fantasy. Finally, the sixth dimension, External Locus of Control (ELC), refers to low self-esteem and negative valence, reflecting the failure to have control over his/her life and execute actions.

In a replication study developed by Primi et al. (2016) with a large sample of students in Rio de Janeiro (N = 24,605), the SENNA 1.0 inventory showed adequate psychometric characteristics, as well as a robust factor structure. Given its objective to be useful and transferrable across schools in the different states of Brazil, it is crucial to evaluate the psychometric properties of the SENNA 1.0 inventory in a broader set of populations. Therefore, the current study aims to obtain validity evidence of the SENNA 1.0 inventory in the Federal District of Brazil. Specifically, evidence of construct validity will be obtained through an exploratory and confirmatory factor analysis and by the inspection of differences between men and women of the factors scores. In addition, convergent validity evidence will be assessed with a related assessment tool, the Reduced Scale of Big Five Personality Factors (ER5FP). Finally, internal consistency evidence of the scale scores will be obtained.

Method

Participants

Data was collected using a convenience sample obtained from seven secondary public schools and one secondary private school in the Federal District of Brazil. A total of 689 students participated, excluding those (8%) for which only partial data was available. The final sample consisted of 634 students (59% female) with a mean age of 16.3 years (SD = 1.21). Of these, 39.1% were in the first grade, 31.4% in the second grade and 29.5% in the third grade. Data collection took place in schools located in different areas of the Federal District: Santa Maria (25.1%), Asa Norte (24.3%), Gama (24.3%), Riacho Fundo II (21.9%) and Lago Norte (4.4%).

Instruments

Sociodemographic questionnaire (Primi et al., 2016). This questionnaire contains 29 multiple-choice questions about family environment, individual characteristics (sex, age, race, place of birth), parents’ and children’s attitudes toward studying and classroom characteristics.

SENNA 1.0 inventory (Primi et al., 2016). This self-report inventory assesses socio-emotional skills and was originally developed for Brazilian youth. It measures six dimensions: Conscientiousness, Neuroticism, Agreeableness, Openness to Experience, Extraversion, and External Locus of Control. The instrument consists of 92 items using a 5-point Likert scale (1 = “Nothing” and 5 = “Totally”). SENNA’s structure was already replicated in an independent sample of 24,605 respondents using Exploratory Structural Equation Modeling (ESEM) analyses. The results indicated an adequate fit of the six-factor model (CFI = .915; TLI = .903; RMSEA = .036) and also supported measurement invariance across different school grades, providing initial evidence for the instrument’s validity. Internal consistency coefficients of the six factors were all above .75.

In the present study, a shorter version of 83 items of the SENNA 1.0 inventory was used by suggestion of its authors. Nine items (external locus of control: 5; agreeableness: 2; extraversion: 1; openness to experience: 1) from the original 92 items of the instrument, were excluded, because of unsatisfactory psychometric characteristics like low factor loadings, and difficulties with their interpretation after factor analysis.

Reduced scale of big five personality factors (ER5FP) (Passos, 2014). The ER5FP was used to obtain evidence on the convergent validity of the SENNA 1.0 inventory. The instrument consists of 20 items and assesses the Big Five Personality Factors (Conscientiousness, Extraversion, Emotional Stability, Agreeableness, and Openness to Experience) using a 6 option semantic differential response scale. The ER5FP was created by Passos (2014) and was applied in the Federal District of Brazil, to a sample of 365 college students with a mean age of 29.5 years (SD = 8.6). Confirmatory factor analysis showed a five-factor structure with reliability coefficients (Lambda 2 of Guttman) between .71 and .85. The mean correlation between items across factors varied from .37 to .58, and the factor loadings were distributed between .62 and .77, indicating a model with robust data. A main advantage of the ER5FP is its short administration time.

Procedure

Data collection. The instruments were initially administered to a pilot sample to evaluate the adequacy of instruction, language and response time. After corrections were made, private and public secondary schools from different areas of the Federal District were contacted to present the study and ask for authorization to administer the instruments. Together with the school principals, the classrooms were selected for the data collection. Instructions were carefully explained to the students and the instruments were administered collectively to those who accepted to participate in the study.

Data analysis. Exploratory Factor Analysis (EFA) of the SENNA 1.0 inventory and ER5FP was performed using Principal Axis Factoring (PAF). Oblique rotation (Promax) was used considering the correlations between factors found in a previous study (Santos & Primi, 2014). For both instruments, the number of factors extracted in EFA was based on Parallel Analysis and theoretical considerations. Lambda 2 of Guttman (λ₂) was used as the reliability coefficient. The value .70 was established as an indication of adequate internal consistency.

In order to test the fit to the data, two analyses were performed: Exploratory Structural Equation Modeling (ESEM) and Confirmatory Factor Analysis (CFA). ESEM is an intermediate way between exploratory approaches and the confirmatory factor analysis. ESEM as CFA provides SEM parameter estimates, standard errors and goodness-of-fit statistics, but it differs from the CFA approach in that it imposes restrictions in the number of factors, but leaves their loadings free to be estimated for all extracted factors and not just for the principal factors (Marsh et al., 2010). This difference is important in the case of factor structures of personality traits, where the majority of trait variables tend to have loadings on more than one factor (De Raad & Mlacic, 2015). Both analyses were performed using Mplus 6.12 (L.K. Muthén & B.O. Muthén, 2012). The type of model estimator chosen was MLM for CFA analyses and MLR for ESEM.

The following indices were used to test the fit of the model: Comparative Fit Index (CFI); Tucker-Lewis index (TLI), Root Mean Square Error of Approximation (RMSEA) and the Standardized Root Mean Square Residual (SRMR). CFI and TLI values greater than .95 indicate a good fit of the model to the data. In the meanwhile, for the RMSEA and SRMR, a value of .06 and .08 or less, respectively, indicates a close fitting model (Ullman, 2006).

An independent-samples t-test was run to determine if there were significant differences between males and females in SENNA 1.0 factor scores. Convergent validity was assessed using Pearson correlations between SENNA 1.0 inventory and the ER5FP. Correlations between .50 and .70 were expected between the corresponding factors of these two instruments.

Ethical Considerations

This study was approved by the Ethics Committee of the Institute of Human Sciences of the University of Brasília (CAAE 38811314.6.0000.5540). All the ethical principles were respected, guaranteeing respondents the anonymity of their responses and voluntary participation.

Results

Assumptions required for univariate and multivariate statistical analysis were verified by exploring the data. The sample size was sufficiently large for exploratory factor analysis (Beavers et al., 2013) considering the moderate factor loadings SENNA 1.0 found in previous studies (Santos & Primi, 2014). The univariate normality assumption was not seriously violated as skewness and kurtosis in almost 90% of the variables were < 1.00 (Weston, Gore, Chan, & Catalano, 2008). In the meanwhile, multivariate normality assumptions were assessed using tests for skewness and kurtosis proposed by Mardia (1970) using the statistical programs Mplus 6.12. The tests were statistically significant, which indicates a violation of the assumption of multivariate normality. In the database no outliers were identified, while the percentage of missing cases was less than 2%. Missing cases were replaced using the linear trend at point method which is part of SPSS version 20.

The factorability of the matrix was corroborated and results supported the use of factor analysis (KMO = .88). Parallel analysis indicated eight factors with empirical eigenvalues higher than the 95th percentile of the randomly generated eigenvalues. Initially eight factors were extracted, but two of them were under-defined and not theoretically interpretable. The Hull method (Lorenzo-Seva, Timmerman, & Kiers, 2011) was used to determine the number of factors to be extracted. The analysis suggested retaining six factors. The six-factor solution proved to be the most comprehensive and interpretable one.

The six-factor solution was inspected to verify the need of item exclusion. The following criteria were used: (1) when an item showed a factor loading less than .32 on its principal factor, (2) when items had factor loadings >.32 on more than one factor with a difference less than .10, (3) when items showed a high factor loading on a different factor than theoretically expected. Using these criteria, 18 items were excluded. The final solution explained 42.7% of the total variance. Table 1 shows a summary of the factor structure of the SENNA 1.0 inventory with the factor loadings and communalities of the four items with the highest values of each factor. The total number of items, as well as the eigenvalues and reliability coefficients of the six factors considering all their items are also presented in Table 1. The items, originally in Portuguese, were translated for publication purposes. The full factor loading matrix in Portuguese can be obtained from the first author.

Table 1:
Highest Factor Loadings of SENNA 1.0 inventory (N = 634)

Item Description	NE	CO	OE	AG	EX	ELC	h2
I get angry easily.	.84	.15	-.04	-.09	.13	.06	68.3%
I get nervous easily.	.82	.08	-.06	-.03	.04	.10	68.3%
I often lose my temper.	.81	.06	-.02	-.11	.08	.07	66.0%
I lose my calm easily.	.75	.06	.05	-.13	.25	.01	57.4%
I am a hard working student.	.08	.72	-.03	.10	-.03	.01	52.2%
I do the tasks well and without wasting time.	.07	.72	.01	-.05	-.05	.10	44.7%
I am a careful and diligent student.	.12	.71	-.04	.12	-.04	.00	50.8%
I finish all my homework every day.	.00	.68	-.08	.00	.01	.02	43.0%
I have an active imagination.	-.07	-.07	.76	-.12	.03	-.11	49.4%
I have a great deal of imagination.	.02	-.04	.75	-.05	-.02	.02	52.2%
I have original and new ideas.	-.06	.05	.62	-.15	.14	.01	42.6%
I like to reflect, play with ideas.	.00	-.06	.58	.19	.03	-.09	43.3%
I am not selfish and like to help others.	-.10	.03	-.11	.62	.00	.00	37.7%
I like to cooperate with others.	-.08	.09	.10	.60	.08	-.05	52.5%
I am affectionate with my colleagues.	-.05	-.05	-.06	.56	.15	.00	33.8%
I try to help people who are sad or sick.	.13	.06	.06	.53	.09	.02	37.4%
I tend to be quiet*.	.12	-.23	-.16	-.03	.64	-.12	42.3%
I generate a lot of enthusiasm in others.	.09	.09	.01	.23	.61	.12	51.9%
I am reserved*.	.08	-.17	-.12	-.01	.60	-.20	39.3%
I am shy, inhibited*.	-.01	-.07	.06	-.09	.56	-.17	36.5%
Others accuse me of being a liar.	-.08	.02	-.05	-.19	.12	.54	26.8%
My classmates pick on me or bully me.	.03	-.05	-.08	.13	.03	.53	30.1%
I am blamed for things that weren’t my fault.	-.07	-.09	.05	-.13	-.01	.49	27.9%
I don’t decide on the TV channel at home.	-.07	.01	.01	.07	-.06	.42	18.9%
Number of ítems	12	14	11	11	10	7
Eigenvalues	9.56	5.54	4.31	3.46	2.54	1.89
Reliability coefficient (λ2)	.89	.88	.82	.81	.81	.66

• Notes. NE = Neuroticism, CO = Conscientiousness, OE = Openness to Experience, AG = Agreeableness, EX = Extraversion, ELC = External Locus of Control. Factor loadings over .32 appear in bold. Correlations between factors: NE-CO = -.25; NE-OE = -.10: NE-AG = -.19; NE-EX = -.11; NE-ELC = .24, CO-OE = .20, CO-AG .31; CO-EX = .12; CO-ELC = -.37; OE-AG = .41; OE-EX = .33; OE-ELC = .01; AG-EX = .39; AG-ELC = -.06, EX-ELC = -.18. *Inversed items.

All eigenvalues were higher than one and all factors showed adequate reliability coefficients above .80, except for the sixth factor - External Locus of Control (λ₂ = .66). The highest correlations were between Openness to Experience and Agreeableness (r = .41, p < .01) and Conscientiousness and External Locus of Control (r = -.37, p < .01).

Means and standard deviations of the six factor scores of the SENNA inventory, as well as mean differences between male and female students are presented in Table 2.

Table 2:
Summary of Goodness of Fit Indices of the Measurement Model of SENNA 1.0

Analysis	CFI	TLI	RMSEA (90% C.I.)	SRMR
CFA	.76	.75	.047 (.046 - .049)	.072
ESEM	.87	.84	.038 (.036 - .040)	.033

• Notes. CFA = confirmatory factor analysis; ESEM = exploratory structural equation modeling; CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; RMSEA = Root-Mean-Square Error of Approximation; SRMR = Standardized Root Mean Square Residual.

Female students showed statistically significant higher scores on Conscientiousness, Agreeableness and Neuroticism than male students. Male students had higher scores on Openness to Experience and External Locus of Control in comparison to female students. However, all mean differences presented a small effect size (Cohen’s d).

A summary of the goodness-of-fit indices of the model for CFA and ESEM is presented in Table 3. In the CFA and ESEM analyses, the resulting indices of RMSEA and SRMR indices reflected a close fit to the data, considering the expected maximum values of .06 and .08, respectively. In the case of the CFI and TLI indices, none of them reached the expected benchmark value (CFI ≥ .95; TLI ≥ .95), although ESEM indices (CFI = .87; TLI = .84) were notably higher than the CFA ones (CFI = .76; TLI = .75).

Table 3:
Mean Differences Between Men and Women on the Six Factors of the SENNA 1.0 Inventory

Factor	Total	Male	Female	df	t	Cohen’s d
	M (SD)	M(SD)	M(SD)
Neuroticism	3.21 (.63)	2.54 (.77)	2.86 (.80)	629	-5.08**	-.41
Conscientiousness	2.73 (.80)	3.09 (.62)	3.29 (.62)	629	-3.98**	-.32
Openness to Experience	3.42 (.65)	3.52 (.63)	3.35 (.66)	629	3.20**	.26
Agreeableness	3.52 (.60)	3.45 (.55)	3.57 (.62)	629	-2.35*	-.19
Extroversion	3.44 (.67)	3.49 (.62)	3.41 (.70)	629	1.64	.13
External Locus of Control	2.05 (.60)	2.19 (.64)	1.95 (.56)	629	5.00**	.41

• Note. *p < .05. **p < .01.

In the context of exploratory factor analysis of the ER5FP (KMO = .84), the final five-factor solution explained 46.9% of the total variance. Internal consistency coefficients of all factors were satisfactory except for Openness to Experience (λ₂ = .55). Table 4 presents summarized information of the five-factor solution of the ER5FP. The full factor loading matrix is available from the first author.

Table 4:
Extracted factors of the Reduced Scale of Big Five Personality Factors (ER5FP) (N = 634)

Factor	n ítems	Eigenvalue	Factor loadings (M)	Communalities (M)	λ2
Agreeableness	4	5.10	.54 - .80 (.68)	.38 - .59 (.49)	.76
Extroversion	4	2.04	.37 - .86 (.69)	.31 - .72 (.55)	.81
Conscientiousness	4	1.72	.43 - .73 (.60)	.39 - .54 (.44)	.73
Emotional Stability	3	1.20	.68 - .70 (.69)	.46 - .54 (.49)	.73
Openness to Experience	3	0.96	.32 - .62 (.46)	.22 - .47 (.30)	.55

• Note. Correlations between components: AG-EX = .33; AG-CO = .64; AG-ES = .39; AG-OE = .36; EX-CO = .43, EX-ES = .13, EX-OE .51; CO-ES = .25; CO-OE = .54; ES-OE = .17. The eigenvalues refer to the values before the rotation of the factors. The component loadings refer to the values after the rotation of the factors. Means of the factor loadings and communalities are between parentheses.

Male students (M = 3.98; SD = 1.30) showed significant higher mean scores on Emotional Stability than women (M = 3.46; SD = 1.35); t (629) = 4.84, p < .01, d = 0.39. Also, they (M = 4.17; SD = 1.12) had significant higher mean scores on Extraversion compared to female students (M = 3.91; SD = 1.18); t (629) = 2.79, p < .01, d = 0.23.

As for the convergent validity evidence (Table 5), the correlations between correspondent factors of the SENNA 1.0 inventory and ER5FP were strong in size and statistically significant. The lowest correlation was observed between the two Conscientiousness factors (r = .37, p < .01), while the two Extraversion factors showed the largest correlation (r = .81, p < .01). The Emotional Stability component of ER5FP showed a strong negative correlation with the Neuroticism factor of the SENNA 1.0 inventory (r = -.70, p < .01), which reflects their inverse conceptual interpretation.

Table 5:
Correlations Between the Six Factors of the SENNA Inventory and the Five Factors of the ER5FP

SENNA	ER5FP
	EX	AG	ES	CO	OE
Conscientiousness	.14 (.13)	.31 (.29)	.17 (.16)	.37 (.35)	.20 (.19)
Neuroticism	-.14 (-.13)	-.26 (-.25)	-.70 (-.66)	-.19 (-.18)	-.20 (-.19)
Openness to Experience	.24 (.22)	.13 (.12)	.10 (.09)	.23 (.21)	.51 (.46)
Agreeableness	.26 (.24)	.56 (.51)	.15 (.14)	.32 (.29)	. 27 (.25)
Extroversion	.81 (.73)	.32 (.29)	.06 (.05)	.39 (.35)	. 39 (.35)
External Locus of Control	-.18 (-.15)	-.14 (-.11)	-.07 (-.06)	-.28 (-.23)	-.10 (-.08)

• Note. EX = Extraversion; AG = Agreeableness; ES = Emotional Stability; CO = Conscientiousness; OE = Openness to Experience. The first values presented in the table are referring to correlations corrected for attenuation: the uncorrected correlations are between parentheses. All correlation values > = .09 are significant at the 1% level. All other values are significant at the 5% level.

Discussion

This study verified the reliability and validity of the SENNA 1.0 inventory based on data collected in the Federal District of Brazil. The results of the present study corroborate the results obtained in previous studies of the SENNA 1.0 inventory and confirm satisfactory validity evidence for the instrument. The six-factor structure of the SENNA 1.0 inventory was recovered in the sample of the study, although a group of items had to be eliminated. Thirteen items of the SENNA 1.0 inventory were excluded because of low factor loadings and shared loadings on two factors. The largest number of excluded items belonged to the factors Openness to Experience and Extraversion. In a previous study of the SENNA inventory, a total of 11 items of Openness to Experience, External Locus of Control and Agreeableness also presented some problems after a Differential Item Functioning (DIF) analysis but were not eliminated (Santos & Primi, 2014).

Similar to a previous study of the SENNA 1.0 inventory (Primi et al., 2016), the present research also showed some moderate correlations between factors, especially between Openness to Experience and Agreeableness; as well as Extraversion and Conscientiousness. This finding is in agreement with the McCrae et al. (2010) study, where some of the defining facets of Openness to Experience loaded in the factors of Extraversion and Conscientiousness. Indeed, as pointed out by De Raad and Mlacic (2015), the majority of trait variables in personality research tend to have substantial loadings on two or more factors.

In regards to the adequacy of data of the six-factor structure of the SENNA 1.0 inventory, the goodness-of-fit indices using ESEM were better than the ones using CFA. However, the values of CFI and TLI were lower than the expected benchmark values recommended by the literature (Ullman, 2006); while the RMSEA and SRMR indices did attain the benchmark values. In this regard, Marsh et al. (2010) warned that benchmark values (CFI ≥. 95 and TLI ≥ .95) should be considered as “rough guidelines or rules of thumb” in the case of instruments with large factor structures (e.g., instruments with at least 50 items and at least five factors), due to the fact that they are typically unable to satisfy the minimally acceptable standards of fit.

Reliability scores of the present study were, in general, satisfactory, which implies that the measurement of the factors of the instrument is repeatable. However, the factor with the lowest reliability was External Locus of Control, which also presented the lowest factor loadings. In a similar vein, in a previous study of the SENNA inventory (Santos & Primi, 2014), this factor, characterized by indicators of distress, ineffectiveness, and hopelessness; also presented unstable characteristics.

Several reasons could be hypothesized to explain the low reliability of the factors that are associated with the construction of the scale (Cohen, Swerdlik, & Sturman, 2014). For example, some of the External Locus of Control items, taken from the Nowick-Strickland Locus of Control Scale, had an original response scale of “yes” or “no” that was theoretically and empirically supported by the authors when the instrument was constructed (Nowicki & Strickland, 1973). The response scale was changed when the items were included in the SENNA 1.0 instrument, which could have increased the chances of measurement errors. In addition, reports from the data collection process of this study, indicated that some items were not well understood by some of the students and had to be explained by the administrators. Finally, some items were probably influenced by social desirability, due to their negative content (e.g. “I am filled with doubts about my competence”; “I feel that one of the best ways to handle most problems is just not to think about them”; “I am blamed for things that weren’t my fault”). Due to this evidence, it is recommended to revise the External Locus of Control items, in order to improve the psychometric characteristics of the factor.

Results of this study indicated gender differences with small effect sizes for Conscientiousness, Agreeableness and Neuroticism, with female students scoring higher than male students. The same results were found in a previous study of the SENNA 1.0 inventory where girls scored higher on the same factors, in addition to Extraversion (Santos & Primi, 2014). This has been corroborated in similar studies in the field of personality psychology, in research with a large cross-sectional sample conducted by Soto, John, Gosling, and Potter (2011). The authors found that females in mid-adolescence were more prone to anxiety and other negative emotions in comparison with males. The authors explained that during adolescence, girls experience more social and psychological difficulties, including gender expectations and stereotypes. This and other studies also found that in adolescence and early adulthood, girls tend to be more conscientious and agreeable than boys (McCrae et al., 2002; Soto et al., 2011).

Regarding convergent validity, the SENNA 1.0 inventory and the ER5FP presented moderate correlations between factors. The strongest correlations were between Extraversion and Agreeableness factors, and between Neuroticism and Emotional Stability; which indicates that these factors assess similar dimensions. Conscientiousness factors had the lowest correlation, but still moderate in size. The relationship between Openness to Experience factors of the two instruments was also moderate. However, this result should be interpreted cautiously, as in the process of validation of the ER5FP, the scores on this factor failed to present satisfactory psychometric characteristics, such as a low reliability coefficient and relative low factor loadings (Passos & Laros, 2015).

These results show that there is adequate evidence of convergent validity of the SENNA 1.0 inventory with ER5FP, although they are not as robust as expected. Overall, this confirms that the majority of the socio-emotional items of the inventory can be interpreted by the model of the Big Five Personality Framework. This supports the close relationship between socio-emotional skills measured by the SENNA 1.0 inventory and the underlying dimensions of the Big Five. This particular characteristic is claimed by its authors as an advantage, in the way that the Big Five Personality Framework contributes to structure the socio-emotional skills and facilitates their classification and interpretation (Santos & Primi, 2014).

The limitations of the study should be acknowledged. The study used a convenience sample mostly from public schools and the data presented a multivariate non-normal distribution. In terms of the data analysis, modifications indices were not used to improve the adequacy indices of the models, to avoid increasing the risk of type I and type II errors in the models (Ullman, 2006). However, future studies might consider applying model modifications or re-specifications with strong statistical and theoretical argumentations to improve the initial model. In addition, future studies that seek to examine the validity and reliability of the scale scores of the SENNA 1.0 inventory, should consider other statistical procedures like test-retest reliability assessments, evaluation of measurement invariance or a more detailed analysis of the items using Item Response Theory.

In conclusion, the results of the present study indicate that the SENNA 1.0 inventory presents adequate evidence of construct and convergent validity in the Federal District of Brazil. In this sense, a similar structure of six underlying dimensions has been reproduced in a different sample with different characteristics. However, some items presented weak psychometric characteristics, especially the ones from the External Locus of Control factor, the only dimension that supplements the other five that have a strong parallel with the Big Five personality factors. Considering this result, further analyses should be performed to contribute to the theoretical distinction between the socioemotional constructs measured by the SENNA 1.0 inventory and the personality constructs measured by other instruments. Finally, future studies should continue to obtain data on the psychometric properties of the instrument with complementary methods and considering the inclusion of diverse samples of the Brazilian population.

References

• Almlund, M., Duckworth, A. L., Heckman, J., & Kautz, T. (2011). Personality, and economics. In E. A. Hanushek, S. J. Machin, & L. Woessmann (Eds.), Handbook of the economics of education (Vol. 4, pp. 1-158). Amsterdam, The Netherlands: Elsevier B.V.
• Barbaranelli, C., Caprara, G. V., Rabasca, A., & Pastorelli, C. (2003). A questionnaire for measuring the Big Five in late childhood. Personality and Individual Differences, 34(4), 645-664. doi:10.1016/S0191-8869(02)00051-X
  » https://doi.org/10.1016/S0191-8869(02)00051-X
• Beavers, A. S., Lounsbury, J. W., Richards, J. K., Huck, S. W., Skolits, G. J., & Esquivel, S. L. (2013). Practical considerations for using exploratory factor analysis in educational research. Practical Assessment, Research & Evaluation, 18(6). Retrieved from http://pareonline.net/getvn.asp?v=18&n=6
  » http://pareonline.net/getvn.asp?v=18&n=6
• Cohen, R. J., Swerdlik, M. E., & Sturman, E. D. (2014). Testagem e avaliação psicológica: Introdução a testes e medidas [Psychological testing and assessment: An introduction to tests and measurements] (M. C. G. Monteiro, Trad., 8a ed.). Porto Alegre, RS: AMGH.
• De Fruyt, F., Wille, B., & John, O. P. (2015). Employability in the 21st century: Complex interactive) problem solving and other essential skills. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8(2), 276-281. doi:10.1017/iop.2015.33
  » https://doi.org/10.1017/iop.2015.33
• De Raad, B., & Mlacic, B. (2015). Big five factor model, theory and structure. In J. D. Wright (Ed.), International encyclopedia of the social & behavioral sciences (2nd ed., Vol. 2, pp. 559-566). Oxford, United Kingdom: Elsevier.
• Duckworth, A. L., & Quinn, P. D. (2009). Development and validation of the Short Grit Scale (Grit-S). Journal of Personality Assessment, 91(2), 166-174. doi:10.1080/00223890802634290
  » https://doi.org/10.1080/00223890802634290
• Duckworth, A. L., Tsukayama, E., & May, H. (2011). Establishing causality using longitudinal hierarchical linear modeling: An illustration predicting achievement from self-control. Social Psychology and Personality Science, 1(4), 311-317. doi:10.1177/1948550609359707
  » https://doi.org/10.1177/1948550609359707
• Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237-251. doi:10.3102/0013189X15584327
  » https://doi.org/10.3102/0013189X15584327
• Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405-432. doi:10.1111/j.1467-8624.2010.01564.x
  » https://doi.org/10.1111/j.1467-8624.2010.01564.x
• Friedman, H. S., & Kern, M. L. (2014). Personality, well-being, and health. Annual Review of Psychology, 65, 719-42. doi:10.1146/annurev-psych-010213-115123
  » https://doi.org/10.1146/annurev-psych-010213-115123
• Goodman R. (1997). The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38(5), 581-586. doi:10.1111/j.1469-7610.1997.tb01545.x
  » https://doi.org/10.1111/j.1469-7610.1997.tb01545.x
• John, O. P., Donahue, E. M., & Kentle, R. L. (1991). The Big Five Inventory-versions 4a and 54 Berkeley, CA: University of California/Institute of Personality and Social Research.
• Judge, T. A., Erez, A., Bono, J. E., & Thoresen, C. J. (2003). The core self-evaluations scale: Development of a measure. Personnel Psychology, 56(2), 303-331. doi:10.1111/j.1744-6570.2003.tb00152.x
  » https://doi.org/10.1111/j.1744-6570.2003.tb00152.x
• Kautz, T., Heckman, J. J., Diris, R., ter Weel, B., & Borghans, L. (2014). Fostering and measuring skills: Improving cognitive and non-cognitive skills to promote lifetime success. OECD Education Working Papers, 110, 87. doi:10.1787/5jxsr7vr78f7-en
  » https://doi.org/10.1787/5jxsr7vr78f7-en
• Lorenzo-Seva, U., Timmerman, M. E., & Kiers, H. A. (2011). The Hull method for selecting the number of common factors. Multivariate Behavioral Research, 46(2), 340-364. doi:10.1080/00273171.2011.564527
  » https://doi.org/10.1080/00273171.2011.564527
• Mardia, K. V. (1970). Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3), 519-530. doi:10.2307/2334770
  » https://doi.org/10.2307/2334770
• Marsh, H. W., Lüdtke, O., Muthén, B., Asparouhov, T., Morin, A. J. S., Trautwein, U., & Nagengast, B. (2010). A new look at the big five factor structure through exploratory structural equation modeling. Psychological Assessment, 22(3), 471-491. doi:10.1037/a0019227
  » https://doi.org/10.1037/a0019227
• McCrae, R. R., Costa, P. T., Jr., Terracciano, A., Parker, W. D., Mills, C. J., De Fruyt, F., & Mervielde, I. (2002). Personality trait development from age 12 to age 18: Longitudinal, cross-sectional, and cross-cultural analyses. Journal of Personality and Social Psychology, 83(6), 1456-1468. doi:10.1037/0022-3514.83.6.1456
  » https://doi.org/10.1037/0022-3514.83.6.1456
• McCrae, R. R., Terracciano, A., De Fruyt, F., De Bolle, M., Gelfand, M. J., & Costa, P. T., Jr. (2010). The validity and structure of culture-level personality scores: Data from ratings of young adolescents. Journal of Personality, 78(3), 815-838. doi:10.1111/j.1467-6494.2010.00634.x
  » https://doi.org/10.1111/j.1467-6494.2010.00634.x
• Muris, P. (2001). A brief questionnaire for measuring self-efficacy in youths. Journal of Psychopathology and Behavioral Assessment, 23(3), 145-149. doi:10.1023/A:1010961119608
  » https://doi.org/10.1023/A:1010961119608
• Muthén, L. K., & Muthén, B. O. (2012). Mplus user’s guide Los Angeles, CA: Author.
• Nowicki, S., & Strickland, B. R. (1973). A locus of control scale for children. Journal of Consulting and Clinical Psychology, 40(1), 148-154. doi:10.1037/h0033978
  » https://doi.org/10.1037/h0033978
• Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135(2), 322-338. doi:10.1037/a0014996
  » https://doi.org/10.1037/a0014996
• Organisation for Economic Co-operation and Development. (2015). Skills for social progress: The power of social and emotional skills [DX Reader version]. Paris, France: OECD.
• Passos, M. F. D. (2014). Elaboração e validação de escala de diferencial semântico para avaliação de personalidade [Elaboration and validation of the semantic differential scale for personality assessment] (Doctoral dissertation). Retrived from http://repositorio.unb.br/bitstream/10482/17993/1/2014_MariaFabianaDamasioPassos.pdf
  » http://repositorio.unb.br/bitstream/10482/17993/1/2014_MariaFabianaDamasioPassos.pdf
• Passos, M. F. D., & Laros, J. A. (2015). Construção de uma escala reduzida de Cinco Grandes Fatores de personalidade [Construction of a reduced scale of the Big Five Personality Factors]. Avaliação Psicológica, 14(1), 115-123. Retrieved from http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712015000100014& lng=pt&nrm=iso
  » http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1677-04712015000100014& lng=pt&nrm=iso
• Primi, R., Santos, D., John, O. P., & De Fruyt, F. (2016). Development of an inventory assessing social and emotional skills in Brazilian youth. European Journal of Psychological Assessment, 32(1), 5-16. doi:10.1027/1015-5759/a000343
  » https://doi.org/10.1027/1015-5759/a000343
• Rosenberg, M. (1979). Conceiving the self New York, NY: Basic Books.
• Santos, D., & Primi, R. (2014). Desenvolvimento socioemocional e aprendizado escolar: Uma proposta de mensuração para apoiar políticas públicas [Socioemotional development and academic performance: A measurement proposal to support public policies]. São Paulo, SP: Instituto Ayrton Senna.
• Soto, C. J., John, O. P., Gosling, S. D., & Potter, J. (2011). Age differences in personality traits from 10 to 65: Big five domains and facets in a large cross-sectional sample. Journal of Personality and Social Psychology, 100(2), 330-348. doi:10.1037/a0021717
  » https://doi.org/10.1037/a0021717
• Ullman, J. B. (2006). Structural equation modeling: Reviewing the basics and moving forward. Journal of Personality Assessment, 87(1), 35-50. doi:10.1207/s15327752jpa8701_03
  » https://doi.org/10.1207/s15327752jpa8701_03
• Weston, R., Gore, P. A., Jr., Chan, F., & Catalano, D. (2008). An introduction to using structural equation models in rehabilitation psychology. Rehabilitation Psychology, 53(3), 340-356. doi:10.1037/a0013039
  » https://doi.org/10.1037/a0013039
• Zins, J. E., & Elias, M. J. (2006). Social and emotional learning. In G. G. Bear & K. M. Minke (Eds.), Children’s needs III: Development, prevention, and intervention (pp. 1-13). Washington, DC: National Association of School Psychologists.

• ¹
  Paper deriving from the master dissertation carried out by the first author, under the advice of the second author, defended in the Graduate Program of Social, Work and Organizational Psychology at the University of Brasilia, Brazil. Support: Programa Estudantes-Convênio de Pós-Graduação - PEC-PG, CAPES/CNPq (CAPES/CNPq, Grant # nº 190324/2013-9).

• 4
  How to cite this article: Pancorbo, G., & Laros, J. A. (2017). Validity evidence of the Social and Emotional Nationwide Assessment (SENNA) inventory. Paidéia (Ribeirão Preto), 27(68), 339-347. doi: 10.1590/1982-43272768201712

Publication Dates

• Publication in this collection
  Sep-Dec 2017

History

• Received
  24 Sept 2015
• Reviewed
  06 June 2016
• Accepted
  14 Nov 2016