Mapping self-report questionnaires for socio-emotional characteristics: What do they measure?

Mapeando instrumentos de autorrelato de competências socioemocionais: o que eles medem?

Abstract

Whereas the structure of individual differences in many social and emotional attributes is well understood in adults, much less work has been done in children and adolescents. The main goals of this research were to specify the major content domains that are assessed across multiple socioemotional instruments (self-esteem, grit, self-efficacy, strengths and difficulties, Big Five) in research in the United States and Europe, to test them in a less developed context with considerable educational challenges (Brazilian schools). We selected the five most promising instruments and studied their structure at the item level in a large sample of Brazilian school students (N = 3,023). The extracted factors to capture the major domains of child differences represented in these instruments closely resembled the Big Five personality dimensions. We discuss the contribution of our findings to the assessment of socio-emotional skills in education research, as well as limitations of the current study, and suggestions for future research.

Keywords
Big Five; Large-scale educational assessment; Socio-emotional learning; 21^st Century skills

Resumo

Enquanto a estrutura das diferenças individuais em muitos atributos sociais e emocionais está bem estabelecida em adultos, consideravelmente menos estudos foram conduzidos em crianças e adolescentes. Os objetivos do presente estudo foram especificar os principais domínios de conteúdo de múltiplos instrumentos socioemocionais (autoestima, garra, autoeficácia, forças e dificuldades e os cinco grandes fatores) usados em pesquisas nos Estados unidos da América e na Europa, e testá-los em um contexto muito menos desenvolvido e com consideráveis desafios educacionais, a saber, escolas brasileiras. Foram eleitos os cinco principais instrumentos socioemocionais utilizados internacionalmente, sendo investigada a sua estrutura fatorial em uma grande amostra de estudantes escolares (N = 3,023). Os fatores extraídos para representar os domínios socioemocionais dos instrumentos testados se mostraram bastante coerentes com as dimensões do modelo dos cinco grandes fatores. Discutem-se as contribuições do estudo na avaliação de competências socioemocionais na pesquisa educacional, bem como as limitações do estudo e sugestões para futuras investigações.

Palavras-chave
Cinco grandes fatores; Avaliação em larga escala; Aprendizagem socioemocional; Competências do Século 21

Learning is a valuable resource for societies. It produces the secondary abilities, that is, secondary new brain systems of abilities formed from cultural needs via schooling (Geary, 2007). Learning is multi-determined phenomena. For instance, Carroll’s (1963) model of school learning postulates that degree of learning is related to the extent students spend the amount of time needed to learn. He breaks down time needed as inversely related to cognitive ability and quality of instruction. Time spend is directly related to the opportunity to learn and socio-emotional factors such as desire to learn and perseverance – defined as the amount of time a learner is willing to be engaged actively in learning. This illustrates that learning is a complex product of cognitive, motivational, and socio-emotional factors as well as environmental socio-cultural factors (such as family and school, economy and culture), all of which can influence opportunities to learn and the quality of learning experiences.

Recent developments in the fields of education, economics, psychology, and neuroscience have emphasized that Social-Emotional skills (SE) are as important as cognitive measures in predicting not only learning but various important life outcomes (Organization for Economic Cooperation and Development [OECD], 2016). SE are also frequently referred to as 21^st Century Skills, personal characteristics that are considered crucial for individual’s life and work adaptation (Trilling & Fadel, 2009; Battelle for Kids, 2015). The topic of socio emotional skills has become increasingly important for education and policy makers. Several frameworks have proposed definitions, measures, interventions and applications in educational systems. Table 1 lists seven important frameworks that define constructs and measures for SE (lines 6 to 10): Organization for Economic Co-operation and Development (OECD, 2016), Collaborative for Academic, Social and Emotional Learning (2012), California Office to Reform Education (CORE, 2014), CHICAGO consortium (Farrington et al., 2012), National Research Council (2012), Partnership for 21^st Century Skills, and Strive Task Force on Measuring Social and Emotional Learning (2013).

Table 1
Socio-emotional Big Five Model and Its Relationship to Six Important Frameworks for Socio-emotional Skills

Socio-emotional frameworks	C	N	O	E	A
Socio-emotional Big Five model
Core themes (De Raad, & Shouwemburg, 1996)	Task oriented behavior, will to achieve.	Affect and emotional regulation.	Imagination, intellect, autonomy.	Expression of energy and power.	Love, aceptance, peacefulness interpersonal relationships.
Educational related features (Poropat, 2009 and De Raad, & Shouwemburg, 1996 )	Will to achieve, sustained effort and goal setting, compliance with and concentration on homework, learning- related time management and effort regulation.	Self-efficacy, confidence, adaptive mastery-oriented style in the face of obstacles and negative emotionality, perceived self control over outcomes.	Desire for learning, intellectual drive and engagement, interest, idea production, creativity and originality.	Energy and socialization that could facilitate and/or prevent learning.	Cooperation with learning processes, compliance with teacher instructions.
Frameworks
Organizational for Economic Co-operationa dn Development (OECD) Longitudinal Study of Skills Development in Cities	Achieving goals.	Managing emotions.		Working with others (engagement).	Working with others (getting along).
Colaborative for Academic, Social and Emotional Learning (CASEL)	Responsible decision-making, Self-management.	Self-management.	Self-awareness.		Relationship skills, Social awareness.
California Office to Reform Education (CORE)	Self-Management.	Growth Mindset, Self-Efficacy.			Social awareness
CHICAGO consorsium	Academic perseverance, learning strategies, academic behaviours.	Academic mindset.		Social Skills .	Social Skills.
National Research Council Review	Intra-personal: work ethic/conscientiousness Type 1 self-regulation (metacognitive skills, including forethought, performance, and self- reflection).	Intra-personal: Positive Core Self-Evaluation Type 2 self-regulation (self- monitoring, self-evaluation, self-reinforcement).	Intra-personal: intellectual openness.	Inter-personal: leadership.	Inter-personal: teamwork and collaboration.
Partnership for 21st Century Skills	Initiative and Self-Direction, Productivity and Accountability, Responsibility.	Flexibility and Adaptability.	Flexibility.	Leadership, Initiative.	Social and Cross-Cultural Skills.
Strive Together Network	Academic self-efficacy, Grit/ perseverance, Self-regulated learning/ study skills.	Growth mindset/ mastery orientation, Emotional competence.

• Note: A: Amity (Kindness); C: Conscien-tious self-Management; E: Engage-ment (Extra-version); N: Negative emotion regulation; O: Open-mindedness.

In the absence of a comprehensive and explicitly specified taxonomic framework, SEs have been operationalized by an amalgam of different constructs and measures as it is illustrated in Table 1. One potential problem is that different frameworks can use different “construct names” or measures that access one single psychological construct or propose one measure that comprises a complex mix of more than one psychological construct. The National Research Council (2012, p.25) describes this problem using the known term of “jingle-jangle fallacy” and points out that “today measurement experts continue to struggle with the question of whether various constructs represent different names for the same underlying psychological phenomenon or are truly different dimensions of human competence”.

For instance, Marsh, Hau, Artelt, Baumert, and Peschar (2006) reported a study of 14-scales measure of Students’ Approaches to Learning assessing broad SE skills, such as self-regulated learning, motivational preferences, self-regulated cognitions, effort and persistence and learning preferences used in the Program for International Student Assessment in 2000. Although 14 supposedly distinct SE skills were measured, some of them were highly overlapping (i.e., correlated), such as math self-concept and interest in math (r = 0.86), effort/persistence and self-efficacy (r = 0.73), or academic self-concept with self-efficacy (r = 0.72). Such results illustrate the fact that presumably distinct SE skills being measured might be indicators of a smaller set of underlying constructs.

But what constructs? What taxonomic framework could shed light on the underlying variables of SE skills? In line with recent research in the literature (Kyllonen, Lipnevich, Burrus, & Roberts, 2014; National Research Council, 2012; Roberts, Martin, & Olaru, 2015) we propose that a socio-emotional Big Five model is a potentially rich and useful taxonomic framework to organize the conceptual space of SE skills and measures (John & De Fruyt, 2016; Primi, Santos, John, & De Fruyt, 2016). As De Raad and Schouwenburg (1996) suggested, individual differences can be broadly classified into systems of knowing (cognition), feeling (affection), and willing (motivation). Individual performance in schools and work is determined by capacities to perform (cognition) and willingness to perform (motivation and affection), as well by opportunities to perform, which are determined by environmental factors – family, school, social and socio-economic resources (Poropat, 2009). Thus, according to De Fruyt, Wille, and John (2015) and John and De Fruyt (2016), SE skills refers to consistent patterns of thoughts, feelings and behaviors specially those derived from the domains of motivation and feeling – reason why some historically researchers have used the term non-cognitive factors – that influence important socioeconomic outcomes throughout the individual’s life. Also, SE skills are those traits that are potentially malleable and whose development is shaped by environmental factors, that is, formal and informal learning experiences.

This broad definition is borrowed from contemporary personality psychology and focused on those person attributes that can be taught and learned during the school years and that predict important outcome, both in school and later in life. By approximating the definition of SE skills with personality psychology, one could use the extensive knowledge about that Big Five taxonomic framework as a conceptual anchor space to help classify SE skills and measures. In the context of education, the Big Five domains organize a set of core themes of socio emotional functioning that allow us to locate distinct skills and anchor various conceptions and measures of SE skills in an integrated, multi-dimensional, conceptual space. The two first lines in Table 1 present these core themes for each of the five domains (i.e., each column): (C): involves Conscientious task oriented behavior that rely executive self-management; (N): involves regulation of Negative affect and emotions that afford resilience under stress; (O): involves aspect of Open-mindedness like imagination, intellect, autonomy, and interest in learning and change that challenge one’s current understandings; (E): involves the expression of energy, power, and engagement (with others); and finally, (A): involves Amity, the perception and acceptance of others as allies, friends, and teammates, as loving and loveable, and thus the quality, depth, and closeness of interpersonal relationships (John & De Fruyt, 2016).

In Table 1 we also present a preliminary conceptually derived classification of where the diverse set of SE skills proposed by the seven frameworks can be located within the conceptual space of the socio-emotional Big Five. But to what extent can these five domains capture similarities and differences among these broad sets of SE skills measured by the most important classificatory frameworks and their assessment instruments? One possible way to test this taxonomic model is to perform structural analyses of a broad set of measures developed or used by each of these different frameworks, and to test whether the proposed classifications indeed holds.

The main objective of this research was to perform structural equation modeling to investigate the self-report SE skill measures that are commonly used in education research. Specifically, we asked how much of the variance in these measures is related to the Big Five domains. Our main hypothesis is that these measures, even though they have substantively different names and theoretical origins, are substantially related, at the latent level, to marker measures of the Big Five.

Method

Participants

Participants were 3.023 students who attended one of 86 classes within 16 schools; these schools were selected to represent typical levels of performance on standardized achievement in the State of Rio de Janeiro in Brazil. Students were in the 5^th (N = 697), 6^th (N = 710), 9^th (N = 674), 10^th (N = 488), and 12^th (454) grades. They were recruited from two distinct educational systems in Brazil, one funded by the local municipality (student N = 2.081) and the other funded by the state (N = 942). The number of students who completed each of the instruments was as follows: Locus of Control: N = 1026, Big Five Inventory (BFI): N = 927, Strengths and Difficulties Questionnaire (SDQ): N = 1055, Self-efficacy: N = 1011, Grit: N = 985, Rosenberg Self-esteem: N = 602 (only 4-6^th) and CORE Self-Evaluation: N = 409 (only 10-12^th).

Measures: Six Self-Report Questionnaires

Six instruments⁶ were included in the present study: Locus of Control Scale (Nowicki & Strickland,1973); SDQ (Goodman, 1997); BFI (John, Donahue, & Kentle, 1991); Self-Efficacy Questionnaire for Children (SEQ-C) (Muris, 2001); Core Self-Evaluations (CORE) (Judge, Erez, Bono, & Thoresen, 2003); Grit Scale (Duckworth & Quinn, 2009). Santos and Primi (2014) carefully reviewed measures that assess social and emotional characteristics and were available in Brazil or could be translated into Portuguese. They included measures that satisfied four criteria (predictive power, feasibility, malleability and robustness). This provided an initial set of measures of personality traits, self-concepts, self-esteem, motivation, attitudes, and beliefs about the locus of control over events and personal outcomes (i.e., attributing control internally or externally), and socio-emotional adaptation issues.

Data Collection: Balanced incomplete block design

In order to conduct factor analysis, it is necessary to estimate the full inter-scale correlation matrix. Ideally this is accomplished by having all student participants answer all items of all instruments. However, in practice, with 199 items, this is not feasible because diminished concentration and fatigue are likely to occur when children have to answer even as small scales as 60 items (Santos & Primi, 2014).

In order to obtain a full correlation matrix, a Balanced Incomplete Block design was employed (Sailer, 2005). We created booklets of two instruments for 5th and 6th grade students and three instruments for the 9^th, 10^th and 12^th grades. We systematically produced combinations of instruments in booklets in a way to balance all pairwise combinations of the eight scales, such that each pair was included in at least one booklet, and instruments were equally distributed among students. These combinations generated 20 booklets for younger children and seven for older students. Each booklet had an average of 67 items.

When data were collected each student was administered one booklet. Booklets were systematically distributed within classes, always returning to the first booklet after the 20th student (younger children) or the 7^th student (older children) had received a booklet. This spiraling strategy produced equivalent random samples of respondents for each booklet allowing the calculation of the full inter-item correlation matrix. Power calculations indicated 85 students would provide power of 0.80 for the detection of correlations with a magnitude of 0.30 (Primi et al., 2016). The final samples answering each booklet ranged from 93 to 252.

Adaptation of instruments to Brazilian context and length of questionnaires

Translation of items and adaptation of instruments aimed to adapt item content to the Brazilian target culture in a manner that preserved the intended relationships between indicators and construct. We followed Borsa, Damásio, and Bandeira (2012) guidelines. To do so five steps were followed: (a) translation; (b) sensitivity analysis; (c) consensus translation; (d) pilot study and (e) back translation. A small study was conducted to estimate the optimum number of items that could be administered in a class period of 50 minutes. This study applied the six tests in a sample of 228 children (5^th grade: N = 60, 6th grade: N = 42 and 10^th grade: N = 126) to estimate the average time each student spent on each item. The main finding was that students answered 4.1 items per minute but, as expected, there was substantial age differences. Final conclusions were that (a) the slowest younger children would answer about 60 items; (b) the average-speed younger children up to 94 items; and (c) the typical 7^th and 8^th graders up to about 160 items. Because of that our booklets were limited to 60 items for 5^th graders, and to 120 items for 9^th to 12^th graders. Detailed descriptions of test adaptation and timing study can be found on Primi et al. (2016) and Santos and Primi (2014).

Data Analyses

Our rationale was to first investigate whether the known 5-factor structure of the BFI (John, Naumann, & Soto, 2008; Soto, John, Gosling, & Potter, 2008) would replicate in a Brazilian sample of children and adolescents in public schools. If so, we could then regress latent SE skills variables on the BFI-based Big Five factors using Structural Equation Models (SEM). We approached both questions using item parcels as indicators because we were not interested in individual items per se and are aware of the extensive literature showing that item-level analysis of socio-emotional and personality measures tend to be fraught with error, such as acquiescence bias (e.g., Soto et al., 2008). BFI items were summed on the basis of their inter-correlations and observed internal consistency, so that each Big Five factor had three parcels as indicators. This strategy is called Homogenous Item Composites (HIC) and selects items within a domain items that had similar level of endorsement and inter-item correlations producing homogenous and internally consistent subset of items.

As personality data tend to be complex and multidimensional in nature, we tested a constrained and an unconstrained model for the BFI as a means to establish the best measurement approach for the further analyses. In the fully constrained version, indicators were allowed to load only on its hypothesized factor (with no cross-loadings at all). This model was tested via Confirmatory Factor Analysis (CFA). In the unconstrained version, items were free to load on each one of the hypothesized five factors. This less prohibitive model was tested via Exploratory Structural Equation Modeling (ESEM). ESEM can be more suitable for multi-dimensional personality data where secondary item loadings are expected (Marsh et al., 2010), as is the case for cross-cultural analyses of BFI data (Benet-Martínez & John, 1998; Schmitt, Allik, McCrae, & Benet-Martinez, 2007). Accordingly, we hypothesized the ESEM model would provide a better fit to our data.

In a second step, we sought to map the SE skills measured by each of the other instruments onto the Big Five framework (as defined by the BFI analysis). To do so, a latent variable of each SE skill (again measured with item parcels) was regressed on the Big Five factors, connecting the two measurement models. By relying solely on latent variables for the regression analyses, we were able to model only the common, reliable variance of indicators of the SE skills and the Big Five. For instance, when modeling grit, three item parcels were used as indicators of a grit latent factor, which was in turn regressed on the Big Five. Similarly, parcels were used for modeling the sub-dimensions of self-efficacy, self-esteem, and strengths and difficulties. This approach is illustrated in Figure 1. We expected at least one of the Big Five factors would be able to explain each SE skill – namely, one dimension of grit and of external locus of control, 3 sub-dimensions of self-efficacy, four sub-dimensions of CORE self-evaluation, and five sub-dimensions of strengths and difficulties.

Figure 1
Structural Equation Models of interest. SE (socioemotional) latent variables measured by item parcels are regressed on the Big Five variables. For the sake of simplicity, the diagram depicts only two indicators (item parcels) per latent variable, and it suppresses error terms.

Given the non-normal distribution of indicator variables, as revealed by an exploratory analysis of data, we employed a robust estimator for correcting the chi-square test value, namely Maximum Likelihood Robust. Models were tested using Software Mplus 7.11 (L.K. Muthén & Muthén, 2014).

Results

Testing the Big Five Factor Structure in Brazilian Public School Kids

In a first step, we tested the hypothesized five-factor structure of the BFI using HIC item parcels. Items were summed on the basis of their inter-correlations and observed internal consistency, so that each Big Five factor had three parcels as indicators. We fitted two models to the data, a fully constrained and an unconstrained model. In the constrained version, HICs were specified to load only on their hypothesized factor (no cross-loadings allowed). This model was tested via CFA. In the unconstrained version, items were free to load on each one of the hypothesized five factors. This less prohibitive model was tested via Exploratory Structural Equation Modeling (ESEM). Given the non-normal distribution of indicator variables as revealed by an exploratory analysis of data, in all analyses we employed a robust estimator for correcting the qui-square test value, namely Maximum Likelihood Robust.

The constrained CFA model fitted the data poorly, c² (80) = 445.91, p < 0.001, Root Mean Square Error of Approximation (RMSEA) = 0.070, Confirmatory Fit Index (CFI) = 0.865, Tucker-Lewis Index (TLI) = 0.823, whereas the unconstrained ESEM achieved an excellent fit, c² (40) = 52.96, p = 0.082, RMSEA = 0.019, CFI = 0.995, TLI = 0.987. However, when we examined the cross-loadings that presumably caused the fully constrained model to provide a poor fit to the data, we found that all the cross-loadings were small in size (≤ 0.21) when estimated in the ESEM solution. As shown in Table 2, this ESEM model was entirely consistent with the Big Five factors. Factor inter-correlations ranged from almost null (0.02) to more substantial in size (0.50). Specifically, the A factor was more difficult to differentiate from “C” and “N”; all the other correlations remained below 0.32.

Table 2
Confirmatory factor analysis of the BFI using item parcels

Item Parcel	E	A	C	N	O
E_1	0.74*	-0.10	-0.18*	-0.00	0.03
E_2	0.60*	0.13	0.00	-0.05	0.08
E_3	0.48*	0.16	0.08	0.07	-0.07
A_1	-0.12	0.56*	-0.04	0.10	0.06
A_2	-0.00	0.70*	0.01	-0.11	0.04
A_3	0.20*	0.53*	0.05	0.06	-0.19*
C_1	-0.05	0.02	0.67*	-0.03	0.20*
C_2	-0.02	-0.02	0.71*	0.15*	-0.06
C_3	0.09*	0.01	0.65*	-0.01	0.21*
N_1	-0.19*	0.07	0.03	0.76*	-0.01
N_2	0.05	-0.08	-0.07	0.64*	0.01
N_3	0.11	0.12	0.08	0.40*	0.06
O_1	0.05	0.09	-0.02	0.00	0.62*
O_2	-0.02	0.01	0.04	-0.05	0.53*
O_3	0.02	-0.02	0.04	0.07	0.79*
Factor inter-correlations
E	-	0.08	0.02	0.07	0.31*
A		-	0.49*	0.50*	0.30
C			-	0.30*	0.25*
N				-	-0.08
O					-

• Note:
• ^*
  p < 0.05. Factor loadings - 0.40 are boldfaced. A: Agreeableness; C: Conscientiousness; E: Extraversion; N: Negative Emotion Regulation; O: Open-mindedness; BFI: Big Five Inventory.

Linking the Big Five Factors to Socio-emotional Skills at the Scale Level

Next, we proceeded to the mapping of SE skills onto the Big Five factors via SEM, connecting the ESEM model of the BFI and the measurement model of each SE skill via linear regression. Results from the analyses are presented in Table 3.

Table 3
Mapping of socio-emotional variables from major instruments onto the Big Five Factors

Tests Mapped	E	A	C	N	O	R 2
Grit
Grit	0.01	0.03	0.90*	0.05	-0.09	0.83
Locus
External Locus	-0.28*	-0.01	-0.20	-0.41*	-0.20	0.45
Self-efficacy
Achievement	0.00	-0.04	0.78*	-0.13	0.37*	0.82
Emotional	0.24*	0.03	0.07	0.57*	0.27*	0.57
Social	0.62*	0.26*	-0.07	0.12	0.30*	0.78
Self-evaluation (CORE)
S-Eff	0.08	-0.10	0.56*	0.24*	0.18	0.47
LC	0.14	-0.17	0.47*	0.05	0.05	0.21
N	0.32*	-0.09	0.03	0.56*	-0.33*	0.50
SE	0.39*	-0.19	0.29*	0.24	-0.22	0.28
Strengths and Difficulties (SDQ)
Conduct problems	0.16	-0.28	-0.15	-0.32*	0.10	0.37
Emot. symptoms	-0.12	0.51*	-0.14	-0.73*	-0.05	0.46
Hyperactivity	0.30*	0.21	-0.49*	-0.42*	-0.12	0.47
Peer Problems	-0.26*	-0.12	-0.07	-0.10	0.19	0.12
Prosocial Skills	0.22*	0.58*	0.15	-0.30	0.05	0.43

• Note:
• ^*
  p < .05.
• ** Additionally, we tested a model in which a general factor of self-esteem, measured by item parcels (GF, GS, LC, N and SE), was regressed on the Big Five factors. The model had a good approximate fit to the data, c²(97) = 204.86, p < 0.001, RMSEA = 0.030,CFI = 0.965, TLI = 0.938, and the standardized regression coefficients were: 0.48 (p < 0.05) for E, -0.24 (p > 0.05) for A, 0.53 (p < 0.05) for C, 0.48 (p < 0.05) for Em, and -0.22 (p > 0.05) for O. This general factor of self-esteem had 72% of variance accounted for by the Big Five factors. A: Amity; AC: Academic Self-Efficacy; CP: Conduct Problems; E: Engagement; EM: Emotional Self-Efficacy; ER: Emotional Resilience (Em: Emotional Stability); ES: Emotional Symptoms; HA: Hyperactivity; LC: Locus of Control; N: Neuroticism; O: Openness; PP: Peer Problems; PS: Pro-Sociality; S: Self Management (C Conscientiousness); SE: Self Esteem; S-Eff: Generalized Self Efficacy; SC: Social Self-Efficacy.

As hypothesized, SE skills – grit, locus of control, self-efficacy, self-esteem, and strengths and difficulties –, were linearly related to the Big Five personality factors. Accounted variance of SE skills by the Big Five factors ranged from a low of 12% for SDQ Peer Problems to 83% for Grit, with a mean R² of 0.50. Grit was strongly and uniquely associated with “C”, and r was estimated at 0.90 suggesting that Grit shared virtually all its reliable variance with the “C” factor.

As for locus of control, self-efficacy, self-evaluation, and strengths and difficulties, the analyses indicated a specific profile of Big Five factors as predictors of each SE skill. External locus of control was explained by both low “E” and low “N” (i.e., low levels of Negative Emotion Regulation). The three self-efficacy scales were all related modestly (about 0.30) to “O”, consistent with the “can do” belief associated with all self-efficacy measures; however, each scale also showed a substantial correlation with one of the Big Five that differed by the domain in which self-efficacy beliefs were held: academic self-efficacy was highly related to “C” (0.78), emotional self-efficacy to “N” (i.e., negative emotion regulation that is effective), and social self-efficacy to the two interpersonal dimensions in the Big Five, primarily to “E” (0.62) but also to “A” (0.26).

The CORE elements of Self-evaluation represented various mixes of “C”, “N”, and “E”, which means that Brazilian school children with positive CORE self-evaluations tended to see themselves as dedicated and hardworking students (C) that regulate negative emotions well (N) and are able to engage at school, with their teachers and their peers.

The dimensions from the SDQ revealed a nuanced picture. Prosocial skills (the only non-problem on the SDQ) were related only to the two interpersonal Big Five factors, primarily “A” and to a lesser extent to “E”, whereas Peer problems were related solely to low “E”, indicating this scale captures shy and withdrawn kids who struggle with engaging with their peers and initiating and maintaining contact and friendships. We had expected Emotional Symptoms to be related substantially and primarily to ineffective regulation of negative emotions, and indeed that correlations was -0.78; however, Emotional symptoms were also negatively related to low “A”, indicating that this scale measures emotional problems that interfere with emotional close and supportive relationships. The Hyperactivity Scale, finally, measures a complex construct that involves three of the Big Five factors. As expected from work on Attention-Deficit/Hyperactivity Disorder (ADHD) (Nigg et al., 2002), it was related to both low “C” (inability to focus on tasks and school work and sustain attention) and low “N” (in effective regulation of negative emotion) as well as to high “E”, indicating the high levels energy that accompany hyperactivity in childhood. Overall, these relationships were conceptually meaningful and consistent with expectations.

Item-level analyses of the Multi-Dimensional Socio-emotional Scales

These findings revealed complex Big Five profiles for the three multi-dimensional instruments (i.e., self-efficacy, self-esteem and strengths and difficulties). It is possible that the some of the socio-emotional scales (e.g., hyperactivity) have heterogeneous item content (i.e., contain a mix of items sampled from different Big Five) which creates their apparent multi-dimensional profile on the Big Five. Thus, we performed more in-depth, item-level analyses of the scales on these three instruments. To do so, we tested three ESEM models: (a) an ESEM model regressing the self-efficacy items on the five factors measured by the BFI parcels; (b) another ESEM model regressing the self-evaluation items on the five factors measured by the BFI parcels, and (c) an ESEM model regressing the strengths and difficulties items on the five factors measured by the BFI parcels. The analyses were designed to test the extent to which the dimensions from the Big Five can explain the specific socio-emotional skills assessed by the items on these three inventories.

Results revealed a good approximate fit to the data for the model including items measuring self-efficacy, c² (280) = 319.77, p = 0.051, RMSEA = 0.009, CFI = 0.994, TLI = .983, self-evaluation (CORE), c² (160) = 263.27, p < 0.001, RMSEA = 0.023, CFI = 0.971, TLI = 0.937, and strengths and difficulties (SDQ), c² (290) = 440.02, p < 0.001, RMSEA = 0.017, CFI = 0.974, TLI = 0.930. Table 4 shows detailed tables with standardized regression coefficients obtained from the analyses for SDQ and CORE tests that are the most factorially complex ones.

Table 4
Standardized regression coefficients obtained from the analyses for SDQ and CORE

ESEM Model Regressing Self-Esteem (CORE) Items onto the Big Five Factors
Item	Scale	E	A	C	N	O
6. Sometimes, I do not feel in control of my work.	LC	-0.11	0.01	-0.49*	0.18	0.31*
9. I determine what will happen in my life.	LC	0.05	-0.28	0.18	0.29*	0.27*
10. I do not feel in control of my success in my career.	LC	-0.12	0.07	-0.23	0.10	-0.11
2. Sometimes I feel depressed.	N	-0.24*	0.13	-0.27*	-0.35*	0.39*
4. Sometimes when I fail I feel worthless.	N	-0.13	0.22	0.19	-0.46*	0.06
12. There are times when things look pretty bleak and hopeless to me.	N	-0.32*	-0.05	-0.12	-0.42*	0.26*
3. When I try, I generally succeed.	SEf	0.09	-0.18	0.44*	0.18	0.10
5. I complete tasks successfully.	SEf	-0.04	-0.09	0.68*	0.09	0.16
11. I am capable of coping with most of my problems.	SEf	0.10	0.09	0.09	0.21	0.19
1. I am confident I get the success I deserve in life.	SE	0.15	-0.28	0.28*	0.30	-0.02
7. Overall, I am satisfied with myself.	SE	0.28*	-0.01	0.27	0.14	-0.17
8. I am filled with doubts about my competence.	SE	-0.34*	0.18	-0.23	-0.08	0.25
ESEM Model Regressing SDQ Items onto the Big Five Factors
Item	Factor	E	A	C	N	O
5. I get very angry and often lose my temper.	CP	0.22*	-0.17	0.06	-0.55*	-0.03
7. I usually do as I am told.	CP	0.05	0.33*	0.19*	-0.18	-0.18
12. I fight a lot. I can make other people do what I want.	CP	0.15	-0.25	0.08	-0.36*	0.09
18. I am often accused of lying or cheating.	CP	0.09	0.03	-0.25*	-0.17	-0.01
22. I take things that are not mine from home, school or elsewhere.	CP	0.01	-0.12	-0.06	0.03	-0.05
8. I worry a lot.	ES	0.00	0.24	0.20	-0.39*	0.06
3. I get a lot of headaches, stomach aches or sickness.	ES	-0.11	0.09	-0.06	-0.26*	-0.04
13. I am often unhappy, down-hearted or tearful.	ES	-0.11	0.05	0.09	-0.41*	0.02
16. I am nervous in new situations. I easily lose confidence.	ES	-0.13	0.33*	-0.25*	-0.52*	-0.08
24. I have many fears, I am easily scared.	ES	-0.03	0.44*	-0.11	-0.53	0.14
2. I am restless, I cannot stay still for long.	HA	0.49*	0.03	-0.17	-0.25*	-0.04
10. I am constantly fidgeting or squirming.	HA	0.17*	0.21	-0.08	-0.39*	-0.13
15. I am easily distracted, I find it difficult to concentrate.	HA	0.00	0.26*	-0.47*	-0.37*	-0.04
21. I think before I do things.	HA	0.17*	0.05	0.29*	0.07	0.16
25. I finish the work I’m doing. My attention is good.	HA	0.03	-0.07	0.43*	0.16	0.11
6. I am usually on my own. I generally play alone or keep to myself.	PP	-0.33*	0.00	-0.10	-0.09	0.10
11. I have one good friend or more.	PP	0.15*	-0.04	0.11	-0.09	-0.05
14. Other people my age generally like me.	PP	0.21*	0.27*	0.06	-0.06	-0.05
19. Other children or young people pick on me or bully me.	PP	0.15	-0.10	0.06	-0.23	-0.06
23. I get on better with adults than with people my own age.	PP	-0.15	-0.20	0.12	-0.07	0.29*
1. I try to be nice to other people. I care about their feelings.	PS	-0.03	0.52*	-0.06	-0.14	0.22*
4. I usually share with others (food, games, pens etc.).	PS	0.11	0.37*	0.05	-0.09	-0.11
9. I am helpful if someone is hurt, upset or feeling ill.	PS	0.18*	0.25*	-0.07	-0.21*	0.11
17. I am kind to younger children.	PS	0.18*	0.25	0.17	-0.10	-0.13
20. I often volunteer to help others (parents, teachers, children).	PS	0.07	0.34*	0.20*	-0.29*	0.13

• Note:
• ^*
  p < 0.05. A: Amity; CORE: Core Self-Evaluations; E: Engagement; ESEM: Exploratory Structural Equation Modeling; C: Conscientious self-management; N: Negative emotion regulation; O: Open-mindedness; SDQ: Strengths and Difficulties Questionnaire.

Considering the Self-Efficacy Questionnaire for Children items from the three self-efficacy scales, most academic self-efficacy items were substantially related to Big Five “C”, conscientious self-management. Some of these items had cross-correlations or sole correlations on “O”, which seem to ask about the ability to tell other kids about one’s dislikes. Social self-efficacy items were associated either with “E” (addressing social skills useful for initiating contact or interacting with strangers) or with “A” (addressing social skills that result in harmonious and collaborative, rather than conflictual, interactions). Emotional self-efficacy was mostly related to “N” (effective negative emotion regulation) but have had some associations with “A”. In general, these items had unique and coherent links with the Big Five.

Considering the CORE self-evaluation test. Locus of control items were associated to effective “N” (true keyed item), “O”, and “C”. CORE-Neuroticism items mostly referred to feelings of sadness, worthlessness, and depression and thus were sensibly related to “N” but also low “E”. Some items had unexpectedly high links with “O”. Generalized self-efficacy items were most like the Academic self-efficacy items on the Self-efficacy scale for kids and thus related mainly to “C”. Items on self-esteem were related to “C” and “E”.

Considering SDQ items conduct problems items, as we expected, captured a range of different Big Five content, showing complex but coherent relations: “losing temper” with ineffective emotion regulation (low N) and high “E”, “obediently doing as I’m told” with high “A”, “fighting” with low “A” and low “N”, and “lying” low “C”. Most emotional symptoms items were, as expected, low on “N” (problems with negative-emotion regulation) but 3 items had unexpectedly strong links with “A”. Peer problems related mostly to low “E” and prosocial skills to high “A”.

Discussion and Conclusion

The present research tested whether a broad set of scales from six instruments commonly used to assess SE skills in educational settings could be mapped on the space of the socio-emotional Big Five.

We found that the Grit scale is mainly an expression of the theme of conscientious self-management. This is consistent with recent findings that empirically grit is the same trait as Conscientiousness, both genetically and phenotypically (Rimfeld, Kovas, Dale, & Plomin, 2016).

Self-efficacy items ask subjects how well they can perform a diverse set of tasks or behaviors. Our analyses suggest that each task in these measures refers to a specific Big Five content domain. The “how well” part of the items refers to a general self-perception of being able to do something. Interestingly the content part, that is, the specific “something” where subjects report their confidence matched coherently to four of the five core themes of the Big Five: self-management and task oriented behavior (as captured by “C”) linked to academic self-efficacy items (ability to pay attention, to study, avoiding distractions, do homework). Social self-efficacy was about interpersonal relationships and so Engagement linked to social task like making friends, telling funny things to friends, and express opinions, whereas A was linked to social tasks that involved harmony and getting along (e.g., preventing quarrels). Negative emotion regulation was related to emotional self-efficacy, where the task was to deal with internal, emotional challenges, like preventing becoming nervous, controlling feelings, or suppressing unpleasant thoughts.

It is interesting to know that the generalized self-efficacy items of the CORE self–evaluation instrument mapped primarily onto conscientious self-management and negative emotion regulation. The content of the items refers to these two core themes of the Big Five. Overall, the Big Five model accounted for a high proportion of the variance of Grit and self-efficacy scales (57% to 83%).

The constructs underlying CORE self-esteem, Locus of Control, and SDQ were more complex representing mixed combination of the core themes. CORE self-esteem is comprised of fundamental traits involved in the evaluation of the self: self-esteem, generalized self-efficacy, emotional stability and locus of control. We found that generalized self-efficacy was related to “C” (self-management) and “N” (negative emotion regulation). Locus of control was also related to self-management. And self-esteem to engagement and self-management. One conclusion is that the operationalization of self-esteem on CORE is a mix of resilience as afforded by the effective regulation of negative emotion, self-management, and engagement (i.e., energy and positive emotions).

The Nowicki and Strickland (1973) scale to measure their version of locus of control was associated with low emotional regulation and low engagement. These results are different from the associations found on the subscale of the same-named construct in the CORE test. Locus of control refers to “belief that desired effects result from one’s own behavior rather than by fate or powerful others” (Ferris, Johnson, Rosen, & Tan, 2012, p.83). Beliefs reflect knowledge inferred from experiences with particular types of events and thus provide an understanding about the way the world works, such as beliefs about control, about cognitive growth (i.e., growth mindset), and about the chances of succeeding on an undertaking or task (i.e., self-efficacy). Our initial hypothesis was that these control beliefs would be substantially influenced by the individual’s traits in the negative emotion regulation (N) domain (De Raad & Schouwenburg, 1996). However, this influence may be weaker than expected and these SE skills, particularly the beliefs of control and growth, are not that well represented within the socio-emotional Big Five.

Finally, the SDQ is an instrument focused on adaptive versus non-adaptive constructs and has more scales related to the absence of relevant SE skills than their presence. Its scales were predicted by specific and coherent core themes: emotional symptoms were linked to low emotion regulation; hyperactivity symptoms to a combination of low self-management, low emotion regulation, and high engagement (energy); prosocial skills to amity/collaboration and engagement; and finally peer problems to low engagement and conduct problems to low emotion regulation.

Overall, the approach taken here illustrates the power afforded by a consensual and replicable taxonomy of socio-emotional characteristics, such as the Big Five. New constructs and measures can be described, compared, and understood as combinations of elements that have long been studied and are well-understood, thus helping us progress as a unified discipline. This taxonomy can even be helpful when measures are applied in different cultural contexts, like public school children in Brazil. The present findings give us some hope that knowledge about socio-emotional characteristics gained in the West can be usefully applied to help solve the serious problems with education in Brazil.

Some limitations of the present study should be acknowledged. First, the empirical mapping of big-five is restricted to the scales we selected for this study. There are other instruments created to operationalize socio emotional skills such as CASEL (Damásio, 2017), OECD measure of educational constructs used in Program for International Student Assessment 2000 (Marsh et al., 2006) or MESH skills used in CORE districts (www.transformingeducation.org) that were not included in this study. Second limitation is that most educational instruments are unbalanced mostly composed of positively worded items. Because of that they are more susceptible to be influenced by acquiescence specially in young samples (Primi, De Fruyt, Santos, Antonoplis, & John, 2019). Finally, our big five markers were operationalized at the level of domains. It would be much rich if we had an instrument that have facet scales for each domain such as BFI-II (Soto & John, 2017) or Senna v2.0 (Primi, Santos, De Fruyt, & John, 2019). So as a goal for future studies we advise they should map these new instruments on conceptual socio and emotional big-five, preferably operationalized at the level of facets and also including some control for acquiescence for all scales.

Como citar esse artigo/ How to cite this article

• Primi, R., Santos, D. D., Hauck, N., De Fruyt, F., & John, O. P. (2019). Mapping self-report questionnaires for socio-emotional characteristics: What do they measure? Estudos de Psicologia (Campinas), 36, e180138. http://dx.doi.org/10.1590/1982-0275201936e180138
• Support: This study was financed in part by the Instituto Ayrton Senna, Conselho Nacional de Desenvolvimento Científico e Tecnológico (Process n° 310909/2017-1), and Fundação de Amparo à Pesquisa do Estado de São Paulo (Process n° 2018/10933-8).

• 6
  These six instruments were specifically adapted to Brazilian Portuguese for the present study. If the reader has an interest in any ofthem, please contact the first author so he can make the Portuguese version available.

References

• Battelle for Kids. (2015). Partnership for 21^st Century Skills Hilliard, Ohio. Retrieved October 10, 2019, from www.p21.org
  » www.p21.org
• Benet-Martínez, V., & John, O. P. (1998). Los Cinco Grandes across cultures and ethnic groups: Multitrait-multimethod analyses of the Big Five in Spanish and English. Journal of Personality and Social Psychology, 75(3), 729-750. http://dx.doi.org/10.1037/0022-3514.75.3.729
  » https://doi.org/10.1037/0022-3514.75.3.729
• Borsa, J. C., Damásio, B. F., & Bandeira, D. R. (2012). Adaptação e validação de instrumentos psicológicos entre culturas: algumas considerações. Paidéia (Ribeirão Preto), 22(53), 423-432. http://dx.doi.org/10.1590/S0103-863X2012000300014
  » https://doi.org/10.1590/S0103-863X2012000300014
• California Office to Reform Education (2014). Social-Emotional Skills Retrieved October 20, 2019, from http://www.ousd.org/cms/lib07/CA01001176/Centricity/Domain/3154/Social%20and%20Emotional%20Learning.pdf
  » http://www.ousd.org/cms/lib07/CA01001176/Centricity/Domain/3154/Social%20and%20Emotional%20Learning.pdf
• Carroll, J. (1963). A model of school learning. The Teachers College Record, 64(8), 723-723.
• Collaborative for Academic, Social, and Emotional Learning (CASEL). (2012). 2013 Casel guide: Effective social and emotional learning programs. Preschool and Elementary School Edition Chicago: KSA-Plus Communications.
• Damásio, B. F. (2017). Measuring socioemotional skills of children and adolescents: Development and validation of battery. Temas em Psicologia, 25(4), 2043-2050. http://dx.doi.org/10.9788/TP2017.4-24Pt
  » https://doi.org/10.9788/TP2017.4-24Pt
• De Fruyt, F., Wille, B., & John, O. P. (2015). Employability in the 21st century: Complex (interactive) problem solving and other essential skills. Industrial and Organizational Psychology, 8(2), 276-281.
• De Raad, B., & Schouwenburg, H. C. (1996). Personality in learning and education: A review. European Journal of Personality, 10, 303-336.
• Duckworth, A. L., & Quinn, P. D. (2009). Development and validation of the short grit scale (grit-s). Journal of Personality Assessment, 91(2), 166-174. http://dx.doi.org/10.1080/00223890802634290
  » https://doi.org/10.1080/00223890802634290
• Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., & Beechum, N. O. (2012). Teaching adolescents to become learners - the role of noncognitive factors in shaping school performance: A critical literature review Chicago: University of Chicago Consortium on School Research.
• Ferris, D. L., Johnson, R. E., Rosen, C. C., & Tan, J. A. (2012). Core self-evaluations a review and evaluation of the literature. Journal of Management, 38(1), 81-128.
• Geary, D. C. (2007). An evolutionary perspective on learning disability in mathematics. Developmental Neuropsychology, 32(1), 471-519.
• Goodman, R. (1997). The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581-586.
• John, O. P., & De Fruyt, F. (2016). Framework for the longitudinal study of socio and emotional skills in cities Paris: OECD. Retrieved October 20, 2019, from http://www.oecd.org/callsfortenders/CfT%20100001311%20Longitudinal%20Study%20of%20Social%20and%20Emotional%20Skills%20in%20Cities.pdf
  » http://www.oecd.org/callsfortenders/CfT%20100001311%20Longitudinal%20Study%20of%20Social%20and%20Emotional%20Skills%20in%20Cities.pdf
• John, O. P., Donahue, E. M., & Kentle, R. L. (1991). The Big Five Inventory-Versions 4a and 54 Berkeley: Institute of Personality and Social Research.
• John, O. P., Naumann, L., & Soto, C. J. (2008). Paradigm shift to the integrative Big Five trait taxonomy: Discovery, measurement, and conceptual issues. In O. P. John, R. W. Robins & L. A. Pervin (Eds.), Handbook of personality: Theory and research (3rd ed., pp.114-158). New York: Guilford.
• Judge, T. A., Erez, A., Bono, J. E., & Thoresen, C. J. (2003). The core self-evaluations scale: Development of a measure. Personnel Psychology, 56(2), 303-331. http://dx.doi.org/10.1111/j.1744-6570.2003.tb00152.x
  » https://doi.org/10.1111/j.1744-6570.2003.tb00152.x
• Kyllonen, P. C., Lipnevich, A. A., Burrus, J., & Roberts, R. D. (2014). Personality, motivation, and college readiness: A prospectus for assessment and development. ETS Research Report Series, (1), 1-48.
• Marsh, H. W., Lüdtke, O., Muthén, B., Asparouhov, T., Morin, A. J., Trautwein, U., & Nagengast, B. (2010). A new look at the Big Five factor structure through exploratory structural equation modeling. Psychological Assessment, 22(3), 471.
• Marsh, H. W., Hau, K. -T., Artelt, C., Baumert, J., & Peschar, J. L. (2006). OECD’s brief self-report measure of educational psychology’s most useful affective constructs: Cross-cultural, psychometric comparisons across 25 countries. International Journal of Testing, 6(4), 311-360. http://dx.doi.org/10.1207/s15327574ijt0604_1
  » https://doi.org/10.1207/s15327574ijt0604_1
• Muris, P. (2001). A brief questionnaire for measuring self-efficacy in youths. Journal of Psychopathology and Behavioral Assessment, 23(3), 145-149.
• Muthén, L. K., & Muthén, B. O. (2012). Mplus User’s Guide (7th ed.). Los Angeles: Muthén & Muthén.
• National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century Washington: The National Academies Press.
• Nigg, J. T., John, O. P., Blaskey, L., Huang-Pollock, C., Willcutt, E. G., Hinshaw, S. P., & Pennington, B. (2002). Big Five dimensions and ADHD symptoms: Links between personality traits and clinical symptoms. Journal of Personality and Social Psychology, 83(2), 451-469.
• Nowicki, S., & Strickland, B. R. (1973). A locus of control scale for children. Journal of Consulting and Clinical Psychology. 40(1), 148-154. http://dx.doi.org/10.1037/h0033978
  » https://doi.org/10.1037/h0033978
• Organization for Economic Cooperation and Development (OECD). (2016) Longitudinal study of social and emotional skills in cities. Paris: OECD. Retrieved October 20, 2019 from http://www.oecd.org/callsfortenders/CfT%20100001311%20Longitudinal%20Study%20of%20Social%20and%20Emotional%20Skills%20in%20Cities.pdf
  » http://www.oecd.org/callsfortenders/CfT%20100001311%20Longitudinal%20Study%20of%20Social%20and%20Emotional%20Skills%20in%20Cities.pdf
• Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135(2), 322-338.
• Primi, R., De Fruyt, F., Santos, D., Antonoplis, S., & John, O. P. (2019). True or false? Keying direction and acquiescence influence the validity of socio-emotional skills items in predicting high school achievement. International Journal of Testing, 0(0), 1-25. http://dx.doi.org/10.1080/15305058.2019.1673398
  » https://doi.org/10.1080/15305058.2019.1673398
• Primi, R., Santos, D., De Fruyt, F., & John, O. P. (2019). SENNA V2.0 Technical Manual. São Paulo: Instituto Ayrton Senna.
• Primi, R., Santos, D., John, O. P., & De Fruyt, F. (2016). Development of an inventory assessing social and emotional skills in brazilian youth. European Journal of Psychological Assessment, 32(1), 5-16. http://dx.doi.org/10.1027/1015-5759/a000343
  » https://doi.org/10.1027/1015-5759/a000343
• Rimfeld, K., Kovas, Y., Dale, P. S., & Plomin, R. (2016). True grit and genetics: Predicting academic achievement from personality. Journal of Personality and Social Psychology, 111(5), 780-789. http://dx.doi.org/10.1037/pspp0000089
  » https://doi.org/10.1037/pspp0000089
• Roberts, R., Martin, J. E., & Olaru, G. (2015). A rosetta stone for noncognitive skills: Understanding, assessing, and enhancing noncognitive skills in primary and secondary education New York: Asia Society.
• Sailer, O. (2005). Crossdes: A package for design and randomization in crossover studies. Rnews, 5(2), 24-27.
• Santos, D., & Primi, R. (2014). Social and emotional development and school learning: A measurement proposal in support of public policy (Technical Report). São Paulo: Ayrton Senna Institute. Retrived October 20, 2019, from http://educacaosec21.org.br/wp-content/uploads/2013/07/Social-and-emotional-developmente-and-school-learning.pdf
  » http://educacaosec21.org.br/wp-content/uploads/2013/07/Social-and-emotional-developmente-and-school-learning.pdf
• Schmitt, D. P., Allik, J., McCrae, R. R., & Benet-Martinez, V. (2007). The geographic distribution of Big Five personality traits: Patterns and profiles of human self-description across 56 nations. Journal of Cross-Cultural Psychology, 38(2), 173-212. http://dx.doi.org/10.1177/0022022106297299
  » https://doi.org/10.1177/0022022106297299
• Soto, C. J., & John, O. P. (2017). The next Big Five inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of Personality and Social Psychology, 113(1), 117-143. http://dx.doi.org/10.1037/pspp0000096
  » https://doi.org/10.1037/pspp0000096
• Soto, C. J., John, O. P., Gosling, S. D., & Potter, J. (2008). The developmental psychometrics of big five self-reports: Acquiescence, factor structure, coherence, and differentiation from ages 10 to 20. Journal of Personality and Social Psychology, 94(4), 718–737. http://dx.doi.org/10.1037/0022-3514.94.4.718
  » https://doi.org/10.1037/0022-3514.94.4.718
• Strive Task Force on Measuring Social and Emotional Learning (2013). Beyond content: incorporating social and emotional learning into the strive framework New York: Philiber Research Associates.
• Trilling, B., & Fadel, C. (2009). 21st century skills: Learning for life in our times San Francisco: Jossey-Bass.

Publication Dates

• Publication in this collection
  11 Nov 2019
• Date of issue
  2019

History

• Received
  28 Feb 2019
• Reviewed
  25 July 2019
• Accepted
  23 Aug 2019