Validity evidences for the Emotion Regulation Test

Abstract This study sought evidence of validity based on the internal structure (factorial validity) and its relationship with external variables (psychopathological indicators and personal data) for the Emotional Regulation Test (ERT). This instrument is composed by conflict-related scenarios (vignettes) involving eight basic emotions, and questions about the effectiveness of different ways to deal with the emotional conflict of each scenario. The sample consisted of 289 adults, who answered the ERT via internet. A smaller portion of the sample (N = 191) also responded to the Clinical Dimensional Inventory of Personality - Screening. The study resulted in a two-factor structure related to effective (factor 1) and non-effective (factor 2) strategies to regulate emotions, with good reliability indexes. No significant correlations with psychopathological indicators were found. We recommended the use of the instrument for research purposes.


Introduction
Emotional Regulation (ER) can be understood as the maintenance or modulation of the various components of emotional experience in order to influence the trajectory of emotion (Gross & Thompson, 2007;Gross, 2015). For the study of ER it is important to know which emotions exist and how they work and can be regulated. Although they have long been studied by psychological science, defining emotions is not simple. For this reason there are numerous theories and models that define and quantify them in different ways.
In Plutchik's (2003) psychoevolutionary theory, emotions are forms of communication that have adaptive or survival value, establishing relationships with various areas of mental functioning. The author identifies eight basic emotions (joy, fear, sadness, acceptance, anger, surprise, disgust and expectation or curiosity) for which he has established a pattern of sequences, in which an emotion is always associated to cognition, an observable behavior that tends to an effect on the environment.
On the other hand, ER is a skill that can be enhanced, increasing as one matures and becomes educated (Zimmermann & Iwanski, 2014;Gross & Urry, 2010). In addition, it was also pointed out that women demonstrated better performances in regulating emotions (Nolen-Hoeksema & Aldao, 2011). It is noticed that ER is a construct of great impact in various areas of people's lives, having been pointed as one of the fastest growing theoretical fields in Psychology (Gross, 2015).
Peña-Sarrionandia, Mikolajczak and Gross (2015) state that there are usually two relatively independent research traditions that address the regulation of emotions. The first focuses on the processes by which individuals influence the emotions they have, when they do, and how they experience and express these emotions. The second is the tradition of emotional intelligence (EI), which focuses, among other things, on individual differences in ER.
As an example of the first research tradition in ER is the theory proposed by Gross (1998;2015), based on the Modal Model of Emotion. According to the author, an emotion occurs in a context in which a situation compels an individual's attention (personsituation context), which will assess it cognitively and generate a response (explicit or implicit), under a given circumstance. In this model, each process involved in eliciting an emotion (Situation, Attention, Appraisal, and Response) offers an opportunity to regulate it: the choice of situation, the modification of the situation, the redirection of attention, the cognitive restructuring, and the modulation of the emotional response (Gross & Thompson, 2007).
For Gross and Jazaieri (2014) it is the objective of emotional regulation that will determine whether emotional experience, expression or physiology should be increased, maintained, or decreased in duration and / or intensity. Thus, once the goal of emotion regulation specifies ends, emotional regulation strategies specify the means that may be more or less efficient in achieving the goal (Peña-Sarrionandia et al., 2015).
The characteristic studies of the second tradition of research in ER come from the understanding of EI. For Mayer, Caruso and Salovey (2016), EI presents a skill model branched into four levels, namely: (1) perceiving emotions; (2) facilitating thought; (3) understanding emotions and (4) managing emotions.
As the fourth and most complex skill, managing emotions requires reflective control of emotions to promote emotional and intellectual growth and involves skills such as staying open to feelings, even if they are unpleasant; being able to distance oneself to reflect on one's own emotions; monitor them to recognize whether they are positively or negatively influencing behavior; and also managing emotion in oneself and others by moderating or valuing them, without repressing or exaggerating the information they convey (Mayer and Salovey, 1997). This approach emphasizes the results of ER rather than its basic processes, noting that the various cases of ER are not completely independent of each other, but individuals show some consistency in their regulatory habits. (Peña-Sarrionandia et al., 2015;Ramzan & Amjad, 2017).
Whether by one method of study or another, somehow the regulation of emotions needs to be evaluated. In this sense, an important step for conducting research on the subject is the availability of measuring instruments with good psychometric properties to measure it. Ireland, Clough and Day (2017), for example, point out that it is necessary to study, build and validate assessment tools of ER strategies, for different populations.
Although these instruments contribute to the evaluation of ER, they also have limitations. The ERQ and the DERS, for example, are self-report instruments that often have higher correlations with personality traits than with intelligence. Although there is evidence of incremental validity in relation to the five major factors for predicting a large number of criterion variables, measures with this type of instrument have been more related to the field of personality and self-efficacy rather than intelligence (Petrides, 2017).
The TRE_Stroop, the ERP-R and the MSCEIT, in turn, are performance tools that require the respondent to use the mental function to evaluate and / or consider a correct alternative between response options. Due to this characteristic, they often have positive and significant correlations with other intelligence and performance measures and low and non-significant correlations with personality measures (Petrides, 2017). Mayer et al. (2016) advocate the use of this type of instrument to assess EI-related skills, as they are similar to those used to measure other traditional cognitive skills.
The TRE_Stroop, however, is an instrument based on the Emotional Stroop technique, which measures time, in milliseconds, assuming that people who take longer to perform a cognitive task (such as naming colors, for example) in the face of an emotional distractor (a word or image), are those that least regulate emotions. In the case of TRE_Stroop, there was no evidence to support this assumption (Bueno, 2013).
The ERP-R and the MSCEIT are instruments that assess the regulation of emotions by situational performance, but the ERP-R is not punctuated by hit-or-miss, but by the difference between choices of adaptive and non-adaptive regulatory strategies, where it is assumed that all strategies available in the test have the same weight, degree of difficulty, or equivalent impact on people's lives, which may lead to distortions in the interpretation of scores, compromising their validity. MSCEIT is an instrument developed for the North American cultural context, which, although translated into other languages and cultures, including Brazilian Portuguese, provides little information about its psychometric properties in this context and its use is very costly for the local standards.
Therefore, this study aimed to evaluate the psychometric properties of the Emotions Regulation Test (Lira, 2017), which was built based on the psychoevolutionary theory of emotions (Plutchik, 2003) and on the ER model proposed by Gross (2015). More specifically, we intended to seek evidence of validity based on the internal structure of the instrument (factorial validity) and on the relationship with external variables: psychopathological indicators and personal data. We expected to find a one-factor solution compatible with the general ability of regulating emotions, as previously obtained with the MSCEIT (Mayer et al., 2002), by structural similarity to this instrument; negative and significant correlations with a self-report measure to screen for personality disorders, as obtained in previous studies (Aldao et al., 2010;Sheppes et al., 2015;Weissman et al., 2019); and significant gender effects (in favor of women), as noted by Nolen-Hoeksema and Aldao (2011) and education effects (in favor of higher levels of education) on emotion regulation scores, as observed by Zimmermann and Iwanski (2014) and Gross and Urry (2010).

Instruments
To characterize the sample of participants in this research, we used a Sociodemographic Questionnaire, which collected information on age, sex and education. To achieve the objectives proposed in this research, two instruments were applied, whose characteristics are described below.

Emotions Regulation Test: Instrument construction process
The first step in the creation of the TRE was to compose situations operationalized in vignettes, in which a character experiences one of the eight basic emotions proposed by Plutchik (2003): joy, fear, sadness, acceptance, anger, surprise, disgust, and expectation or curiosity. The vignette for surprise, for example, was presented this way: "Ricardo is walking downtown when he comes across a naked person behaving strangely" (Lira, 2017). Then, for each of the eight vignettes, three items were presented, representing different action options to regulate emotions. These options are based on the ER strategies proposed by Gross (2008) (situation selection, situation modification, attention redirection, cognitive change, and response modulation), in line with the natural sequences of each emotion predicted by Plutchik (2003). For example, for surprise, Plutchik (2003) suggests that the elicitation sequence for this emotion would be: (1) the occurrence of an unexpected event -in the case of this instrument: being in an unusual place with a naked person acting strangely; (2) ask yourself about the nature of the event; (3) present the emotional state (be surprised); (4) a likely observable behavior would be the interruption of what is being done; (5) aiming to gain time for orientation. Both the stories (vignettes) and the items were created by the authors of this article.
This way it is possible to ask the participant to judge the effectiveness of each behavior presented to deal with the situation. The instrument can be characterized as a situational judgment test (Ambiel, Campos, Alves & Silva, 2015), constituting a test to evaluate the participants' judgment regarding the adequacy of solutions to situations that represent realistic emotional challenges and cases (scenarios). Thus, the respondent should judge the effectiveness of each behavior presented for the situation described in a vignette, assuming that the better the judgment presented, the better the ability in what is being evaluated by the test.
The items presented to the subjects are operationalizations of the ER strategies proposed by Gross (2015) or none of them (non-regulatory situation), respecting the type of behavioral response described by Plutchik (2003) for each emotion. Therefore, in the surprise vignette, the items presented to be judged by the participants were: (1) Scold the person for his or her inappropriate behavior (non-regulation); (2) Ask yourself if that person is well and seek help among passersby (cognitive restructuring); (3) Redirect your path as not to approach the person (change of situation). Next, participants were instructed to use a five-point Likert scale to evaluate the effectiveness of each strategy, assigning 1 for very ineffective strategies or 5 for very effective strategies. Scores 2, 3 or 4 could be employed to represent intermediate effectiveness at extreme values.
The instrument was reviewed and discussed for theoretical and practical adequacy in a psychological assessment study group consisting of graduate and undergraduate students engaged in research, and psychological assessment of skills related to emotional intelligence such as emotion regulation. From these discussions, the authors incorporated the modifications -adequacy and clarity of the vignettes in relation to their emotions, improvements in the offer of ER strategies and adjustments in the scoring scale -and finalized the version used in this study, computerizing the instrument to perform the data collection.

Clinical Dimensional Personality-Screening Inventory (IDCP-T)
The IDCP was developed in Brazil by Carvalho and Primi (2011), based on Millon's theory and the diagnostic criteria of the categories presented in axis II of the DSM-IV-TR. For this study we used its version for screening (IDCP-T), consisting of 15 items, which refer to the following clinical dimensions of personality: eccentricity, mood instability, aggressiveness, self-sacrifice, and grandiosity (Carvalho, Pianowski, & Reis, 2017).
The instrument features statements such as "I usually do things I don't like to help others", which are rated by respondents using a four-point Likert scale, where 1 corresponds to "It has nothing to do with me" and 4 to "It's all about me". The score of each participant on the scale was calculated by summing the values assigned to the items, and the items that were towards health and not mental disorders were inverted before being added. From the application of the instrument we obtain information that discriminates the probability of a positive diagnosis for people with personality disorders and people without this diagnosis. Studies showed that the screening version of the IDCP presented satisfactory results, which suggest diagnostic accuracy in the identification of patients with personality disorders, primarily related to imbalances in interpersonal relationships, psychological distress experience, and self-neglect Carvalho, 2017).

Procedures
This study was approved by a Research Ethics Committee under approval number CAAE 51159715.9.0000.5208. Participants were contacted via the Internet, being informed about the research objectives and procedures and those who agreed to participate accessed a link provided in the invitation message that led to the research page. There, they accessed the Informed Consent Form (ICF) and the instruments that were part of the research. Data were automatically stored in a spreadsheet and subsequently statistically analyzed.

Data Analysis
To verify the internal structure of the TRE, the exploratory factor analysis technique was employed using the method of robust weighted least square mean and variance adjusted (WLSMV), more suitable for dichotomous data, with geomin rotation, implemented with the aid of the software MPlus 6.11. The hypothesis of obtaining a one-factor structure related to the ability to regulate emotions was formulated.
To investigate the relationship between ERT and indicators of personality disorders, Pearson's correlation analysis was employed, and we expected to find a negative and significant value between the measurements. In addition, a multivariate analysis of variance was used, using the partial eta-square index to verify the effect of sex and education on the regulation of emotions, controlling the effect of age. In this case, significant effects were expected from both variables, with women with significantly higher scores than men, and participants with higher educational levels with higher scores than those with lower levels. Correlational and effect analyzes of other variables were performed with the aid of JASP software.

Results
Solutions with 1, 2, 3 and 4 factors were extracted, whose adjustment indices are shown in Table 1. The model fit was evaluated using the following thresholds: TLI and CFI above 0.95; RMSEA less than 0.06 and SRMR less than 0.08 (Hooper, Coughlan, & Mullen, 2008;Kline, 2011).
It is noted that there is a great improvement in the fit indices from the first to the second factorial structure, but little change in the subsequent structures. Therefore, we chose to use the two-factor solution, whose factor loads are shown in Table 2.
Despite the expectation of obtaining a single factor solution, factors were obtained depending on whether the strategy used to regulate emotions was effective (Factor 1) or ineffective (Factor 2). The items that make up each factor are highlighted in Table 2. Items whose factor loadings were below 0.3 (items 5, 6, 13 and 20) were eliminated. A new factor analysis resulted in a structure with the need to eliminate other items, which would compromise the representation of emotions in the instrument. Therefore, we chose to continue the analysis, based on the factorial structure found in the first analysis, only with the elimination of items 5, 6, 13 and 20.
The scores on the two factors found were calculated based on the Item Response Theory Rasch Model (Lira, 2017). The mean theta (skill of participants) in Factor 1 was 0.25, with a standard deviation of 1.79, while the mean theta in Factor 2 was 0.76 and the standard deviation, 1.48. The internal consistencies, measured by the Kuder-Richardson coefficient, of factors 1 and 2 were, respectively, 0.75 and 0.62. In addition, there was a positive and statistically significant correlation between the two emotion regulation factors (r = 0.369; p <0.01), but not with the psychopathological indicator of the IDCP. Table 3 presents the results of the analysis of the effects of gender and schooling on the emotion regulation factors, controlling for age.
It was observed that there was a significant effect of education and sex, but there was no interaction effect between these two categorical variables. Figure 1 shows the characteristics of observed effects.
We observed that in both factors women had higher averages than men and that the means in Factor 2 were higher than in Factor 1. There was also a tendency for scores to increase in emotion regulation as one walks from high school to higher education. However, participants with postgraduate level had lower averages (M = 0.2163 in factor 1 and M = 0.8584 in factor 2) than those with graduate level (M = 0.3983 in factor 1; M = 0.8224 in factor 2). In addition, factors 1 and 2 showed non-significant (r = 0.097) and significant (r = 0.152, p <0.01) correlations with age, respectively. Note. root mean square error of approximation (RMSEA); comparative fit index (CFI); tucker-lewis index (TLI); standardized root mean square residual (SRMR)

Discussion
The present study reports the process of evaluation of the psychometric properties of the Emotions Regulation Test. More specifically, evidence of validity based on internal structure and relationship with other variables was sought. In the investigation of internal structure, it was expected to obtain a one-factor structure, related to the general ability to regulate emotions. However, the result of the factor analysis indicated a two-factor structure, suggesting that two mental processes are related to emotion regulation: detecting that a strategy is effective (Factor 1) and detecting that a strategy is not effective (Factor 2) to regulate emotions.
These results are in line with the proposition that the emergence of emotions and strategies for regulating them are contextual (individual-situation) (Gross, 1998(Gross, , 2015 and suggest that individual differences in the ability to regulate emotions derive from the ability to identify its effectiveness (Factor 1) or ineffectiveness (Factor 2) in the context of the situation presented. In the case of the test, the situation (context) was presented in the vignettes and the strategies in the items. Thus, information processing seems to involve the analysis of the consequences of maintaining, increasing or decreasing the intensity and expression of the emotional state in the presented context, followed by an evaluation (response) about the effectiveness or inefficiency of the described strategy. This seems to constitute what can be inferred from the test scores based on the internal structure obtained. Moreover, the moderate correlation between the two factors suggests that being able to detect that a strategy is not effective in dealing with an emotion does not necessarily imply that a person has the ability to detect which strategy is most effective to use. In fact, Naragon-Gainey, McMahon, and Chacko (2017) claim that it is plausible that the common uses of specific mismatched strategies are interrelated, as well as the common uses of adaptive strategies.
The authors further suggest that the ER strategy model could be consistent with two structures: a single factor model (as expected in this paper) in which supposedly effective strategies would have positive charge and ineffective negative charge; or two factors, as actually found here, one for effective strategies and one for ineffective strategies. The first would imply that people who often use adaptive strategies also rarely use inappropriate strategies, while the second would indicate a greater degree of independence between the use of effective and ineffective strategies (Naragon-Gainey, McMahon, & Chacko, 2017). The structure found in this paper is compatible with the conception of greater independence between the use of effective and ineffective strategies.
Additionally, the moderate correlation between the two factors (r = 0.369; p <0.01) is consistent with that found in previous studies both in Brazil (r = 0.491, p <0.01) (Jesus Junior & Noronha, 2007) and in other Latin-American countries (e.g. Curci, Lanciano, Soleti, Zammuner & Salovey, 2013;Monteiro, 2009). Based on the results found in these studies, it is plausible to expect these factors to come together in a second-order  Figure 1. Effects of sex and education on emotion regulation factors factor related to the regulation of emotions, but this remains to be further investigated in a study that should involve other measures of emotional intelligence. A factorial study previously conducted with MSCEIT showed that emotion regulation items had factor loadings in two factors, although they predominated in only one of them (Brannick, Wahi & Goldin, 2011). Multifactorial results were also found with self-reported emotion regulation assessment tools. The ERP study found two factors, related to the regulation of positive and negative emotions (Gondim et al., 2015), and the study with the Emotional Skills Inventory found one factor related to regulation in other people and two related to the regulation of high and low power emotions in oneself (Bueno et al., 2015).
Thus, the results obtained in this study, as in previous studies, seem to suggest that emotion regulation encompasses a distinct but correlated skill set, although each instrument identifies a nuance of that skill set. It also highlights the importance of having a variety of instruments to measure the same construct, as they end up capturing different aspects, which contribute not only to the evaluation of this construct, but to their own theoretical understanding.
Therefore, it can be said that the ERT is valid for the evaluation of knowledge about the use of RE strategies in particular situations. More specifically, the ability to detect whether an emotion regulation strategy is effective (Factor 1) or not (Factor 2). These results may be relevant for a better understanding of the skills that need to be emphasized, for example, by reducing the use of maladaptive (ineffective) ER strategies and / or increasing the use of adaptive (effective) strategies consistent with the situation in which they are used. (Conklin et. al., 2015).
In this study we also sought relationships between ERT and psychopathological indicators of personality disorders through the IDCP-T. Unlike expected, no correlations were found between the variables, as seen in other studies. DERS, for example, showed strong correlations with IDCP factors (Miguel et al., 2017) and with the Brief Symptom Inventory and Structured Clinical Interview for DSM-IV Disorders (SCID-I / II), measures of self-report assessing general psychiatric symptomatology (Coutinho et al., 2010;Fowler et al., 2014). Indeed, previous research using IS self-reported measures has already suggested that personality disorders and other psychological disorders would be associated with a lower ability to regulate emotions (Aldao et al., 2010;Gratz & Roemer, 2004).
However, there are also studies in which this relationship was not found. Researchers found that patients with Borderline personality disorder did not have impaired IE skills (including in the MSCEIT's "emotional regulation" subscale), although they reported severe ER deficiencies using the DERS scale (Beblo et al., 2010). It has been questioned why these patients report serious problems with emotional regulation and yet show intact performance in the ability of EI. The authors realized that one of the variables to which this may be associated is the fact that self-report instruments (such as the DERS scale) focus on self-perceived behavior in everyday life, while tests such as those on the MSCEIT and those in the present study are based on the respondents' knowledge about how to regulate emotions, in items that have a response considered more adequate than the others. Performance tests, for example, tend to capture individual differences in a more basic skill, while selfreport tests tend to capture the concomitant effects that a given mental process triggers on behaviors. These two types of measurements often have low correlations with each other, as previous studies in the field of emotional information processing indicate (Zeidner, Roberts & Matthews, 2008).
Following the same logic, other emotional skills tests also showed low and non-significant correlations with self-report measures. A good example is the study by Miguel, Finoto and Miras (2013), in which the IDCP also did not correlate with the Computerized Primary Emotions Perception Test. The authors point to the independence between the emotional intelligence constructs (this time being assessed through the ability to perceive emotions) and personality traits. In addition, the IDCP-T is a reduced instrument for detection of psychopathological symptoms and it would be best to investigate this relationship directly in patients with (experimental group) and without (control) psychopathological diagnoses.
The analysis of ERT in relation to personal characteristics showed a better performance of women compared to men, as in previous studies on IS (e.g. Cabello, Sorrel, Fernández-Pinto, Extremera, & Fernández-Berrocal, 2016), consisting on a result that can be used as evidence of validity for this instrument. The study by Nolen-Hoeksema and Aldao (2011) also captured these gender differences in favor of women, indicating that they report using a wider range of RE strategies than men, such as for example, rumination, reassessment, problem-solving, acceptance and social support.
A schooling effect was also found, even when age was controlled, showing the importance of education and environmental experiences for the development of this ability. Other studies have also found a general tendency to increase adaptive regulation of emotions according to age and schooling (Scheibe, Sheppes, & Staudinger, 2015;Zimmermann & Iwanski, 2014). According to Gross and Urry (2010) with advancing age, adults would be able to select and optimize certain processes of emotional regulation to compensate for changes in internal and external resources, such as functional losses resulting from aging.
The results found in this paper constitute an initial study on the psychometric properties of the Emotions Regulation Test. Although its results were encouraging, further studies are needed to form a broader mass of results that allows more consistent conclusions to be derived. In addition, the study has important limitations, such as restricting the sample to participants predominantly from the state of Pernambuco and, more specifically, from the metropolitan region of Recife, restricting the power of generalizing the results to other regions of the country. Another limitation is the predominance of female participants with high education who may have facilitated the achievement of good psychometric results, as it places the instrument in more favorable conditions for its understanding and resolution.
Despite these limitations, this study contributes to the development of knowledge about emotion regulation by presenting a Brazilian instrument, with good psychometric properties, to assess the ability to regulate emotions. Its use will make it possible to investigate the participation of this ability in various other fields, such as education, love relationships, mental disorders, etc. Moreover, the results suggested the need to consider the regulation of emotions as a complex process that involves different types of information processing and that should also be better investigated in future studies.