Protocol for Investigating the Process Underlying Responses in Personality Assessments

Protocolo para Investigação do Processo Subjacente às Respostas em Avaliações da Personalidade

Protocolo para la Investigación del Proceso Subyacente a las Respuestas en Evaluaciones de Personalidad

Abstract

The literature investigating the process underlying the answers given in self-report tests for personality assessments is scarce. This study aimed to develop a protocol to investigate the response process of people who responded to a self-report instrument for personality assessment. It also sought evidence of content validity for this protocol. The protocol presented focused on grandiosity, representing the narcissistic functioning. A total of 35 people answered the Dimensional Clinical Personality Inventory (IDCP) and the Personality Inventory for DSM-5 (PID-5). Favorable evidence was identified, indicating the proper functioning of the developed protocol, since the literature showed consistent information about it. For example, it was observed that people who scored higher in grandiosity on self-report tests were those who chose the alternatives that represented grandiosity characteristics and were also those who presented a higher baseline to consider someone as narcissistic. The protocol is expected to be replicable by other researchers who aim to verify the response strategies adopted in self-report type personality tests.

Keywords:
personality measures; personality disorders; test validity

Resumo

A literatura é escassa quanto a estudos investigando o processo subjacente às respostas dadas em testes de autorrelato para avaliação da personalidade. Este estudo teve como objetivo desenvolver um protocolo para investigação do processo de resposta de pessoas que responderam a um instrumento de autorrelato para avaliação da personalidade. Buscaram-se também evidências de validade de conteúdo desse protocolo. O protocolo aqui apresentado teve como foco o traço grandiosidade, representando o funcionamento narcisista. Para tanto, 35 pessoas responderam o Inventário Dimensional Clínico da Personalidade (IDCP) e o Personality Inventory for DSM-5 (PID-5). Evidências favoráveis foram identificadas, indicando o funcionamento apropriado do protocolo desenvolvido, já que foram encontradas informações consistentes de acordo com a literatura. Por exemplo, foi observado que pessoas que pontuaram mais alto em grandiosidade nos testes de autorrelato, foram aquelas que escolheram as alternativas que representavam o traço grandiosidade e também foram aquelas que apresentaram uma linha de base mais alta para considerar alguém como narcisista. Espera-se que o protocolo seja replicável por outros pesquisadores que objetivem verificar as estratégias de respostas adotadas em testes de personalidade no formato de autorrelato.

Palavras-chave:
medidas de personalidade; distúrbios da personalidade; validade do teste

Resumen

En la literatura son escasos estudios que investigan el proceso subyacente a las respuestas contestadas en pruebas de autoinforme para evaluar la personalidad. Este estudio tuvo como objetivo desarrollar un protocolo para la investigación del proceso de respuesta de los participantes que respondieron a un instrumento de autoinforme para evaluar la personalidad. También se buscó evidencias de validez de contenido de este protocolo. El protocolo presentado se centró en el rasgo grandiosidad, que representa el funcionamiento narcisista. Para ello, 35 personas respondieron el Inventario Dimensional Clínico de Personalidad (IDCP) y el Personality Inventory for DSM-5 (PID-5). Se encontraron evidencias favorables, que indican el correcto funcionamiento del protocolo desarrollado, ya que las informaciones encontradas fueron consistentes con la literatura. Por ejemplo, los participantes que obtuvieron una puntuación más alta en grandiosidad en las pruebas de autoinforme fueron los que eligieron las alternativas que representaban el rasgo grandiosidad y también los que presentaron una línea de base más alta para considerar a alguien como narcisista. Se espera que el protocolo sea replicable por otros investigadores que tengan como objetivo verificar las estrategias de respuestas adoptadas en las pruebas de personalidad en el formato de autoinforme.

Palabras clave:
medidas de personalidad; trastornos de la personalidad; validez del teste

According to the Standards for Psychological and Educational Testing (American Educational Research Association, American Psychological Association & National Council on Measurement in Education, 2014, p. 15), validity evidence based on response processes is related to the “questioning test takers from various groups, making up the intended test-taking population about their performance strategies or responses to particular items, can yield evidence that enriches the definition of a construct.” Studies on the response processes aim to understand the strategies used by the subjects to answer a particular item or set of items on a test.

There are some suggestions in the literature for investigating this process, such as the analysis of responses by structured techniques (e.g., interviews) (Padilla & Benítez, 2014; Sireci, 2012), think-aloud (Ericsson & Simon, 1999; Gadermann, Ghun, & Zumbo, 2011), and focus group (Webber & Huxley, 2007). According to Padilla and Benítez (2014), the vignettes-situations in which respondents make a judgment about a given story-re also a tool that shows good understanding the response process.

The cognitive field of study was the one that presented the highest number of publications found by Padilla and Benítez on the response process. However, other areas involving psychological aspects also use the response process. For instance, Backlund, Skanér, Montgomery, Bring, and Strender (2003) investigated how physicians decided on medications and guidelines when treating patients with a given diagnosis; Barnhofer, Jong-Meyer, Kleinpass, and Nikesch (2002) verified the strategies for recovery of memories in patients diagnosed with depression; and Reze (2014) analyzed the process of maternal decisions regarding palliative cardiac surgery for their children through a semi-structured interview. Response processes verification studies assessing typical psychological constructs, such as personality, are scarce, and surveys with personality tests involving task resolution are more common (Acklin & Wu-Holt, 1996; Exner, 1996). The same cannot be said about the self-report measures, whose field of study presents an apparent gap.

One of the few studies in the area was carried out by Langston and Sykes (1997). Participants’ beliefs were assessed and related to the responses to the NEO-FFI self-report test and also by a semi-structured interview, conducted according to the think-aloud procedure. The authors concluded that knowing the beliefs underlying the response of the individual to the test (in this case, the NEO-FFI) may facilitate the understanding of personality functioning.

Based on Bornstein (2007), in cases of self-report tests, the response process is related to introspection, in which the person assesses how much a given statement has to do with him/her; to comparison, in which the person compares him or herself with the people around regarding the statements; and to communicative/self-representativeness, about how much the person is prone (voluntarily or not) to report about him or herself in the test.

Considering the procedures underlying the search for validity evidence based on the response process and the lack of studies in the area, this study aimed to develop a protocol to investigate the response process of people who answered a self-report instrument for the assessment of personality traits. In this sense, the developed and administered protocol is expected to be replicated by other researchers according to their demands, in situations where it is desired to relate the strategies used by people to respond personality tests in a self-report format. This study focuses on the protocol presentation; however, we also present initial data using it.

For this purpose, we carried out a survey on procedures that could be used to make up the protocol and, then, we performed the administration. For this study, the choice of the personality trait, grandiosity, had illustrative purposes, both testing the possibility of the developed protocol use, and demonstrating how the protocol should be used. Finally, it is important to consider that the data obtained from the protocol, when related to items of personality tests, may give evidence of validity based on the response process.

Method

Participants

The study included 35 students, aged 18 to 35 (M = 22.11; SD = 4.87), who were psychology students at a University in the countryside of the state of São Paulo. The sample was mostly female (68%; n = 24), Caucasian (71.4%; n = 25), and single (88.6%; n= 31). When asked about psychiatric treatment, 11.4% (n = 4) of participants reported having undertaken it and are still doing it; regarding psychological treatment, 34.2% (n = 12) reported have undertaken it, but only 11.4% (N = 4) remain in therapy.

Instruments

The Dimensional Clinical Personality Inventory - IDCP (Carvalho & Primi, 2015). It is a self-report test for the pathological personality traits assessment based on the diagnostic criteria found in Theodore Millon’s theory and in the fourth edition of the Diagnostic Manual and statistical of Mental Disorders. It consists of 163 items, grouped into 12 dimensions, which must be answered by a Likert-like scale of 4 points ranging from “it has nothing to do with me” to “it has a lot to do with me”. Studies demonstrate the adequacy of the IDCP from the psychometric perspective (Carvalho, 2014; Carvalho & Primi, 2015, 2016; Carvalho, Primi, & Stone, 2014).

The Personality Inventory for DSM-5 - PID-5 (Krueger, Markon, Watson, & Skodol 2012) is a self-report test aimed at assessing the 25 facets of maladaptive personality traits described in section III of the DSM-5, which can be combined into five domains. It consists of 220 items with Likert-like responses ranging from 0 for “very false or often false” up to 3 for “very true or often true.” There were no national studies verifying the psychometric properties of the Brazilian version of the instrument, but Krueger et al. (2012) presented data indicating the adequacy of the original test. However, studies have successfully used the Brazilian version in validity studies (Carvalho & Sette, 2017; Carvalho & Silva, 2016).

We administered the set of items composing Grandiosity dimension, and the protocol focused on this construct for the two instruments. In the IDCP, the items related to this dimension refer to a tendency characterized by exaggerated need for acknowledgement and admiration of others, persecutory beliefs, as well as beliefs of merit and superiority (Carvalho & Primi, 2015). In the PID-5 (Krueger et al., 2012), in turn, the items refer to beliefs of superiority and merit. For the protocol development, we used the grandiosity pathological characteristics found in both instruments. In addition, we used a research protocol of the response process, which was developed by the authors of this study. Detailed description of the instrument can be verified in Procedure.

Procedure

Protocol development. A search was carried out in the literature to find procedures that could be used to build a protocol to response process assessment by people who respond to a personality test in a self-report format. The search was unsystematic in databases such as Scielo, PubMed, Science Direct, and Google Scholar. From these bases, we found studies using unstructured procedures to access the response process, such as interviews based on the think-aloud procedure, and only one procedure structured with the same purpose, created by Langston and Sykes (1997), in which the authors developed statements to access the beliefs of participants. As the objective of this study was to develop a protocol that would be replicable by other researchers, according to the personality trait desired to be assessed, we opted for a structured format for the material, facilitating its replication.

Therefore, one of the sources used as a basis was the proposal by Langston and Sykes (1997), i.e., the presentation of beliefs concerning the evaluated construct. Furthermore, based on Padilla and Benítez (2014), who suggest the use of procedures that allow to evaluate the way in which the subjects judge situations, statements in the form of vignettes, developed based on Castillo-Díaz and Padilla (2013) and Kirschbaum and Hoelz (2014) and statements in the format of critical incidents, developed based on Backlund et al. (2003), Barnhofer et al. (2002), and Ployhart and Ehrhart (2003) were also used. For this study, we selected grandiosity as the personality trait to be investigated.

The development of the vignettes, critical incidents and issues linked to them were based on the focused trait and, more specifically, on the (12) IDCP items and (6) items of the PID-5, representing the grandiosity pathological personality trait. The total set of 17 items were grouped by the authors of the study in 7 categories, established in accordance with the content of these items. These categories were used as the basis for the number of vignettes and critical incidents developed.

Each critical incident is followed by five questions, (a) regarding what the respondent would do if he/she were the character of the situation; (b) a justification for the choice in the previous question; (c) how much the person would act according to the chosen alternative in (a); (d) how much the person believes others would act according to the chosen alternative; and (e) how the person thought to answer the question (a). As an example, Figure 1 is the first critical incident of the first protocol set.

Figure 1
Critical incident example.

Six questions were prepared for the vignettes. The first assessed how the respondent would describe, through adjectives, the character of the vignette; the second, how the person thought to answer the previous question; in the third, the respondent should assign levels (lower, moderate, and higher) to the character of the vignette, to him/herself, and to people in general, based on the adjective chosen in the first question; then, in the next question the respondent had to mark the beliefs that represented the character on the vignette, with reference to the list of beliefs presented; in the penultimate question, the respondent had to choose the alternative that best represented how intense the character was in the chosen adjective, the respondent and the people in general; and, finally, the respondent should answer how much he/she believed to be honest at answering on the adjective chosen for him/herself. The first four questions were based on the adjective chosen by the respondent; in the last two, the adjective that was typically related to how the character on the vignette (established by the authors) was imposed. Figure 2 and Figure 3 are examples of the first vignettes of the first set of the protocol.

Figure 2
Vignette example (part 1).

Figure 3
Vignette example (part 2).

The study participants responded to the vignettes and critical incidents for the seven categories created based on the items of the instruments. It is noteworthy that the order of presentation of the issues, in an attempt not to bias the response of the subjects, was as follows, (1) the critical incident and their respective questions; (2) the vignette and the first four questions; (3) the last two questions related to the vignettes (whose adjective was imposed according to the category); and, after responding to points 1, 2 and 3 for the seven categories, the subject should answer the items in the IDCP and PID-5, followed by two questions. The first, related to how the subject responded to the items; and second, to how honest he was at performing this task. To access the protocol, please contact the authors of this study.

Data collection. The application was done collectively, with 17 people in a group and 16 in another, lasting 30 minutes on average. For each group, a researcher was present to answer questions and ensure the participants were responding to the questions of the protocols.

Data analysis. After collection, the data were tabulated and analyzed using SPSS software according to the study objectives, proceeding to descriptive and inferential analyses (specifically, t test/ANOVA and Cohen’s d). The homogeneity test (i.e., Leven´s test) was not significant. We focused more in the effect size than in the significance, given the sample size.

Ethical Considerations

This study followed the research ethical procedures in accordance with the Declaration of Helsinki (WMA, 2013), and was approved by a Research ethics committee (protocol approval number is CAAE: 21992113.1.0000.5514).

Results and Discussion

In the first analysis, the participants (N = 35) were divided into two different groups, one composed by subjects who chose the response related to grandiosity in the critical incident, a construct that is typical of narcissistic functioning (American Psychiatric Association [APA], 2013; Millon, 2011), and the other by individuals who chose solutions related to the dependent and impulsive personality traits. Most participants (N = 10; 28.5%) chose the focus solution in three of the seven protocols, nine participants chose the focus solution in two protocols, six subjects chose it in four protocols, six individuals chose only in one protocol, and two people in five protocols, and no participant chose the focus solution in six or seven of the protocols. In addition, two people did not choose the solution related to grandiosity in any of the protocols. It is noteworthy that the solutions were developed by the authors based on the set of items of the IDCP and the PID-5, both for the assessment of pathological personality traits (Carvalho & Primi, 2015; Krueger et al., 2012), therefore, they tended to express maladaptive resolutions in the situations of critical incidents.

Using ANOVA, we found that people who chose narcissistic solutions in five protocols presented significantly higher scores in grandiosity items of the IDCP and PID-5 when compared to those who chose the focus solution in three or less protocols (p < = 0.05). These data demonstrate a consistency of the participants’ functioning throughout the protocols, suggesting that the assessment of traits by the instruments also occurred in the protocols presently developed. Moreover, this pattern was expected if the protocol were truly measuring narcissistic characteristics (specifically, grandiosity) such as the other tests. Thus, a first important aspect about understanding people with higher scores in grandiosity is that they tend to pick solutions that exhibit narcissistic elements (e.g., disregard for others and requiring special treatment) (APA, 2013). Table 1 presents the means of participants in the instruments according to having chosen (or not) the solution focus in critical incidents. The first column of the table expresses the category of the critical incident, since the administration protocol was divided into seven categories; the next column shows the specific functioning of each category, always related to grandiosity.

Table 1
Mean Comparison of Participants in the IDCP and PID-5 According to Choice in the Critical Incident

Critical incident	Related functioning	Focus	Not focus	F(df)	p	d
		M(SD)	M(SD)
1	Need for acknowledgement	1.81(1.03)	1.68(0.45)	3.38(33)	0.75	0.26
2	Jealousy	1.75(0.52)	1.60(0.42)	0.53(33)	0.47	0.32
3	Beliefs of Superiority	1.72(0.48)	1.67(0.48)	0.16(33)	0.68	0.10
4	Self-centered	1.64(0.49)	1.74(0.46)	0.28(33)	0.59	0.21
5	Persecutory Beliefs	1.81(0.52)	1.45(0.25)	8.95(33)	0.01	0.81
6	Attention Seeking	1.84(0.55)	1.61(0.43)	3.29(33)	0.78	0.49
7	Selfishness	2.62(-)	1.66(0.45)	- (33)	-	2.16

• Note. Focus (participants who chose the solution related to grandiosity in the critical incident); Not focus (participants who did not choose the solution related to grandiosity in the critical incident). F and p were not calculated for selfishness, as there was no standard deviation in focus group.

It is observed in critical incidents that participants who mostly chose the solution related to grandiosity had higher mean scores in the instruments when compared to those who chose solutions related to dependent and impulsive traits. However, this result was not found in the incident related to the self-centered functioning, to the extent that participants who did not choose the focus solution showed a higher mean score with a low magnitude effect (d = 0.21), and among these (n = 16), most (n = 12) chose the response related to impulsivity. The data collected do not allow to directly ascertain the tendency of choice for the impulsivity solution, but observing the situation (Luiza was walking on a street nearby her house and noticed that two people looked and commented on something when she passed.), what may have happened is that, faced with the fact that people actually looked and commented on something, the more likely would be to question people about what they were saying (as highlighted in the impulsivity alternative). Table 2 presents the participants’ response patterns according to the choice or not of the focus resolution in critical incidents, listed 1-7.

Table 2
Responses of Participants in the Critical Incidents According to Choice of Focus Solution

CI	G/D/I (p)	Adequacy	Would act	People would act	Thought about
		+Ad./~Ad./none(p)	Yes/Maybe/No	Majority/50%/Minority	Himself/People/Both/nota
1	2/32/1	1/1/0	1/1/0	0/2/0	1/1/0/0
2	20/8/7	7/12/1	10/8/2	4/13/3	12/3/5/0
3	12/17/6	1/5/6	4/2/6	5/3/4	6/3/2/1
4	19/4/12	13/4/2	7/9/3	3/11/5	10/1/6/2
5	23/11/1	8/12/3	4/16/3	5/16/2	8/3/12/0
6	11/23/1	2/5/4	4/4/3	2/5/4	3/1/6/1
7	1/33/1	0/1/0	0/0/1	0/0/1	1/0/0/0

• Note. CI (critical incident); G (chose the option related to grandiosity in the critical incident); D (chose the option related to dependency in the critical incident); I (chose the option related to impulsivity in the critical incident); +Ad. (chose the solution related to grandiosity for considering it the most adequate); ~Ad. (chose the solution related to Grandiosity, for considering it partially adequate); none (none of the above).

Overall, people who chose the grandiosity solution tended to consider this solution as appropriate or as the most appropriate, given the options. They also tended to say they would act or might have acted according to the grandiosity solution, and, although they also tended to believe that people in general would act similarly, this pattern is less evident, demonstrating that people who interpreted the grandiosity solution as the most appropriate do not necessarily believe that this is a consensus. We must consider that the assessment of the person in the face of an explicit task, such as in this case, is introspection (Bornstein, 2007), that is, the person evaluates what he or she would do and then evaluate people around (compared to them). Thus, the way to assess what he/she would do in a given situation and what others would do does not necessarily have to be equivalent. Still, it is worth noting that most people reported having a greater perception of relying only on themselves to answer what they would do in a given situation, rather than reporting a process in which the behavior of people in general is considered.

Additionally, although not part of the scope of this study, it is notable that the dependence and impulsivity solutions were based on the items of the dimensions Dependence and Impulsivity from the IDCP (Carvalho & Primi, 2015). The response pattern evidenced that most of the answers to critical incidents occurred around the dependent functioning (M = 18.2; SD = 11.4), followed by responses associated with grandiosity (M = 12.5; SD = 8.6), and impulsivity (M = 2.7; SD = 2.6) is equivalent to the agreement pattern for the IDCP items, as we can observe in the study by Carvalho and Primi. This suggests that the developed situations are equivalent to the items of the instrument and, therefore, suitable for the research purpose Table 3 shows the response patterns of participants who chose the focus solution in incidents according to the choice of focus adjective on the vignettes.

Table 3
Responses from Participants in Vignettes According to Choice of Focus Solution

Focus Adjective	Chose the focus adjective	Compared the character	Considers himself (adjective) in relation to the character	Considers people (adjective) in relation to the character	Honesty reported
	Yes/No	Himself/People/Both/None	-/equal/+	-/equal+-	T/M-/M+/N
Conceited	1/1	1/1/0/0	0/1/1	1/0/1	0/0/2/0
Boastful	2/18	7/7/6/0	0/3/17	1/4/15	11/6/2/1
Arrogant	10/2	4/6/2/0	0/1/11	1/3/8	9/3/0/0
Presumptuous	13/6	9/6/4/0	0/6/13	3/6/10	12/6/1/0
Suspicious	21/2	11/2/10/0	0/3/20	2/5/16	11/6/5/1
Pretentious	10/1	3/4/3/1	0/3/8	2/5/4	6/5/0/0
Selfish	1/0	1/0/0/0	0/0/1	0/0/1	1/0/0/0

• Note. Himself (compared the character to himself in order to confer himself the adjective related to grandiosity); People (compared the character to most people to confer himself the adjective related to grandiosity); Both (compared to both); None (none of the previous alternatives); T (totally honest); M- (moderately honest as it says it is less than it actually is); M+ (moderately sincere, once it says it is more than it actually is); N (not honest when associated to the focus adjective).

In most vignettes, participants who chose the grandiosity solution as the most appropriate were those who also recognized the character (of the vignette) as having some narcissistic trait (see first column of Table 3). Similarly to what was previously shown in critical incidents, these people reported comparing the characters (in vignettes) with themselves to conclude about the most appropriate adjective, rather than thinking about the character in relation to people they know. Along with this, we observed that these people, who represent the group with the highest mean in the IDCP and PID-5 items in grandiosity, tended to evaluate themselves as having more narcissistic characteristics, or at least on a similar level, compared to all the characters in the vignettes.

At first this finding may suggest that these people recognize themselves as having some level of narcissistic trait, but when we aggregate these data to the assessment that this group of people makes of the people who they know, i.e., also assessing them as more narcissistic than the characters of vignettes, we observe that the data in fact suggest that the baseline to consider a functioning or behavior as narcissistic tends to be high and that is probably the reason they do not recognize the acts described in vignettes as narcissistic. Along with this, these people tended to report having been completely honest when answering the protocol. Table 4 shows the means for beliefs of grandiosity, associated with the character, based on the choice or not of the focus solution in the critical incident. To obtain the belief scores, we assigned a score of 1 when the individual signaled a high belief of grandiosity (i.e., a belief representing the narcissistic functioning for each adjective in the seven categories), −1 when a low belief of grandiosity was marked, and 0 when the belief was not marked; after obtaining the raw score for each subject, we calculated the mean score for each group.

Table 4
Means of Beliefs Associated to the Character According to Choosing or Not the Focus Solution in Critical Incidents

Adjective	Beliefs		Beliefs		F(df)	p	d
	High	Low	Focus	Not Focus
			M(DP)	M(DP)
Conceited	2	4	-1.50 (0.70)	0.90 (1.07)	9.65(33)	0.01	2.28
Boastful	4	1	2.00 (0.56)	1.66 (0.72)	2.47(33)	0.12	0.54
Arrogant	4	2	2.33 (1.49)	2.56(1.27)	0.23(33)	0.63	0.17
Presumptuous	5	0	3.68 (1.24)	3.43 (1.78)	0.24(33)	0.62	0.17
Distrustful	4	0	3.04 (1.39)	1.33 (1.43)	11.70(33)	0.01	1.22
Show-off	5	2	2.36 (2.06)	2.58 (1.72)	0.19(33)	0.74	0.12
Selfish	4	2	3.00 (0.00)	2.17 (2.09)	0.15(33)	0.69	0.40

• Note. Focus (participants who chose the solution related to grandiosity in critical incident); Not the focus (participants who did not choose the solution related to grandiosity in the critical incident).

The first column shows the adjectives for each category; the second column shows the number of high and low beliefs by category; and in the other columns, the mean of each group and the size effect found. We can observe that participants who chose the solution related to grandiosity in critical incidents, presented higher means in the high beliefs, i.e., they tended to choose the beliefs representing high levels in the grandiosity construct and not choosing beliefs representing low levels, in three cases (Boastful, Suspicious, and Selfish) with expressive effect sizes. On the other hand, the group of people who did not choose the grandiosity solution presented higher mean in one case (conceited) with expressive effect size. This may suggest that people with narcissistic tendencies can better understand narcissistic functioning and, therefore, recognize the beliefs as typical in cases of more narcissistic people (i.e., vignette characters). Thus, the ability to identify the underlying beliefs in others, when typical of themselves seems to be part of the response process.

These data corroborate the findings of Langston and Sykes (1997), as they seem to help differentiate people with lower and higher levels in the latent construct. For example, we observed in this study that people with higher scores on self-report items were those that tended to choose the grandiosity solution and were also those who chose more often high beliefs in the latent construct. In this sense, another element evident in the response process is the ability to recognize beliefs typical of the functioning assessed by people who make up the group with the highest level in this construct.

Based on results presented, this study aimed to develop an investigative protocol on the response process of individuals who respond to self-report instruments assessing personality traits. Besides the application of the protocol to seek response process validity evidence for assessment tests, as highlighted by Langston and Sykes (1997), researchers could use it as a way to study causal relation between behaviors and underlying psychological process (e.g., beliefs), and also as a tool helping in the development of personality theories.

We observed positive evidence regarding the adequate functioning of the material produced. In other words, a corresponding response pattern between the items of self-report tests and the developed protocol was observed. Despite the positive results with the developed protocol, the empirical data presented is just an initial stage to bring validity to the protocol. More studies are needed in order to establish the strengths and weaknesses of the instrument. Some limitations must be accounted. The sample size and the restriction for the assessment of one construct (grandiosity) are relevant limitations of this study. Moreover, while there is no reason to believe that the Brazilian version of PID-5 will show a different performance in Brazil comparing to the original version (i.e., EUA), the absence of specific validity studies with the Brazilian version can bring concerns on the adequacy in using it in this study.

We expect that the material developed in this study will be replicated by other researchers aiming to relate the response strategies used by respondents in personality instruments in the self-report format. Specifically, future studies should verify the need to use the protocol in its full version or if short optimized versions are feasible.

References

• Acklin, M. W., & Wu-Holt, P. (1996). Contributions of cognitive science to the Rorschach technique: Cognitive and neuropsychological correlates of the response process. Journal of Personality Assessment, 67(1), 169-178. doi:10.1207/s15327752jpa6701_13
  » https://doi.org/10.1207/s15327752jpa6701_13
• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). The standards for educational and psychological testing. Washington, DC: AERA.
• American Psychiatric Association. (2013). Diagnostic and Statistical Manual of Mental Disorders: DSM-5 (5th ed.). Washington, DC: Author.
• Backlund, L., Skanér, Y., Montgomery, H., Bring, J., & Strender, L.-E. (2003). Doctor’s decision process in a drug-prescription task: The validity of rating scales and think-aloud reports. Organizational Behavior and Human Decision Processes, 91(1), 108-117. doi:10.1016/S0749-5978(02)00529-0
  » https://doi.org/10.1016/S0749-5978(02)00529-0
• Barnhofer, T., Jong-Meyer, R., Kleinpass, A., & Nikesch, S. (2002). Specificity of autobiographical memories in depression: An analysis of retrieval processes in a think-aloud task. British Journal of Clinical Psychology, 41(Pt 4), 411-416.
• Bornstein, R. F. (2007). Toward a process-based framework for classifying personality tests: Comment on Meyer and Kurtz (2006). Journal of Personality Assessment, 89(2), 202-207. doi:10.1080/00223890701518776
  » https://doi.org/10.1080/00223890701518776
• Carvalho, L. F. (2014). Avaliação dos transtornos da personalidade no Brasil: O Inventário Dimensional Clínico da Personalidade [Personality disorders assessment in Brazil: The Dimensional Clinical Personality Inventory]. In C. R. Campos & T. C. Nakano (Orgs.), Avaliação psicológica direcionada a populações específicas: Técnicas, métodos e estratégias [Psychological assessment directed to specific populations: Techniques, methods and strategies] (pp. 163-180). São Paulo, SP: Vetor.
• Carvalho, L. F., & Primi, R. (2015). Development and internal structure investigation of the Dimensional Clinical Personality Inventory (DCPI). Psicologia: Reflexão e Crítica, 28(2), 322-330. doi:10.1590/1678-7153.201528212
  » https://doi.org/10.1590/1678-7153.201528212
• Carvalho, L. F., & Primi, R. (2016). Prototype matching of personality disorders with the Dimensional Clinical Personality Inventory. Psicologia: Teoria e Pesquisa, 32(2), e322214. doi:10.1590/0102-3772e322214
  » https://doi.org/10.1590/0102-3772e322214
• Carvalho, L. F., Primi, R., & Stone, G. E. (2014). Psychometric properties of the Inventário Dimensional Clínico da Personalidade (IDCP) using the Rating Scale Model. Avances en Psicología Latinoamericana, 32(3), 433-446. doi:10.12804/apl32.03.2014.09
  » https://doi.org/10.12804/apl32.03.2014.09
• Carvalho, L. F., & Sette, C. P. (2017). Revision of the criticism avoidance dimension of the Dimensional Clinical Personality Inventory. Estudos de Psicologia (Campinas), 34(2), 219-231. doi:10.1590/1982-02752017000200004
  » https://doi.org/10.1590/1982-02752017000200004
• Carvalho, L. F., & Silva, G. F. C. (2016). Review of the self-sacrifice dimension of the clinical Dimensional Personality Inventory. Psicologia: Reflexão e Crítica, 29, 6. doi:10.1186/s41155-016-0022-z
  » https://doi.org/10.1186/s41155-016-0022-z
• Castillo-Díaz, M., & Padilla, J. L. (2013). How cognitive interviewing can provide validity evidence of the response processes to scale items. Social Indicators Research, 114(3), 963-975. doi:10.1007/s11205-012-0184-8.
  » https://doi.org/10.1007/s11205-012-0184-8
• Ericsson, K. A., & Simon, H. A. (1999). Protocol analysis: Verbal reports as data (2nd ed.). Cambridge, MA: MIT Press.
• Exner, J. E., Jr. (1996). Critical bits and the Rorschach response process. Journal of Personality Assessment, 67(3), 464-477. doi:10.1207/s15327752jpa6703_3
  » https://doi.org/10.1207/s15327752jpa6703_3
• Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2011). Investigating the substantive aspect of construct validity for the satisfaction with life scale adapted for children: A focus on cognitive processes. Social Indicators Research, 100(1), 37-60. doi:10.1007/s11205-010-9603-x
  » https://doi.org/10.1007/s11205-010-9603-x
• Kirschbaum, C., & Hoelz, J. C. (2014). A confiança em situações ambivalentes e incongruentes: A utilização de vinhetas como método exploratório [Confidence in ambivalent and incongruent situations: The use of vignettes as an exploratory method] RAM: Revista de Administração Mackenzie, 15(3), 42-68. doi:10.1590/1678-69712014/administracao.v15n3p42-68
  » https://doi.org/10.1590/1678-69712014/administracao.v15n3p42-68
• Krueger, R. F., Derringer, J., Markon, K. E., Watson, D., & Skodol, A. E. (2012). Initial construction of a maladaptive personality trait model and inventory for DSM-5. Psychological Medicine, 42(9), 1879-1890. doi:10.1017/S0033291711002674
  » https://doi.org/10.1017/S0033291711002674
• Langston, C. A., & Sykes, W. E. (1997). Beliefs and the big five: Cognitive bases of broad individual differences in personality. Journal of Research in Personality, 31(2), 141-165. doi:10.1006/jrpe.1997.2178
  » https://doi.org/10.1006/jrpe.1997.2178
• Millon, T. (2011). Disorders of personality: Introducing a DSM/ICD spectrum from normal to abnormal (3rd ed.). Hoboken, NJ: Wiley.
• Padilla, J. L., & Benítez, I. (2014). Validity evidence based on response processes. Psicothema, 26(1), 136-144. doi:10.7334/psicothema2013.259
  » https://doi.org/10.7334/psicothema2013.259
• Ployhart, R. E., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and Assessment, 11(1), 1-16. doi:10.1111/1468-2389.00222
  » https://doi.org/10.1111/1468-2389.00222
• Reze, B. (2014). O processo de decisão da mãe sobre a cirurgia cardíaca para o filho [The decision process of the mother on the cardiac surgery for the son]. Psicologia: Ciência e Profissão, 34(2), 288-301. doi:10.1590/1982-3703000072012
  » https://doi.org/10.1590/1982-3703000072012
• Sireci, S. G. (2012, April). “De-constructing” test validation. Paper presented at the Annual Conference of the National Council on Measurement in Education, Vancouver, Canada.
• Webber, M. P., & Huxley, P. J. (2007). Measuring access to social capital: The validity and reliability of the Resource Generator-UK and its association with common mental disorder. Social Science & Medicine, 65(3), 481-492. doi:10.1016/j.socscimed.2007.03.030
  » https://doi.org/10.1016/j.socscimed.2007.03.030
• World Medical Association. (2013). World Medical Association Declaration of Helsinki: Ethical Principles for Medical Research Involving Human Subjects. JAMA. 310 (20), 2191-2194. doi:10.1001/jama.2013.281053
  » https://doi.org/10.1001/jama.2013.281053

Publication Dates

• Publication in this collection
  11 Sept 2020
• Date of issue
  2020

History

• Received
  23 Mar 2017
• Reviewed
  11 Aug 2017
• Reviewed
  30 Jan 2018
• Accepted
  12 Feb 2018