Accuracy of the semantic fluency test to separate healthy old people from patients with Alzheimer’s disease in a low education population

Acurácia do teste de fluência semântica para separar pessoas saudáveis de pacientes com doença de Alzheimer em uma população de baixa escolaridade

ABSTRACT

Objective

Evaluate the accuracy of two semantic categories of the verbal fluency test (supermarket and animal categories) to separate healthy elderly individuals and lower educated Alzheimer’s disease patients.

Methods

We evaluated 69 older adults with less than 5 years of schooling, consisting of 31 healthy elderly, and 38 patients diagnosed with Alzheimer’s disease. Semantic verbal fluency was evaluated using the animal and supermarket categories. Mann-Whitney U and Independent t Tests were used to compare the two groups, and the diagnostic accuracy of the tests was analyzed by sensitivity, specificity, likelihood ratio’s, and the Area Under the Curve (AUC).

Results

We found a significant difference between the healthy older and Alzheimer’s disease groups, in both, animal (p = 0.014) and supermarket verbal fluency (p < 0.001). The supermarket category showed better overall diagnostic accuracy (AUC = 0.840, 95% CI = 0.746-0.933; p < 0.001) compared to the animal category (AUC = 0.671, 95% CI = 0.543-0.800; p = 0.014).

Conclusion

The supermarket category of semantic verbal fluency provides better accuracy than the animal category for the identification of dementia in a Brazilian elderly population with low educational level.

Semantic verbal fluency; low education; elderly; Alzheimer’s disease

RESUMO

Objetivo

Avaliar a acurácia de duas categorias semânticas do teste de fluência verbal (categorias de supermercado e animal) para separar idosos saudáveis e pacientes com doença de Alzheimer com baixa escolaridade.

Métodos

Avaliamos 69 idosos com menos de 5 anos de escolaridade, consistindo em 31 idosos saudáveis e 38 pacientes diagnosticados com a doença de Alzheimer. A fluência verbal semântica foi avaliada nas categorias animal e supermercado. O teste de Mann-Whitney U e o teste t independente foram usados para comparar os dois grupos, e a precisão diagnóstica dos testes foi analisada por sensibilidade, especificidade, razão de verossimilhança e área sob a curva (AUC).

Resultados

Encontramos uma diferença significativa entre os grupos de idosos saudáveis e com doença de Alzheimer, tanto na fluência verbal de animais (p = 0,014) quanto na de supermercado (p < 0,001). A categoria supermercado apresentou melhor precisão diagnóstica geral (AUC = 0,840; IC 95% = 0,746-0,933; p < 0,001) em comparação com a categoria animal (AUC = 0,671; IC 95% = 0,543-0,800; p = 0,014).

Conclusão

A categoria supermercado de fluência verbal semântica fornece melhor acurácia do que a categoria animal para a identificação de demência em uma população idosa brasileira com baixo nível educacional.

Fluência verbal semântica; baixa escolaridade; idosos; doença de Alzheimer

INTRODUCTION

According to International Alzheimer’s Disease¹ , the highest prevalence of Alzheimer’s Disease (AD) is the highest number of cases occurring in Low and Middle-Income Countries (LMICs), reaching 58% of the elderly population, i.e., people over 60-79 years old¹ . Among the diagnostic evaluation methods tests and examinations to diagnose Alzheimer’s disease methods the neuropsychological assessment is used to determine the cognitive characteristics of the individual² . The neuropsychological assessment may provide differentiation of the types of dementia, together with neuroimaging and cerebrospinal biomarker exams³ . However, educational level is an issue to take into consideration in most neuropsychological assessments⁴ . Considering that illiteracy and low education levels are very common in LMICs⁵ , the use of neuropsychological tests influenced by literacy may be an even greater problem in detecting cognitive decline in this population.

It has been reported that higher educated individuals tend to perform better in neuropsychological assessments because of more efficient and dense neural networks, which compensate for cognitive losses related to the aging process⁶ . Therefore, researchers are increasingly searching for strategies to reduce the influence of socio-cultural and educational factors on neuropsychological testing in individuals who are illiterate or have low educational levels. Cut-off points, according to literacy, have been required to improve the accuracy of neuropsychological evaluations in these populations³ .

As an example of some tests that apply cut-offs based on level of education, the Mini-Mental State Examination (MMSE) is the gold standard⁷ , ⁸ , while other tests, such as the Cambridge Cognitive Examination (CAMCOG)⁹ , the Alzheimer’s Disease Assessment Scale – Cognitive Subscale – Cognitive Subscale (ADAS-Cog)¹⁰ and the Montreal Cognitive Assessment (MoCA)¹¹ , are based on specific averages and scores for the various educational ranges. The MoCA is a brief screening tool designed for screening patients with Mild Cognitive Impairment (MCI), and an additional point is given to individuals with 12 or less years of schooling to correct the educational effects¹² . All these tests have been applied in the Brazilian population and confirm the influence, even if minimal, of schooling on the results.

Cognitive functions evaluated by Verbal Fluency (VF) tests include language, semantic declarative memory, storage capacity, retrieval and executive function¹³ . VF tests are widely used in the evaluation of the elderly, as they are sensitive when used for the assessment of individuals with pathologies such as frontotemporal dementia, AD, schizophrenia, and depression¹² , ¹⁴ . The most common category used for semantic VF is the animal category. However, several other categories have also been adopted are in use. da Silva et al. ¹⁵ verified an interaction between level of illiteracy and semantic categories, showing that the animal category was influenced by education¹⁴ . Brucki and Rocha¹⁶ also verified a strong influence of education in VF, and the performance was not associated with age or gender. The authors pointed out the influence of environment in animal VF, showing that the most commonly mentioned animals were “dog” and “horse”¹⁶ .

Nielsen and Waldemar¹⁷ investigated the effect of literacy on semantic VF in an immigrant population, considering the ecological relevance of the semantic category. They compared animals and supermarket categories of VF and showed that the effect of literacy in semantic VF was reduced in the supermarket category, compared to that in the animal’s category¹⁷ . More recently, Nielsen et al. ¹⁸ verified that the acculturation level predicted supermarket item VF in Turkish immigrants, showing that factors like culture and language can affect the performance¹⁷ . Indeed, compared to the variety of animals and supermarket items, there is a greater diversity of supermarket items in the daily lives of the elderly. Therefore, considering the simplicity of the test, VF related to supermarket items (supermarket VF) may have greater ecological validity and sensitivity than animal VF in elderly populations.

In the present study, we evaluated the accuracy of the two semantic categories for the VF test, namely an animal and a supermarket category to identify healthy elderly (HE) and Alzheimer’s Disease (AD) patients with a lower educational level. Our hypothesis is that supermarket category of semantic verbal fluency provides better accuracy than the animal category for the identification of dementia in a Brazilian elderly population with low educational level.

METHODS

Participants

Participants for the present study were drawn from a pool of participants recruited for a larger study in Brazil¹⁹ . More details can be seen on the article previously published¹⁹ . Older adults were recruited for this study according to the following inclusion criteria: individuals (male and female) aged over 60 years; residents of the city of Rio de Janeiro; with a clinical diagnosis of AD and HE without a diagnosis of mental illness. AD patients were recruited from a university center and a military hospital. Clinical diagnosis of AD was made by a psychiatrist or geriatrician, according to DSM-IV and National Institute of Neurological and Communicative Diseases and Stroke/Association of Alzheimer’s Disease and Related Disorders (NINCDS-ADRDA), with mild or moderate severity stage according to Clinical Dementia Classification (CDR). Diagnostic investigation for patients with AD included complete blood cell count, platelet count, blood glucose, triglycerides, total cholesterol and fractions, phosphatase, glutamic oxaloacetic transaminase and glutamic pyruvic transaminase, bilirubins, urea, creatinine, total proteins, calcium, free levels of T4, TSH, VDRL and dosage of B12 and folates. Cranial computed tomography or magnetic resonance imaging with or without spectroscopy was also performed. Exclusion criteria were: patients with neurological or psychiatric disorder comorbidities, as well as physical limitations, visual or hearing impairments that prevent cognitive testing. Individuals with depressive symptoms were not excluded from the AD group.

The control group was recruited from a research center on aging, a private hospital, and a military hospital. The control group included individuals without neurological or psychiatric diseases, who were found to be cognitively healthy by clinical evaluation and MMSE⁸ .

Only participants with ≤ 4 years of schooling, including illiteracy, were eligible for inclusion in the present study. This study was approved by the Ethics Committee of the Institute of Psychiatry of the Federal University of Rio de Janeiro and of the Naval Hospital Marcílio Dias, and all participants and caregivers provided written informed consent before any procedure took place.

Experimental procedure

All evaluations were carried out in a single visit with the older adults. The participants were informed and agreed about the research procedures before participating in an interview, various scales and a battery of neuropsychological tests. The older adults were assessed using the MMSE⁸ and two semantic verbal fluency tests (animal and supermarket categories). The animal VF subjects are asked to name as many animals as possible within one minute. In supermarket VF, subjects are asked to name as many items that can be found or bought in a supermarket, also with a time-limit of one minute²⁰ . One point is allocated for each different supermarket item. Both main categories and subordinate items of the category are accepted as correct (e.g., fruit, citrus, lemon, each scoring 1 point). Incorrect responses and repetitions were put in brackets on the score sheet¹⁹ . The number of different items named during the task period represented the score in both categories of the VF test¹⁸ . Besides neuropsychological tests, participants were assessed with the 15-item Geriatric Depression Scale (GDS-15)²¹ , the Lawton Instrumental Activities of Daily Living (IADL) Scale²² , and the Clinical Dementia Rating (CDR) Scale²³ .

Statistical analysis

The normality and homoscedasticity of the sample data were analyzed using the Shapiro-Wilk and Levene tests, respectively. The chi-squared test was used to compare the categorical variables. Mann-Whitney U and independent t -tests were applied to compare variables between groups (HE vs. AD). The associations between semantic VF (supermarket and animal categories) and cognitive and Lawton performance were analyzed using Pearson’s correlation or Spearman’s correlation tests. A receiver-operating characteristic (ROC) curve was applied in order to examine the diagnostic accuracy of the animal and supermarket VF tests, using the clinical diagnosis of AD as reference standard. Sensitivity, specificity, positive and negative likelihood ratios (LR+ and LR–), and area under the curve (AUC) values were calculated. All statistical analysis was conducted using SPSS^® software version 20.0 (IBM Corporation, New York, USA). P ≤ 0.05 was considered to indicate statistical significance.

Results

In total, 136 elderly individuals were recruited; 67 subjects did not meet the inclusion criteria of having ≤ 4 years of schooling and were excluded. Thus, 61 elderly subjects with lower educational levels (HE = 24 and AD = 37) and 8 illiterates (HE = 7 and AD = 1) were included in the final sample. A flowchart of the inclusion of participants is presented in figure 1 .

Figure 1
Flowchart of the subject selection.

The demographic characteristics of the groups are shown in table 1 . There were significant differences in age (p = 0.027), gender (p < 0.001), schooling (p = 0.008), Lawton (p < 0.001), MMSE score (p < 0.001), and CDR (p = 0.008) between the groups, showing that HE group was more functionally and cognitively preserved than the AD group. There were significant differences between the groups in both the animal (p = 0.014) and supermarket (p < 0.001) VF tests.

Table 1
Demographic characteristics

Characteristics	HE-LE (n = 31)	AD-LE (n = 38)	T/*U	P
Age, y	76 (17.00)	81 (8.75)	*2.418	0.027#
Women, n	26 (83%)	27 (71%)	**19.841	<0.001#
Schooling, y	3 (3.00)	4 (1.00)	*2.634	0.008#
Lawton, score	20 (3.00)	13 (6.00)	*-9.753	<0.001#
GDS-15, score	2 (2.00)	2 (3.75)	*-0.642	0.613
MMSE, score	22.90±3.72	19.02±4.63	-3.975	<0.001#
CDR
Mild dementia, n	0	27 (71%)	**9.739	0.008#
Moderate dementia, n	0	11 (29%)
Verbal fluency
Number of animals	12 (9.00)	9 (4.75)	*-2.806	0.014#
Number of supermarket items	16.45 ± 4.37	10.69 ± 3.74	-6.001	<0.001#

• HE: health elderly; AD: Alzheimer’s disease; LE: low education; Lawton: IADL scale; GDS: Geriatric Depression Scale; MMSE: Mini-Mental State Examination; CDR: Clinical Dementia Rating; T: independent t-test; * U: Mann-Whitney U test; ** Chi-squared test; * median (interquartile range); # P ≤ 0.05.

Diagnostic accuracy data is presented in table 2 . Overall, the supermarket category showed better accuracy (AUC = 0.840, 95% CI = 0.746-0.933) than the animal category (AUC = 0.671, 95% CI = 0.543-0.800) in differentiating between HE and AD individuals with low levels of education. Supermarket VF test scores ≥15 (the optimal cut-point based on the ROC curve) detected AD with a sensitivity of 67.74% and a specificity of 86.84%. Animal VF test scores ≥12 (the optimal cut-point based on the ROC curve) detected AD with a sensitivity of 54.84% and a specificity of 73.68%.

Table 2
Accuracy, sensibility and specificity of the verbal fluency to separate HE and AD

Verbal fluency	AUC (95% CI)	Optimal cut-point	Sensitivity (%)	Specificity (%)	Correctly classified (%)	LR+	LR-
Animals	0.671 (0.543 - 0.800)	≥12	54.84%	73.68%	65.22%	2.08	0.61
Supermarket itens	0.840 (0.746 - 0.933)	≥15	67.74%	86.84%	78.26%	5.14	0.37

• AUC: area under the ROC curve; 95% CI: confidence interval; LR+: likelihood ratio positive; LR-: likelihood ratio negative.

Both animal and supermarket VF showed significant correlations with other cognitive tests. The animal VF test presented a significant correlation with MMSE (r = 0.438, p < 0.001), CDR (r_s = -0.379, p = 0.001), and Lawton (r_s = 0.306, p = 0.011). However, the correlations were stronger between supermarket category and MMSE (r = 0.632, p < 0.001), CDR (r_s = -0.642, p < 0.001), and Lawton (r_s = 0.619, p < 0.001), showing that the supermarket category was more associated with Lawton, cognition and severity of dementia. There were no significant correlations between GDS score and animal or supermarket VF tests.

DISCUSSION

The present study aimed to evaluate the discriminatory power of two semantic criteria of VF test (supermarket and animal categories) to separate identify HE from and AD patients with lower educational level. According to our results, the supermarket VF provided the best result. a higher degree of accuracy for the identification of AD individuals with lower educational levels. In the present study, the AUC was 0.671 for the animal category VF and 0.84 for the supermarket category VF, showing that supermarket verbal fluency provides greater accuracy in this population. In a Brazilian study, Brucki and Rocha¹⁶ showed that educational performance was associated with higher fluency scores in the animal category. The cutoffs suggested (supermarket = 15.0 and animal = 12.0) in the present study corroborate previous studies in illiterate healthy subjects, both for animals¹⁵ , ¹⁶ and supermarket¹⁵ categories VF.

To our knowledge, this is the first study that investigated animal and supermarket categories of semantic VF to identify people with AD and separate them form healthy older people with low educational level. It can be speculated that the better accuracy of the supermarket categories VF test is related to activities performed by individuals that are independent from relevant socio-cultural characteristics or level of education. This is a relevant ecological categoric test, which is easy to apply and less dependent on education. A recent meta-analysis investigated cognitive screening tools for dementia in LMICs and the authors verified that few instruments were validated in low-literacy settings⁵ . The MMSE demonstrated the highest level of accuracy when used in low-literacy cohorts in Spain and Brazil, and the AUC for the MMSE reported in this meta-analysis was 0.853, very similar to the AUC observed in our result for supermarket category VF. However, it is worth mentioning that this test must be applied within a battery of tests, because although it has presented a reasonable result in specificity, its sensitivity was low, thus not allowing its use in isolation in a screening.

The poorer testing performance of the animal VF could be explained by the influence of education in this category of semantic verbal fluency task²⁴ . In another study, animal category VF was associated with education, but not with age or sex¹⁶ . However, a study of various categories of VF tests, including supermarket items, showed that age had a more significant impact than education on overall VF testing performance²⁵ . In the present study, stronger correlations were observed between VF and Lawton, especially in the supermarket category. This is expected, considering the association between cognition and education with instrumental activities of daily living (IADL)²⁶ , ²⁷ . Indeed, dementia is the leading cause of functional dependence in the elderly and IADL refers to the individual’s ability to organize and plan in the environment, be it at home or at the wider community. These activities are related to the most complex actions, such as social participation, which includes using the telephone, driving, using public transportation, and the ability to shop²⁸ . This last ability in IADL is directly related to the supermarket VF, whereas animal VF is not related to any activity that affects the daily life of the elderly.

We also observed a correlation between MMSE and semantic VF. However, the correlation was stronger for the supermarket category than for the animal category. Although MMSE is influenced by education²³ , it is not merely the number of years of education that is important, but also the quality of the education²⁹ . Normally, studies determine the levels of education based on the number of years, rather that the quality of the education. This may be an important bias, especially in Brazil, as many elderly people in LMICs have had several years of schooling but are unable to perform more complex assessments or do not have good semantic fluency in terms of words that are not common in their daily lives. Finally, the present study has some limitations that needs to be considered, such as a small sample, controls not matched for age, sex, and education, in addition, the study was carried out in an urban area, being questionable whether a sample in the rural area would have a greater ease with the VF of animals than the VF of supermarket items.

CONCLUSION

In the semantic VF test, the supermarket category provides better diagnostic accuracy than the animal category animal category for the identification of dementia in an illiterate and LE elderly population.

Figure 2
ROC curve of animal verbal fluency.

Figure 3
ROC curve of supermarket items verbal fluency.

ACKNOWLEDGMENT

This work was supported by the “Conselho Nacional de Desenvolvimento Científico e Tecnológico” under Grant (CNPq-301483/2016-7); “Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro” under Grant (Faperj-E26/202.523/2019).

REFERENCES

• ¹
  Alzheimer’s Disease International [Internet]. World Alzheimer Report 2018. Available from: https://www.alz.co.uk/info/diagnosis Accessed on: Nov 11, 2019.
  » https://www.alz.co.uk/info/diagnosis
• ²
  Franzen S, van den Berg E, Kalkisim Y, Van De Wiel L, Harkes M, van Bruchem-Visser RL, et al. Assessment of Visual Association Memory in Low-Educated, Non-Western Immigrants with the Modified Visual Association Test. Dement Geriatr Cogn Disord. 2019;47(4-6): 345-54.
• ³
  Malloy-Diniz LF, Fuentes D, Mattos P, Abreu N. Avaliação Neuropsicológica. 2ª ed. Porto Alegre: Artmed Editora; 2018.
• ⁴
  Ardila A, Bertolucci PH, Braga LW, Castro-Caldas A, Judd T, Kosmidis MH, et al. Illiteracy: the neuropsychology of cognition without reading. Arch Clin Neuropsychol. 2010;25(8):689-712.
• ⁵
  Paddick SM, Gray WK, McGuire J, Richardson J, Dotchin C, Walker RW. Cognitive screening tools for identification of dementia in illiterate and low-educated older adults, a systematic review and meta-analysis. Int Psychogeriatr. 2017;29(6):897-929.
• ⁶
  Stern Y. What is cognitive reserve? Theory and research application of the reserve concept. J Int Neuropsychol Soc. 2002;8(3):448-60.
• ⁷
  Bertolucci PH, Brucki S, Campacci SR, Juliano Y. O mini-exame do estado mental em uma população geral: impacto da escolaridade. Arq Neuropsiquiatr. 1994;52(1):1-7.
• ⁸
  Laks J, Baptista EMR, Contino ALB, Paula EOd, Engelhardt E. Mini-Mental State Examination norms in a community-dwelling sample of elderly with low schooling in Brazil. Cad Saúde Pública. 2007;23(2):315-9.
• ⁹
  Moreira IFH, Lourenço RA, Soares C, Engelhardt E, Laks J. Cambridge Cognitive Examination: performance of healthy elderly Brazilians with low education levels. Cad Saúde Pública. 2009;25(8):1774-80.
• ¹⁰
  Schultz RR, Siviero MO, Bertolucci PHF. The cognitive subscale of the “Alzheimer’s Disease Assessment Scale” in a Brazilian sample. Braz J Med Biol Res. 2001;34(10):1295-302.
• ¹¹
  Sarmento ALR. Apresentação e aplicabilidade da versão brasileira da MoCA (Montreal Cognitive Assessment) para rastreio de comprometimento cognitivo leve [dissertação]. São Paulo: Unifesp; 2009.
• ¹²
  Memória CM, Yassuda MS, Nakano EY, Forlenza OV. Brief screening for mild cognitive impairment: validation of the Brazilian version of the Montreal cognitive assessment. Int J Geriatr Psychiatry. 2013;28(1):34-40.
• ¹³
  Fichman HC, Fernandes CS, Nitrini R, Lourenço RA, Paradela EMdP, Carthery-Goulart MT, et al. Age and educational level effects on the performance of normal elderly on category verbal fluency tasks. Dement Neuropsychol. 2009;3(1):49-54.
• ¹⁴
  Rodrigues AB, Yamashita E, Chiappetta A. Teste de fluência verbal no adulto e no idoso: verificação da aprendizagem verbal. Rev CEFAC. 2008;10(4):443-51.
• ¹⁵
  da Silva CG, Petersson KM, Faísca L, Ingvar M, Reis A. The effects of literacy and education on the quantitative and qualitative aspects of semantic verbal fluency. J Clin Exp Neuropsychol. 2004;26(2):266-77.
• ¹⁶
  Brucki SMD, Rocha MSG. Category fluency test: effects of age, gender and education on total scores, clustering and switching in Brazilian Portuguese-speaking subjects. Braz J Med Biol Res. 2004;37(12):1771-7.
• ¹⁷
  Nielsen TR, Waldemar G. Effects of literacy on semantic verbal fluency in an immigrant population. Aging Neuropsychol Cogn. 2016;23(5):578-90.
• ¹⁸
  Nielsen TR, Vogel A, Gade A, Waldemar G. Cognitive testing in non‐demented Turkish immigrants‐comparison of the RUDAS and the MMSE. Scand J Psychol. 2012;53(6): 455-60.
• ¹⁹
  de Araujo NB, Nielsen TR, Engedal K, Barca ML, Coutinho ES, Laks J. Diagnosing dementia in lower educated older persons: validation of a Brazilian Portuguese version of the Rowland Universal Dementia Assessment Scale (RUDAS). Braz J Psychiatry. 2018;40(3):264-9.
• ²⁰
  Troyer AK. Normative data for clustering and switching on verbal fluency tasks. J Clin Exp Neuropsychol. 2000;22(3):370-8.
• ²¹
  Almeida OP, Almeida SA. Reliability of the Brazilian version of the Geriatric Depression Scale (GDS) short form. Arq Neuropsiquiatr. 1999;57(2B):421-6.
• ²²
  Lawton MP, Brody EM. Assessment of older people: self-maintaining and instrumental activities of daily living. Gerontologist. 1969;9(3 Pt 1):179-86.
• ²³
  Chaves MLF, Camozzato AL, Godinho C, Kochhann R, Schuh A, de Almeida VL, et al. Validity of the clinical dementia rating scale for the detection and staging of dementia in Brazilian patients. Alzheimer Dis Assoc Disord. 2007;21(3):210-7.
• ²⁴
  Benito-Cuadrado M, Esteba-Castillo S, Böhm P, Cejudo-Bolivar J, Peña-Casanova J. Semantic verbal fluency of animals: a normative and predictive study in a Spanish population. J Clin Exp Neuropsychol. 2002;24(8):1117-22.
• ²⁵
  Stokholm J, Jørgensen K, Vogel A. Performances on five verbal fluency tests in a healthy, elderly Danish sample. Aging Neuropsychol Cogn. 2013;20(1):22-33.
• ²⁶
  Brigola AG, Alexandre TS, Inouye K, Yassuda MS, Pavarini SCI, Mioshi E. Limited formal education is strongly associated with lower cognitive status, functional disability and frailty status in older adults. Dement Neuropsychol. 2019;13(2):216-24.
• ²⁷
  Carmona-Torres JM, Rodríguez-Borrego MA, Laredo-Aguilera JA, López-Soto PJ, Santacruz-Salas E, Cobo-Cuenca AI. Disability for basic and instrumental activities of daily living in older individuals. PloS One. 2019;14(7):1-13.
• ²⁸
  Pinto AH, Lange C, Pastore CA, Llano PMP, Castro DP, Santos F. Functional capacity to perform activities of daily living among older persons living in rural areas registered in the Family Health Strategy. Ciênc Saúde Colet. 2016;21(11):3545-55.
• ²⁹
  de Melo DM, Barbosa AJG, Neri AL. Miniexame do Estado Mental: evidências de validade baseadas na estrutura interna. Aval Psicol. 2017;16(2):161-8.

Publication Dates

• Publication in this collection
  01 June 2020
• Date of issue
  Apr-Jun 2020

History

• Received
  7 Apr 2020
• Accepted
  10 Apr 2020