Accessibility / Report Error

Translation, cultural adaptation, and validation of the Brazilian Portuguese version of the Higher Education Stress Inventory (HESI-Br)

Abstract

Objectives

There are no validated instruments to measure education-related stress in Brazilian university students. Thus, we aimed to translate and test the internal reliability, convergent/discriminant validity, and measurement equivalence of the Higher Education Stress Inventory (HESI).

Methods

The translation protocol was carried out by two independent translators. The instrument was culturally adapted after a pilot version was administered to 36 university students. The final version (HESI-Br) was administered to 1,021 university students (mean age = 28.3, standard deviation [SD] = 9.6, 76.7% female) via an online survey that lasted from September 1 to October 15, 2020. The factor structure was estimated using exploratory factor analysis (EFA) on the first half of the dataset. We tested the best EFA-derived model with confirmatory factor analysis (CFA) on the second half. Convergent/discriminant validity was tested using the Depression, Anxiety and Stress Scale (DASS-21). Sex, age groups, period of study, family income and area of study were used to test measurement equivalence.

Results

EFA suggested five factors: career dissatisfaction; faculty shortcomings; high workload; financial concerns; and toxic learning environment. CFA supported the five-factor model (15 items), but not a higher order factor, suggesting multidimensionality. All five factors presented acceptable internal reliabilities, with Cronbach’s α ≥ 0.72 and McDonald’s ω ≥ 0.64. CFA models indicated that the HESI-Br and DASS-21 assess different but correlated underlying latent constructs, supporting discriminant validity. Equivalence was ascertained for all tested groups.

Conclusion

The 15-item HESI-Br is a reliable and invariant multidimensional instrument for assessing relevant stressors among university students in Brazil.

Psychological stress; university; education; psychometrics; factor analysis

Introduction

Psychological stress is high among university students globally and prevalence estimates can reach 99.2%.11. Lai AY, Lee L, Wang MP, Feng Y, Lai TT, Ho L, et al. Mental health impacts of the COVID-19 pandemic on international university students, related stressors, and coping strategies. Front Psychiatry. 2020;11:584240.

2. Moutinho ILD, Lucchetti ALG, Ezequiel OS, Lucchetti G. Mental health and quality of life of Brazilian medical students: Incidence, prevalence, and associated factors within two years of follow-up. Psychiatry Res. 2019;274:306-12.

3. Dyrbye LN, Thomas MR, Eacker A, Harper W, Massie FS, Power DV, et al. Race, ethnicity, and medical student well-being in the United States. Arch Intern Med. 2007;167:2103-9.

4. Chaabane S, Chaabna K, Bhagat S, Abraham A, Doraiswamy S, Mamtani R, et al. Perceived stress, stressors, and coping strategies among nursing students in the Middle East and North Africa: an overview of systematic reviews. Syst Rev. 2021;10:136.
-55. Pacheco JPG, Giacomin HT, Tam WW, Ribeiro TB, Arab C, Bezerra IM, et al. Mental health problems among medical students in Brazil: A systematic review and meta-analysis. Braz J Psychiatry. 2017;39:369-78. While a moderate level of psychological stress may increase individuals’ resilience,66. Crane MF, Searle BJ, Kangas M, Nwiran Y. How resilience is strengthened by exposure to stressors: the systematic self-reflection model of resilience strengthening. Anxiety Stress Coping. 2018;32:1-17. exposure to a high level of stress is associated with mental health problems (e.g., insomnia, depression, anxiety, and burnout)55. Pacheco JPG, Giacomin HT, Tam WW, Ribeiro TB, Arab C, Bezerra IM, et al. Mental health problems among medical students in Brazil: A systematic review and meta-analysis. Braz J Psychiatry. 2017;39:369-78.,77. Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604.

8. Shim EJ, Jeon HJ, Kim H, Lee KM, Jung D, Noh HL, et al. Measuring stress in medical education: validation of the Korean version of the Higher Education Stress Inventory with medical students. BMC Med Educ. 2016;16:1-8.
-99. Ribeiro ÍS, Pereira R, Freire IV, de Oliveira BG, Casotti CA, Boery EN. Stress and quality of life among university students: a systematic literature review. Heal Prof Educ. 2018;4:70-7. and worse academic outcomes (e.g., lower grade point average [GPA] and higher dropout rates).1010 Kokou-Kpolou CK, Jumageldinov A, Park S, Nieuviarts N, Noorishad P-G, Cénat JM. Prevalence of depressive symptoms and associated psychosocial risk factors among French university students: the moderating and mediating effects of resilience. Psychiatr Q. 2021;92:443-57.

11. Turner J, Bartlett D, Andiappan M, Cabot L. Students’ perceived stress and perception of barriers to effective study: impact on academic performance in examinations. Br Dent J. 2015;219:453-8.
-1212. Halboub E, Alhajj MN, AlKhairat AM, Sahaqi A-AM, Quadri MFA. Perceived stress among undergraduate dental students in relation to gender, clinical training and academic performance. Acta Stomatol Croat. 2018;52:37-45. Moreover, multiple factors are associated with psychological stress among students, including academic overload,55. Pacheco JPG, Giacomin HT, Tam WW, Ribeiro TB, Arab C, Bezerra IM, et al. Mental health problems among medical students in Brazil: A systematic review and meta-analysis. Braz J Psychiatry. 2017;39:369-78.,1313. Tempski P, Bellodi PL, Paro HBMS, Enns SC, Martins, Milton A Schraiber LB. What do medical students think about their quality of life? A qualitative study. BMC Med Educ. 2012;12:106. uncertainty and insecurity about the future,1313. Tempski P, Bellodi PL, Paro HBMS, Enns SC, Martins, Milton A Schraiber LB. What do medical students think about their quality of life? A qualitative study. BMC Med Educ. 2012;12:106. low income,22. Moutinho ILD, Lucchetti ALG, Ezequiel OS, Lucchetti G. Mental health and quality of life of Brazilian medical students: Incidence, prevalence, and associated factors within two years of follow-up. Psychiatry Res. 2019;274:306-12. and lack of self-esteem and motivation.55. Pacheco JPG, Giacomin HT, Tam WW, Ribeiro TB, Arab C, Bezerra IM, et al. Mental health problems among medical students in Brazil: A systematic review and meta-analysis. Braz J Psychiatry. 2017;39:369-78. However, there are still few scales validated for screening students under high levels of psychological stress related to the higher education setting.

The Higher Education Stress Inventory (HESI)77. Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604. was developed in 2005 aiming to provide a reliable tool to measure stress in higher education level students.77. Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604. Originally inspired by the Perceived Medical School Stress (PMSS) instrument,1414. Vitaliano PP, Russo J, Carr JE, Heerwagen JH. Medical school pressures and their relationship to anxiety. J Nerv Ment Dis. 1984;172:730-6. the scale aims to measure the presence of psychological stress in settings other than medical schools. As such, it was constructed to capture many of the stressors that students are exposed to in higher education, such as those mentioned above (e.g., academic overload, etc.). The HESI has previously been used to assess stress levels among Swedish77. Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604. and Korean88. Shim EJ, Jeon HJ, Kim H, Lee KM, Jung D, Noh HL, et al. Measuring stress in medical education: validation of the Korean version of the Higher Education Stress Inventory with medical students. BMC Med Educ. 2016;16:1-8. medical students, Jordanian nursing students,1515. Masha’al D, Hayajneh AA, Tawalbeh LI. Psychometric evaluation of the Arabic version of the higher education inventory scale for nursing students. BMC Psychiatry. 2021;21:1-13. Ugandan university students,1616. Kamulegeya LH, Kitonsa PJ, Okolimong E, Kaudha G, Maria S, Nakimuli-Mpungu E. Prevalence and associated factors of alcohol use patterns among university students in Uganda. Pan Afr Med J. 2020;37:339. and physicians in their first postgraduate year.1717. Dahlin M, Fjell J, Runeson B. Factors at medical school and work related to exhaustion among physicians in their first postgraduate year. Nord J Psychiatry. 2010;64:402-8. Currently, the scale is validated for Arabic1515. Masha’al D, Hayajneh AA, Tawalbeh LI. Psychometric evaluation of the Arabic version of the higher education inventory scale for nursing students. BMC Psychiatry. 2021;21:1-13. and Korean populations.88. Shim EJ, Jeon HJ, Kim H, Lee KM, Jung D, Noh HL, et al. Measuring stress in medical education: validation of the Korean version of the Higher Education Stress Inventory with medical students. BMC Med Educ. 2016;16:1-8. Total HESI score has been associated with depressive symptoms,77. Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604.,88. Shim EJ, Jeon HJ, Kim H, Lee KM, Jung D, Noh HL, et al. Measuring stress in medical education: validation of the Korean version of the Higher Education Stress Inventory with medical students. BMC Med Educ. 2016;16:1-8. which have estimated pooled prevalence rates that vary from 24.4 to 42.6% among university students.1818. Khan MN, Akhtar P, Ijaz S, Waqas A. Prevalence of depressive symptoms among university students in Pakistan: a systematic review and meta-analysis. Front Public Health. 2020;8:603357.

19. Akhtar P, Ma L, Waqas A, Naveed S, Li Y, Rahman A, et al. Prevalence of depression among university students in low and middle income countries (LMICs): a systematic review and meta-analysis. J Affect Disord. 2020;274:911-9.

20. Gao L, Xie Y, Jia C, Wang W. Prevalence of depression among Chinese university students: a systematic review and meta-analysis. Sci Rep. 2020;10:15897
-2121. Tam W, Lo K, Pacheco J. Prevalence of depressive symptoms among medical students: overview of systematic reviews. Med Educ. 2019;53:345-54.

In Brazil, the rate of university enrollment increased by 283.4% over the past 20 years. With more than 8.6 million people2222. Barros A da SX. Expansion of higher education in Brazil: limits and possibilities. Educ Soc. 2015;36:361-90.,2323. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP). Censo da Educação Superior 2019 - Divulgação dos resultados [Internet]. 2020. https://download.inep.gov.br/educacao_superior/censo_superior/documentos/2020/Apresentacao_Censo_da_Educacao_Superior_2019.pdf
https://download.inep.gov.br/educacao_su...
in higher educational settings, there is a need to assess stress among Brazilian university students. Additionally, students living in low and middle-income countries, such as Brazil, are affected by additional socioeconomic factors, such as lower income and higher discrimination.2424. Bastos JL, Barros AJD, Celeste RK, Paradies Y, Faerstein E. Age, class and race discrimination: their interactions and associations with mental health among Brazilian university students. Cad Saude Publica. 2014;30:175-86. These factors might ultimately result in higher stress in university students.22. Moutinho ILD, Lucchetti ALG, Ezequiel OS, Lucchetti G. Mental health and quality of life of Brazilian medical students: Incidence, prevalence, and associated factors within two years of follow-up. Psychiatry Res. 2019;274:306-12.,33. Dyrbye LN, Thomas MR, Eacker A, Harper W, Massie FS, Power DV, et al. Race, ethnicity, and medical student well-being in the United States. Arch Intern Med. 2007;167:2103-9.,2525. Naser AY, Dahmash EZ, Al-Rousan R, Alwafi H, Alrawashdeh HM, Ghoul I, et al. Mental health status of the general population, healthcare professionals, and university students during 2019 coronavirus disease outbreak in Jordan: A cross-sectional study. Brain Behav. 2020;10:e01730.

To our knowledge, no instruments focusing on measuring academic stress among university students have been validated for the Brazilian population. Thus, our research aimed to 1) translate the HESI scale into Brazilian Portuguese, 2) culturally adapt it, 3) test its structure and internal reliability, 4) test its convergent/discriminant validity, and 5) test its measurement equivalence across groups selected by different characteristics in a large sample of university students from Brazil.

Methods

Recruitment and data collection

Development of the Brazilian Portuguese version of the HESI (HESI-Br) is part of the COVIDPsiq study, which is a longitudinal survey of mental health in the context of the coronavirus disease 2019 (COVID-19) pandemic. Full details can be found elsewhere.2626. Calegaro VC, Negretto BL, Weber LP, Kerber N, Zoratto G, Rodrigues L, et al. Monitoring the evolution of posttraumatic symptomatology, depression and anxiety during the COVID-19 pandemic in Brazilians (COVIDPsiq). Protoc Exch [Internet]. 2020. https://doi.org/10.21203/rs.3.pex-945/v2
https://doi.org/10.21203/rs.3.pex-945/v2...
Briefly, COVIDPsiq aimed to follow-up post-traumatic, depressive, and anxiety symptoms in Brazilians during the COVID-19 pandemic. The survey was conducted from April 2020 to February 2021 in four waves of assessment, using a non-probabilistic convenience sample. The study was publicized through social media platforms, a corporate mailing list, and digital and press media. Data were collected using the SurveyMonkey online platform. The choice of an electronic survey was based on the possibility of reaching more participants while respecting social isolation restrictions in Brazil. The research was approved by the human research ethics committee at the Universidade Federal de Santa Maria (CAAE: 30420620.5.0000.5346).

Participants

The criteria for participation in the study to validate the scale were: (a) being a native Brazilian or residing in Brazil; (b) being over 18 years of age; (c) having access to digital equipment; (d) being literate; and (e) being a university student (at any level, e.g., undergraduate, graduate, postgraduate). All individuals participated voluntarily and provided informed consent online. In total, the survey period covered approximately 11 months. The questionnaire for each phase remained available for 1 month on average. The third phase, in which the HESI-Br was administered, extended from September 1 to October 15, 2020. A total of 2,303 respondents participated in the third phase of the larger longitudinal study. Of these, 1,021 were university students and answered the HESI-Br questions. Sociodemographic data for the sample are shown in Table 1.

Table 1
- University students’ sociodemographic characteristics

Measures

Depression, Anxiety and Stress Scale (DASS-21)

Based on the tripartite model of depression and anxiety, the DASS-21 is a short version derived from the DASS-42, both developed by Lovibond and Lovibond.2727. Lovibond PF, Lovibond SH. The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther. 1995;33:335-43. It is an instrument to measure symptomatology in three domains (depression, anxiety, and stress). It has 21 items with a four-point Likert response scale (0 = strongly disagree; 3 = strongly agree). The DASS-21 was translated into Brazilian Portuguese and validated in patients from two hospitals in Southern Brazil.2828. Vignola RCB, Tucci AM. Adaptation and validation of the Depression, Anxiety and Stress Scale (DASS) to Brazilian Portuguese. J Affect Disord. 2014;155:104-9. A recent study2929. Zanon C, Brenner RE, Baptista MN, Vogel DL, Rubin M, Al-Darmaki FR, et al. Examining the dimensionality, reliability, and invariance of the Depression, Anxiety, and Stress Scale–21 (DASS-21) across eight countries. Assessment. 2020;28:1531-44. examined the psychometric properties of the DASS-21 in eight countries, including Brazil, suggesting that the DASS-21 is best represented with a general distress factor. A second-order model had acceptable fit according to Zanon et al.,2929. Zanon C, Brenner RE, Baptista MN, Vogel DL, Rubin M, Al-Darmaki FR, et al. Examining the dimensionality, reliability, and invariance of the Depression, Anxiety, and Stress Scale–21 (DASS-21) across eight countries. Assessment. 2020;28:1531-44. and was used to test convergent/discriminant validity of the HESI-Br.

Higher Education Stress Inventory (HESI) – Original version

Originally inspired by the PMSS,1414. Vitaliano PP, Russo J, Carr JE, Heerwagen JH. Medical school pressures and their relationship to anxiety. J Nerv Ment Dis. 1984;172:730-6. the HESI aims to assess the presence of educational stress in university students.77. Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604. It is a 33-item self-report instrument that uses a four-point Likert response scale ranging from 1 (does not apply at all) to 4 (applies perfectly). Ten items are reversed because they indicate absence of stress. Therefore, higher scores indicate higher educational stress levels.

The original HESI factor analysis identified a model comprising 24 items loading on seven factors. The factors also presented low to acceptable α values and were identified as: (I) Worries about future competence (α = 0.78); (II) Non-supportive climate (α = 0.71); (III) Faculty shortcomings (α = 0.69); (IV) Workload (α = 0.62); (V) Insufficient feedback (α = 0.65); (VI) Lack of commitment (α = 0.62); and (VII) Financial concerns (α = 0.59).77. Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604.

Translation and cross-cultural adaptation

Translation and cross-cultural adaptation were conducted in eight steps, according to the ISPOR Guidelines3030. Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A, et al. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR Task Force for Translation and Cultural Adaptation. Value Heal. 2005;8:94-104. and with permission from the original author. Permission to publish the final instrument in its entirety was also obtained from the original author. Translation from English to Brazilian Portuguese was performed by two independent Brazilian professionals, both specialized in psychiatry and fluent in English. (1) Initially, the original HESI was translated into Portuguese by the first and second translator. (2) After comparison of the two translations, they produced a consensus version. (3) This version was sent to a third psychiatrist, with extensive knowledge of English, who evaluated and improved the translated version. (4) Next, the HESI-Br was back-translated into English by a professional translator and compared to the original version by the translators from step 1. (5) The Brazilian version was then adapted according to the differences found in the back-translation. (6) Cross-cultural adaptation was performed using a pilot version of the scale, to which 36 students responded via Google Forms and were asked to comment on any difficulties they had with specific items. Subsequently, a video conference was held with nine undergraduate students who are part of the COVIDPsiq project, for further information on how to improve comprehensibility (cognitive interview). A final culturally adapted version, the HESI-Br (online-only Supplementary Material S1), was administered to university students who participated in the third phase of the large longitudinal study. A flowchart illustrating the process is available in Figure S1, available as online-only supplementary material.

Statistical analyses

First, 10 positive-oriented items of the translated version of the HESI scale (Q2, Q6, Q8, Q10, Q13, Q17, Q19, Q26, Q27, and Q33) were reverse-coded. The frequencies of responses per item are shown in Table S1, available as online-only supplementary material. Second, EFA and CFA were performed on randomly split halves of the dataset. Item response theory (IRT) analyses were performed on the CFA sample. Analyses of measurement equivalence and convergent/discriminant validity with DASS-21 were conducted on the whole dataset. A flowchart illustrating the data analysis plan is provided in Figure S2.

Exploratory factor analysis (EFA)

We used the first half of the dataset (n = 511) for the EFA. The Kaiser-Meyer-Olkin (KMO) statistic was calculated to verify the data’s sampling adequacy for analysis and the result was evaluated according to Kaiser.3131. Kaiser HF. An index of factorial simplicity. Psychometrika. 1974;39:31-6. A KMO cutoff of 0.5 was considered for the acceptability of individual items. Bartlett’s test of sphericity3232. Bartlett MS. The effect of standardization on a Chi-square approximation in factor analysis. Biometrika. 1951;38:337-44. was used to evaluate whether correlations between items were sufficiently large for factor analysis. A parallel analysis was conducted using weighted least squares as factoring method (a scree plot is shown in Figure S3).

EFA was conducted on a polychoric matrix of the 33 items with oblimin rotation. The best structure was selected based on the following criteria: (a) items with fewer cross-loadings; (b) items with factor loadings > 0.3; and (c) factors with at least three items per factor. The best model was further filtered to keep the number of items per factor equal across the factors, based on the items with the highest factor loadings.

Confirmatory factor analysis (CFA)

The CFA was conducted using the second half of the dataset (n = 510). It was carried out using delta parameterization and weighted least squares with diagonal weighted least square mean and variance adjusted (WLSMV) estimators. Global model fit was evaluated with root mean square error of approximation (RMSEA), comparative fit index (CFI), Tucker-Lewis index (TLI), and standardized root mean-square residual (SRMR) indices. RMSEA values lower than 0.060 and CFI or TLI values higher than 0.950 indicate a good-to-excellent model.3333. Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model. 1999;6:1-55. An SRMR less than or equal to 0.100 indicates adequate fit, and values less than 0.060 in combination with previous indices indicate good fit.3333. Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model. 1999;6:1-55. Using the EFA-derived model, we tested whether a correlated or a second-order version of the model better represents the HESI factor structure. A χ2 test was performed to test the difference between models. Factor reliability was examined using Cronbach’s α3434. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297-334. and McDonald’s ω.3535. McDonald RP. Test theory: a unified treatment. Mahwah: Lawrence Erlbaum; 1999.

Multidimensional item response theory (IRT) analysis

Item information (IIC) and item characteristic (ICC) curves were generated using the Graded Response Model for polytomous analysis and quasi-Monte Carlo expectation maximization (QMCEM) as estimation algorithm. These curves are based on the two-parameter IRT model, comprising parameter α (item discrimination) and parameter β (item difficulty).

Parameter α represents the rate at which the probability of answering a response category changes, given the construct level. It is the slope of the item characteristic curve, which is constant for all categories of the same item. Item discrimination helps to differentiate individuals with similar levels of the latent construct because it marks where, in the latent construct, the probability of answering items increases. Parameter β indicates the 50% probability of endorsing a given category or higher in the latent construct (i.e., τ thresholds) for each HESI-Br item (e.g., from “totally disagree” to “somewhat disagree”). It therefore informs the construct level that is necessary to change from one category to another. Parameter β is calculated by τ/λ, where λ is the standardized factor loading of a given item.

The IIC is calculated by multiplying the probability of answering a response category by the probability of not answering it, which is represented along the y-axis. The apex of the information curve is where parameter β is located (x-axis). The IIC illustrates the capability of each HESI-Br item to inform on the latent construct of academic stress and can discriminate those items that are more important to capture the information. The ICC depicts parameter α on the slopes of each response category curve, the probability of endorsing a given category (y-axis), and parameter β (x-axis). The IIC and ICC are relevant because, for example, an item may inform little at the lower end of the distribution of a given construct and might therefore work better to discriminate individuals at the upper end of the construct distribution, rather than those at the lower end (i.e., it can better discriminate people with higher rather than lower stress levels).

Measurement equivalence

Measurement equivalence testing allows us to understand whether the mean score differences of a given test/questionnaire across different groups are due to true differences in the mean levels of the latent construct. In other words, it provides information on whether score differences are solely given by changes in the latent construct and not by exogenous sources of variation.

ME testing was carried out for groups selected by sex, age (18 to 25 years; 26 to 39 years; 40+ years), study level (bachelor; residency or specialization; masters, PhD or post-doc), family income (BRL 0 to 2,004; BRL 2,005 to 8,640; BRL 8,641+), and area of study (exact sciences or technology; health sciences; social sciences, education or arts). These tests were conducted using the whole dataset. Missing data were handled with pairwise deletion, since for this part of the analysis some sociodemographic variables had missing values (n missing for gender = 4; for study level = 3; for area of study = 12; for age = 3).

ME was tested by using multigroup CFA (MG-CFA) using the Wu and Estabrook approach.3636. Wu H, Estabrook R. Identification of confirmatory factor analysis models of different levels of invariance for ordered categorical outcomes. Psychometrika. 2016;81:1014-45. It consists of applying a sequence of constraints and comparing global model fit indices between each constrained model. The first step is to establish configural equivalence by constraining the model factor structure to be the same across groups. The second step is to establish threshold equivalence by further constraining item thresholds to be the same across groups. The third step is to establish metric equivalence by further constraining item factor loadings to be the same across groups (i.e., an increase of one unit on the scale has the same meaning across the compared groups). The fourth step is to constrain latent intercepts to be equal to establish scalar equivalence (i.e., respondents from different groups with the same value on the latent factor would have the same score on the observed indicators). Achieving scalar equivalence means that the questionnaire’s scores are comparable between groups. Thus, we tested whether the HESI-Br models in each group are structurally similar (configural equivalence), whether items are informing symptoms at equivalent level (threshold equivalence), whether they are equally correlated with the latent factors (metric equivalence), and whether latent means are equivalent (scalar equivalence). Values of ΔCFI < 0.01 and ΔRMSEA < 0.015 or ΔSRMR < 0.010 between nested models with increasing levels of constraints indicate equivalence.3737. Chen FF. Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct Equ Model. 2007;14:464-504.

38. Meredith W. Measurement invariance, factor analysis and factorial invariance. Psychometrika. 1993;58:525-43.
-3939. Svetina D, Rutkowski L, Rutkowski D. Multiple-group invariance with categorical outcomes using updated guidelines: an illustration using Mplus and the lavaan/semTools Packages. Struct Equ Model. 2020;27:111-30.

Convergent/discriminant validity

CFA models including the HESI-Br and DASS (second order model) were used to test whether the two scales assess the same underlying latent construct (convergent validity) or if they inform on two correlated, but separate constructs (discriminant validity). We fitted a second-order model, where the HESI-Br (five factors) and DASS-21 (three factors) loaded on a higher-order factor (i.e., testing convergence by modeling the correlation between HESI-Br and DASS-21 as originating from the same source/latent factor) and a two-correlated factor model in which DASS-21 was modeled as a second-order model (“internalizing symptoms”) and HESI-Br factors were allowed to correlate with the DASS-21 higher-order factor (i.e., testing discriminant validity by modeling HESI-Br and DASS-21 as independent constructs, while allowing them to correlate). Fit indices (RMSEA, CFI, TLI, and SRMR) were compared between models. A χ2 test was performed to test the difference between models.

Measurement equivalence was analyzed using the measEq.syntax function in the “lavaan” package in R.4040. Rosseel Y. lavaan: An R Package for structural equation modeling. J Stat Softw. 2011;48. CFA and convergent/discriminant validity analyses were carried out using the lavaan package in R.4040. Rosseel Y. lavaan: An R Package for structural equation modeling. J Stat Softw. 2011;48. IRT analysis was carried out using the “mirt” package in R.4141. Chalmers RP. Mirt: A multidimensional item response theory package for the R environment. J Stat Softw. 2012;48. R version 4.1.0 was used for all analyses (The R Foundation for Statistical Computing 2021).

Results

Exploratory factor analysis (EFA)

The Bartlett test p-value was 0, indicating that correlations between items were sufficiently large. The KMO value was = 0.85, indicating that the sample size was good. All KMO values for individual items were also acceptable (> 0.72). Parallel analysis suggested eight factors. Table 2 shows the factor loadings after rotation. Five factors had at least three items with factor loadings > 0.3 without any cross-loadings. Thus, we retained 15 items and five factors for the CFA. After examining the item content of each factor and inspired by the original instrument, we named factor 1 as career dissatisfaction, factor 2 as faculty shortcomings, factor 3 as excessive workload, factor 4 as financial concerns, and factor 6 as a toxic learning environment.

Table 2
Higher Education Stress Inventory (HESI-Br) eight-factor exploratory factor analysis (EFA) results (n = 511)

Confirmatory factor analysis (CFA)

The analysis confirmed the HESI-Br structure with five factors and 15 items (RMSEA = 0.056, 90% confidence interval [90%CI] 0.047-0.066; CFI = 0.97; TLI = 0.967; SRMR = 0.064). The second-order “Educational Stress” model presented worse fit indices (RMSEA = 0.069, 90%CI 0.060-0.077; CFI = 0.960; TLI = 0.950; SRMR = 0.077) in comparison (p-value < 0.001) with the five-correlated factor model, suggesting multidimensionality. All five factors presented acceptable internal reliabilities, with Cronbach’s α ≥ 0.72 and McDonald’s ω ≥ 0.64. Table 3 contains the CFA results. The analysis of covariance suggests low to moderate correlation between factors (coefficients range from 0.13 to 0.66).

Table 3
HESI-Br five-factor model CFA results, item difficulty, and item discrimination parameters (n = 510)

Item response theory (IRT) analysis

IIC demonstrates that the “career dissatisfaction” factor predominantly captures information on those subjects at the higher end of the stress spectrum (Figure 1). Similarly, most items in the “faculty shortcomings,” “excessive workload,” “financial concerns,” and “toxic learning environment” factors capture information on subjects across the whole spectrum (i.e., -2 to 2 standard deviations [SD] of the latent construct), which indicate that they might be good for screening educational-related stress. Table 3 contains item difficulty and discrimination results for all HESI-Br items. As an example, a person with +0.602 SD of the “toxic learning environment” academic stress construct, has a 50% probability of answering “somewhat disagree” to “somewhat agree” to the “Cold and impersonal attitudes enhanced by education” item. These properties are illustrated in the ICC curves, which reveal that most HESI-Br item response categories are informative for increasing levels of stress (Figure S4, available as online-only supplementary material).

Figure 1
Item information curves for the Brazilian Portuguese version of the HESI (HESI-Br). I(θ) = item information in which the apex of the curve corresponds to the difficulty parameter (β); (θ) = standardized latent construct.

Measurement equivalence

Measurement equivalence analysis resulted in ΔCFI < 0.01 and ΔRMSEA < 0.015 or ΔSRMR < 0.010 between nested models with increasing levels of constraints. Results are provided in Table 4. It suggests that HESI-Br is equivalent across groups selected by sex, age, study level, family income, and area of study and, therefore, mean education-related stress levels can be compared between these groups.

Table 4
HESI-Br measurement equivalence testing

Convergent/discriminant validity analysis against Depression, Anxiety and Stress Scale (DASS-21)

The HESI-Br showed discriminant validity in relation to the DASS-21. The second-order factor model presented significantly poorer fit indices (RMSEA = 0.069, 90%CI 0.067-0.071; CFI = 0.983; TLI = 0.982; SRMR = 0.074) in comparison with the model in which DASS-21 and HESI-Br were modeled as two different, but correlated constructs (RMSEA = 0.040, 90%CI 0.037-0.042; CFI = 0.994; TLI = 0.994; SRMR = 0.047). Covariances between HESI-Br factors and the general DASS-21 “distress” factor were low to moderate (coefficients range: 0.19-0.42) and are shown in Table S2, available as online-only supplementary material.

Discussion

The present study aimed to translate and adapt the HESI to Brazilian Portuguese and evaluate its structure, internal reliability, convergent/discriminant validity, and measurement equivalence. The results indicate that a model with five-correlated factors (career dissatisfaction, toxic environment, faculty shortcomings, excessive workload, and financial concerns) and 15 items is the most suitable structure for the HESI-Br. The five factors presented low to acceptable reliability indices. The highest indices were revealed for “career dissatisfaction,” “faculty shortcomings,” and “excessive workload,” meaning that for those factors sum scores tend to point to the same cohesive construct.

Compared to the seven-factor model with 24 items from the original HESI,77. Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604. psychometric analysis of the HESI-Br identified a structure with fewer factors and items. However, the identified factors were similar to five of the seven factors described in the original scale (namely, financial concerns, workload, faculty shortcomings, low commitment, and non-supportive climate). The HESI has already been adapted for the Korean and Arabic languages, being validated in medical and nursing students, respectively. Regarding psychometric properties, the K-HESI (Korean version) found a 22-item seven-factor model.88. Shim EJ, Jeon HJ, Kim H, Lee KM, Jung D, Noh HL, et al. Measuring stress in medical education: validation of the Korean version of the Higher Education Stress Inventory with medical students. BMC Med Educ. 2016;16:1-8. whereas the Arabic-HESI study resulted in a 16-item two-factor model.1515. Masha’al D, Hayajneh AA, Tawalbeh LI. Psychometric evaluation of the Arabic version of the higher education inventory scale for nursing students. BMC Psychiatry. 2021;21:1-13. It is important to highlight that these differences might have emerged due to methodological factors. For example, items with cross-loadings were eliminated for the Korean HESI and HESI-Br, but not for the Arabic HESI. Beyond this hypothesis, the instrument may have different structures between these countries because the HESI may be non-equivalent, given that educational systems and cultural aspects could be significantly different. Thus, future studies should determine the cross-cultural invariance of the HESI.

The IRT analysis showed that, overall, the instrument captures information about stress on students in the mean levels of the latent academic stressors. However, some items are better for discriminating those with high levels of stress and do not detect those with mild levels of academic stress, such as items 10, 17, 6, 2, and 28. This indicates that while the “Career dissatisfaction” construct (composed by items 10, 17, and 6) is suitable for detection of academic stress in those with high levels of academic-related stress, the other four constructs are better for use for screening purposes.

Furthermore, the HESI-Br may be useful for comparing mean levels of stress among students with different characteristics. Hence, to our knowledge, this is the first study to ascertain measurement equivalence for multiple sample characteristics for the HESI. The HESI-Br showed measurement equivalence on all tested levels (sex, age, education level, area of study, and family income) and, therefore, comparisons among these groups are likely to measure true mean differences in psychological stress. It should be noted, however, that as other versions of the HESI have found different structures, the scale is potentially non-equivalent across countries, as mentioned above.

In the present study, DASS-21 scores had higher correlations with the “financial concerns” factor than with the other HESI-Br factors. Indeed, prevalent economic problems, lack of investment in education, and great inequalities between public and private universities may influence educational stress among students in Brazil. This effect has been captured in a previous meta-analysis, showing that lower family income was associated with higher stress in Brazilian medical students.55. Pacheco JPG, Giacomin HT, Tam WW, Ribeiro TB, Arab C, Bezerra IM, et al. Mental health problems among medical students in Brazil: A systematic review and meta-analysis. Braz J Psychiatry. 2017;39:369-78. Beyond correlation, we analyzed whether the DASS-21 and HESI-Br scales both measured the same latent construct (a general distress factor), but the model fit was inferior to the model in which DASS-21 and HESI-Br were estimated as correlated but structurally different constructs. This suggests academic stress has different characteristics from the distress symptoms that are measured by the DASS-21. This is somewhat expected, since the HESI is intended to measure issues related to stress reported in interviews with students,77. Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604. but does not capture symptoms of specific disorders.

Some of the HESI-Br factors presented higher correlation with DASS score than others. As seen in Table S2, toxic learning environment and high workload were more strongly correlated with higher general distress than career dissatisfaction and faculty shortcomings. In the original HESI, however, the “low commitment” factor, which is similar to the HESI-Br “career dissatisfaction” factor, showed the highest odds ratio for depressive symptoms measured by the Major Depression Inventory (MDI). Although depression as measured by the MDI in the original HESI study might not translate perfectly when comparing to DASS scores, depressive symptoms are components of the DASS-21,77. Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604. so it is reasonable to assume some comparability. This comparison highlights potential non-equivalence in student stress between culturally and socioeconomically different countries such as Brazil and Sweden. In Brazil, stress in higher education might be affected by low aspirations tied to uncertain prospects regarding one’s career, which can influence the way that a given student understands items regarding that construct and, therefore, endorses items in a different way. In other words, in high-income countries, lower expectations about one’s career might be more indicative of general distress, whereas in low and middle-income countries, these expectations might already be low, so a high perception of workload and the perception of a toxic learning environment might be more useful for identifying students under significant levels of general distress.

Strengths and limitations

This study has a number of strengths: (a) a large sample (1,021 students); with (b) students from diverse levels of higher education (undergraduate, graduate, postgraduate); and (c) fields of study. The HESI-Br was invariant in several aspects and can be used in various student populations.

This study has at least four important limitations. First, it used a non-probabilistic sample, selected for convenience from an internet-based survey. However, most of the participants were female, self-declared white, and had high or middle income, which partially reflects the characteristics of the university student population. Second, it is noteworthy that the last phase of validation was conducted within a broad research project, designed with the objective of monitoring the mental health of the Brazilian population during the current pandemic. During the COVID-19 pandemic in Brazil, educational stress among students tends to be even greater, due to education-related stressors, such as distance learning and uncertainties regarding the quality of academic education,4242. Araújo FJ de O, de Lima LSA, Cidade PIM, Nobre CB, Neto MLR. Impact of Sars-Cov-2 and its reverberation in global higher education and mental health. Psychiatry Res. 2020;288:112977. as well as to stressful events external to the student environment, such as financial losses and social isolation.4343. Wathelet M, Duhem S, Vaiva G, Baubet T, Habran E, Veerapa E, et al. Factors associated with mental health disorders among university students in france confined during the COVID-19 pandemic. JAMA Netw Open. 2020;3:1-13. Future studies should investigate the psychometric properties of the scale when stressors related to the COVID-19 pandemic are not present. Third, the present study does not enable examination of why the structure was different from the other versions. Fourth, the “financial concerns” factor is limited in terms of reliability, but still comprises the best structure for the scale.

Conclusion

The HESI-Br scale contains 15 items within five factors, namely career dissatisfaction, faculty shortcomings, high workload, financial concerns, and toxic learning environment. The results suggest measurement equivalence by sex, age, educational levels, fields of study, and family income, which indicates comparability of HESI-Br results between groups with different sociodemographic characteristics. Furthermore, IRT analysis suggests the instrument is a potential tool for screening Brazilian university students and can also discriminate those with moderate-to-high levels of stress. Further studies of the HESI-Br should investigate whether it is comparable in different countries and cultures, with different educational systems. Nonetheless, the HESI-Br is a valid tool for screening and assessment of relevant stressors related to higher education in Brazil.

Acknowledgements

Mauricio Scopel Hoffmann is a postdoctoral research fellow at the Universidade Federal do Rio Grande do Sul, supported by the U.S. National Institute of Mental Health (grant R01MH120482) and the Newton International Fellowship (ref: NIF\R1\181942), awarded by the Academy of Medical Sciences through the UK Government’s Newton Fund Programme.

References

  • 1
    Lai AY, Lee L, Wang MP, Feng Y, Lai TT, Ho L, et al. Mental health impacts of the COVID-19 pandemic on international university students, related stressors, and coping strategies. Front Psychiatry. 2020;11:584240.
  • 2
    Moutinho ILD, Lucchetti ALG, Ezequiel OS, Lucchetti G. Mental health and quality of life of Brazilian medical students: Incidence, prevalence, and associated factors within two years of follow-up. Psychiatry Res. 2019;274:306-12.
  • 3
    Dyrbye LN, Thomas MR, Eacker A, Harper W, Massie FS, Power DV, et al. Race, ethnicity, and medical student well-being in the United States. Arch Intern Med. 2007;167:2103-9.
  • 4
    Chaabane S, Chaabna K, Bhagat S, Abraham A, Doraiswamy S, Mamtani R, et al. Perceived stress, stressors, and coping strategies among nursing students in the Middle East and North Africa: an overview of systematic reviews. Syst Rev. 2021;10:136.
  • 5
    Pacheco JPG, Giacomin HT, Tam WW, Ribeiro TB, Arab C, Bezerra IM, et al. Mental health problems among medical students in Brazil: A systematic review and meta-analysis. Braz J Psychiatry. 2017;39:369-78.
  • 6
    Crane MF, Searle BJ, Kangas M, Nwiran Y. How resilience is strengthened by exposure to stressors: the systematic self-reflection model of resilience strengthening. Anxiety Stress Coping. 2018;32:1-17.
  • 7
    Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594-604.
  • 8
    Shim EJ, Jeon HJ, Kim H, Lee KM, Jung D, Noh HL, et al. Measuring stress in medical education: validation of the Korean version of the Higher Education Stress Inventory with medical students. BMC Med Educ. 2016;16:1-8.
  • 9
    Ribeiro ÍS, Pereira R, Freire IV, de Oliveira BG, Casotti CA, Boery EN. Stress and quality of life among university students: a systematic literature review. Heal Prof Educ. 2018;4:70-7.
  • 10
    Kokou-Kpolou CK, Jumageldinov A, Park S, Nieuviarts N, Noorishad P-G, Cénat JM. Prevalence of depressive symptoms and associated psychosocial risk factors among French university students: the moderating and mediating effects of resilience. Psychiatr Q. 2021;92:443-57.
  • 11
    Turner J, Bartlett D, Andiappan M, Cabot L. Students’ perceived stress and perception of barriers to effective study: impact on academic performance in examinations. Br Dent J. 2015;219:453-8.
  • 12
    Halboub E, Alhajj MN, AlKhairat AM, Sahaqi A-AM, Quadri MFA. Perceived stress among undergraduate dental students in relation to gender, clinical training and academic performance. Acta Stomatol Croat. 2018;52:37-45.
  • 13
    Tempski P, Bellodi PL, Paro HBMS, Enns SC, Martins, Milton A Schraiber LB. What do medical students think about their quality of life? A qualitative study. BMC Med Educ. 2012;12:106.
  • 14
    Vitaliano PP, Russo J, Carr JE, Heerwagen JH. Medical school pressures and their relationship to anxiety. J Nerv Ment Dis. 1984;172:730-6.
  • 15
    Masha’al D, Hayajneh AA, Tawalbeh LI. Psychometric evaluation of the Arabic version of the higher education inventory scale for nursing students. BMC Psychiatry. 2021;21:1-13.
  • 16
    Kamulegeya LH, Kitonsa PJ, Okolimong E, Kaudha G, Maria S, Nakimuli-Mpungu E. Prevalence and associated factors of alcohol use patterns among university students in Uganda. Pan Afr Med J. 2020;37:339.
  • 17
    Dahlin M, Fjell J, Runeson B. Factors at medical school and work related to exhaustion among physicians in their first postgraduate year. Nord J Psychiatry. 2010;64:402-8.
  • 18
    Khan MN, Akhtar P, Ijaz S, Waqas A. Prevalence of depressive symptoms among university students in Pakistan: a systematic review and meta-analysis. Front Public Health. 2020;8:603357.
  • 19
    Akhtar P, Ma L, Waqas A, Naveed S, Li Y, Rahman A, et al. Prevalence of depression among university students in low and middle income countries (LMICs): a systematic review and meta-analysis. J Affect Disord. 2020;274:911-9.
  • 20
    Gao L, Xie Y, Jia C, Wang W. Prevalence of depression among Chinese university students: a systematic review and meta-analysis. Sci Rep. 2020;10:15897
  • 21
    Tam W, Lo K, Pacheco J. Prevalence of depressive symptoms among medical students: overview of systematic reviews. Med Educ. 2019;53:345-54.
  • 22
    Barros A da SX. Expansion of higher education in Brazil: limits and possibilities. Educ Soc. 2015;36:361-90.
  • 23
    Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP). Censo da Educação Superior 2019 - Divulgação dos resultados [Internet]. 2020. https://download.inep.gov.br/educacao_superior/censo_superior/documentos/2020/Apresentacao_Censo_da_Educacao_Superior_2019.pdf
    » https://download.inep.gov.br/educacao_superior/censo_superior/documentos/2020/Apresentacao_Censo_da_Educacao_Superior_2019.pdf
  • 24
    Bastos JL, Barros AJD, Celeste RK, Paradies Y, Faerstein E. Age, class and race discrimination: their interactions and associations with mental health among Brazilian university students. Cad Saude Publica. 2014;30:175-86.
  • 25
    Naser AY, Dahmash EZ, Al-Rousan R, Alwafi H, Alrawashdeh HM, Ghoul I, et al. Mental health status of the general population, healthcare professionals, and university students during 2019 coronavirus disease outbreak in Jordan: A cross-sectional study. Brain Behav. 2020;10:e01730.
  • 26
    Calegaro VC, Negretto BL, Weber LP, Kerber N, Zoratto G, Rodrigues L, et al. Monitoring the evolution of posttraumatic symptomatology, depression and anxiety during the COVID-19 pandemic in Brazilians (COVIDPsiq). Protoc Exch [Internet]. 2020. https://doi.org/10.21203/rs.3.pex-945/v2
    » https://doi.org/10.21203/rs.3.pex-945/v2
  • 27
    Lovibond PF, Lovibond SH. The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther. 1995;33:335-43.
  • 28
    Vignola RCB, Tucci AM. Adaptation and validation of the Depression, Anxiety and Stress Scale (DASS) to Brazilian Portuguese. J Affect Disord. 2014;155:104-9.
  • 29
    Zanon C, Brenner RE, Baptista MN, Vogel DL, Rubin M, Al-Darmaki FR, et al. Examining the dimensionality, reliability, and invariance of the Depression, Anxiety, and Stress Scale–21 (DASS-21) across eight countries. Assessment. 2020;28:1531-44.
  • 30
    Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A, et al. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR Task Force for Translation and Cultural Adaptation. Value Heal. 2005;8:94-104.
  • 31
    Kaiser HF. An index of factorial simplicity. Psychometrika. 1974;39:31-6.
  • 32
    Bartlett MS. The effect of standardization on a Chi-square approximation in factor analysis. Biometrika. 1951;38:337-44.
  • 33
    Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model. 1999;6:1-55.
  • 34
    Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297-334.
  • 35
    McDonald RP. Test theory: a unified treatment. Mahwah: Lawrence Erlbaum; 1999.
  • 36
    Wu H, Estabrook R. Identification of confirmatory factor analysis models of different levels of invariance for ordered categorical outcomes. Psychometrika. 2016;81:1014-45.
  • 37
    Chen FF. Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct Equ Model. 2007;14:464-504.
  • 38
    Meredith W. Measurement invariance, factor analysis and factorial invariance. Psychometrika. 1993;58:525-43.
  • 39
    Svetina D, Rutkowski L, Rutkowski D. Multiple-group invariance with categorical outcomes using updated guidelines: an illustration using Mplus and the lavaan/semTools Packages. Struct Equ Model. 2020;27:111-30.
  • 40
    Rosseel Y. lavaan: An R Package for structural equation modeling. J Stat Softw. 2011;48.
  • 41
    Chalmers RP. Mirt: A multidimensional item response theory package for the R environment. J Stat Softw. 2012;48.
  • 42
    Araújo FJ de O, de Lima LSA, Cidade PIM, Nobre CB, Neto MLR. Impact of Sars-Cov-2 and its reverberation in global higher education and mental health. Psychiatry Res. 2020;288:112977.
  • 43
    Wathelet M, Duhem S, Vaiva G, Baubet T, Habran E, Veerapa E, et al. Factors associated with mental health disorders among university students in france confined during the COVID-19 pandemic. JAMA Netw Open. 2020;3:1-13.

Publication Dates

  • Publication in this collection
    20 Oct 2023
  • Date of issue
    2023

History

  • Received
    06 Dec 2021
  • Accepted
    21 Apr 2022
Associação de Psiquiatria do Rio Grande do Sul Av. Ipiranga, 5311/202, 90610-001 Porto Alegre RS/ Brasil, Tel./Fax: (55 51) 3024 4846 - Porto Alegre - RS - Brazil
E-mail: trends@aprs.org.br