Work Environment Evaluation Instrument (WEEI): development, validation, and association with burnout

Abstract

Introduction

Physician burnout is considered an epidemic. In 2019, 44% of U.S. physicians reported feeling burned out. The work environment is a central risk factor for this. The aim of this study is to develop and test an instrument to evaluate work environment factors in medical training courses.

Method

After focus groups, an initial pool of 14 items was generated and tested in a pilot study (n = 66). Face validity was verified, and small adjustments were made. The resulting version was administered to a sample of 115 psychiatry residents. Eleven items were selected based on the correlations between them, principal component analysis, and theoretical reasons, and then tested for internal and construct validity.

Results

The final version had high reliability (Cronbach’s alpha = 0.898) and comprised three dimensions: relations with the institution; with colleagues; and with preceptors. Both total scores and dimensions correlated significantly with burnout scores (p < 0.01). Cutoffs defining the environment as healthy (>32 points); risky (23-31 points); or toxic (<22 points) were suggested and related to the risk of burnout.

Conclusion

Several authors have emphasized the importance of approaching institutional factors as an effective strategy for coping with the increased prevalence of burnout. This instrument should contribute to these efforts.

Burnout; medical education; validation studies; stress disorders; environment

Introduction

Burnout rates are increasing, and many authors are describing the phenomenon as an epidemic. Medical doctors and students are a special concern. In 2014, 54.4% of U.S. physicians reported at least one symptom of burnout, compared with 45.5% in 2011. ¹, ² In 2019, 44% of physicians reported feeling burned out. ³ Rates of burnout are rising, despite greater recognition of the problem. Some authors argue that this may be because efforts to reduce burnout are typically focused on increasing resilience and wellness, rather than combating problematic changes in how medicine is practiced by physicians in the current era, or the roots of the problem that involve institutional factors and culture. ², ⁴, ⁵

With respect to medical education, many studies report rates of burnout and other mental diseases that are much higher than those in the general population. ⁶, ⁷ The prevalence of burnout in residents is generally about 25-75%, varying by specialty, country, and measurement method. ⁸, ⁹ These findings point to the fact that the training process and environment may impact on the health of physicians in training. ¹⁰

Some risk factors are known, such as heavy workload, female gender, presence of physical illness and or mental disorders, medication use, dissatisfaction with career, high demand for perfection, and others. Nevertheless, institutional factors are as yet poorly investigated and understood. ⁴, ¹¹ Medical students and residents spend a great proportion of their time immersed in the institutional environment and are in constant contact with professors, assistants, colleagues, patients, and the institutional staff and culture. The nature of this contact and also the beliefs and values common to the members of the institution may play a fundamental role in development of burnout, especially if they are not compatible with one’s own beliefs. Moreover, problematic interpersonal relations can be highly harmful, for instance, conflict with colleagues and preceptors, competition, abuse of power by superiors, lack of empathy, and lack of support. Furthermore, contemporary physicians are faced with the challenge of delivering increasingly patient-centered care, at the same time that they are continually exposed to ongoing economic, technological, and regulatory changes, posing unrealistic expectations of physician availability. ⁴

The consequences of burnout can be devastating, for both personal and professional life, leading to increased reports of medical errors, suboptimal patient care, reduced empathy in general, reduced quality of life, and development of mental disorders like depression. ⁸, ¹², ¹³ It is both crucial and urgent to elucidate the role played by the work environment in this phenomenon. There is an extreme need for reliable instruments to evaluate the different elements that may be involved. The aim of this study was therefore to develop and test an instrument to evaluate the work environment that encompasses the subject’s relations with the institution, with professors/preceptors, and with peers.

Method

This is an instrument validation study. The objective was to create an instrument to evaluate work environment factors that can be related to development of burnout during medical training. The study comprised two phases – the first consisted of compiling the initial item pool and testing it in a pilot study and the second consisted of final selection of items and validation. The study was approved by the ethics committee at the Hospital de Clínicas de Porto Alegre (CAAE 70231617.6.0000.5327).

Development of the instrument and pilot study

Several focus groups were held with experts, professors, medical students, and residents during the first semester of 2017 to identify and define items that could be related to development of burnout symptoms. Fourteen items were defined and tested in a pilot study that was conducted in July 2017. For this step, all psychiatry residents in the city of Porto Alegre (n = 87) were invited by e-mail to answer the questionnaire and 75% (n = 66) of them participated in the study. ¹⁴ Item comprehensibility and face validity were evaluated and discussed, and small adjustments were made. Additionally, three items were reformulated as negative sentences to detect unreliable answers.

Selection of the final items and validation

Participants

The sample comprised 115 adult Brazilian participants (56 males) aged from 24 to 42 years (mean = 29.34, standard deviation [SD] = 3.50) who were recruited online via e-mail. All psychiatry residents on the 10 different residency programs in the state of Rio Grande do Sul were invited (n = 186), 131 of whom used the instruments. The questionnaire was made available on an online platform widely used for research purposes, since it guarantees the anonymity of the subjects involved (Survey Monkey™). Subjects were assessed with the Work Environment Evaluation Instrument – preliminary version (WEEI-P) and the Maslach Burnout Inventory (MBI). Subjects whose questionnaires were incomplete or missing were excluded. All individuals consented to voluntary participation when they agreed to answer the questionnaire after receiving a full explanation of the study.

Materials

Level of burnout was measured using the Portuguese version of the Maslach Burnout Inventory – Human Services Survey (MBI-HSS). ¹⁵ The MBI-HSS measures burnout on three subscales: emotional exhaustion (EE), depersonalization (DP), and low sense of personal accomplishment (PA). It is a self-administered questionnaire answered on a five-point Likert scale on which 0 = never; 1 = annually; 2 = once a month; 3 = a few times a month; 4 = once a week; 5 = a few times a week; and 6 = every day. The scale has 22 items, 9 related to emotional exhaustion, 5 to depersonalization, and 8 to sense of personal accomplishment. We followed the MBI-HSS recommendations on defining whether a subject meets the criteria for one of the three dimensions.

The preliminary version of the WEEI was a 14 item self-report questionnaire with a Likert response scale on which 0 corresponded to “totally false” and 4 corresponded to “totally true.” Six items evaluated relations with teachers/preceptors, 5 with colleagues/peers, and 3 with the institution (^{online-only supplementary material S1}).

Data analysis

The Statistical Package for the Social Sciences (SPSS) version 23.0 was used for analysis. The Shapiro-Wilk test was used to test normality. Kaiser-Meyer-Olkin (KMO) measure and Bartlett’s test of sphericity were used to evaluate if the data was suitable for factorial analysis. Correlations between the items were calculated to evaluate their adequacy/fitness, both in relation to the whole instrument as a single construct and within each of the three dimensions. Principal component analysis (PCA) was conducted to explore the scale’s underlying structure. These data were used together with a reappraisal of item content to select items that would be retained in the final version (WEEI).

The final version of the WEEI was tested for internal validity using Cronbach’s alpha coefficient (α) and its construct validity was verified by testing correlations between WEEI scores and burnout dimensions. T-tests for independent samples were used to compare means. Cutoff points were suggested for the WEEI, and risk estimation tests were performed. Additionally, the Cronbach’s alpha coefficient was also estimated for each of the dimensions to verify the reliability of using the scores of the three dimensions in separate.

Results

Selection of the final items

The appropriateness of items and the possibility of redundant questions were assessed using the bivariate correlations between them. The PCA revealed three factors (^{online-only supplementary material S2}) and all items loaded above 0.4 in the first factor, pointing to a consistent common construct. Factor loadings, correlations with other items, and the content of the sentences were further analyzed in conjunction to select the items for the final version.

The items “I am ashamed to show my preceptor that I do not know something,” “My colleagues can do me harm at any time,” and “I feel that I have friends in the college/residency on whom I can count, even for matters that have nothing directly to do with the medical school/residency” were excluded from the final version due to their weaker psychometric performance and for theoretical reasons. The first item could reflect phobic traits, the second, paranoid traits, and the last, matters that do not directly relate to the work environment.

Finally, we ran the factor analysis for the new version that comprised 11 items. The KMO measure of sampling adequacy indicated that the strength of the relations among variables was high (KMO = 0.88) and Bartlett’s test of sphericity, which tests the overall significance of all the correlations within the correlation matrix, was significant (p<0.001). Although the PCA yielded 2 factors with eigenvalues > 1, explaining 63% of the variance, due to the fact that all items loaded > 0.54 in the first factor we decided for a one-factor solution. The WEEI measures three dimensions as part of the work environment: relation to the institution (RI - items 2, 4 and 6), relationships with colleagues/peers (RC - items 5, 8 and 10), and relationships with preceptors/superiors (RP - items 1, 3, 7, 9 and 11), generating a total score (TS) as well as dimension scores.

Evaluation of internal consistency and construct validity

According to Burns, at least 10 subjects for each item on a scale are required for proper validation. ¹⁶ There were 115 subjects in our sample. Cronbach’s alpha coefficient (α) for the WEEI was 0.898, showing high internal consistency without reaching excessively high values (i.e. > 0.95) that could indicate redundant items (^{online-only supplementary material S3}). The Cronbach’s Alpha coefficient (α) for each of the dimensions was 0.79 for “institution”, 0.78 for “colleagues”, and 0.87 for “preceptors”.

Based on the assumption that, in accordance with our conceptualization, work environment factors would be related to burnout, we evaluated associations between WEEI scores and burnout scores to analyze construct validity. Both total WEEI score and its dimension scores (RI, RC, and RP) were correlated with Burnout scores (EE, DP, and PA) (p < 0.01) in our sample ( Table 1 ).

Table 1
Correlations between WEEI scores and burnout scores

	EE	DE	PA	Total WEEI	WEEI Peers	WEEI Preceptors	WEEI Institution
EE total score
Spearman’s rho correlation	1	0.510*	-0.255*	-0.562*	-0.268*	-0.535*	-0.573*
Sig. (2-tailed)		<0.001	0.006	<0.001	0.004	<0.001	<0.001
DP total score
Spearman’s rho correlation	0.510*	1	-0.393*	-0.486*	-0.278*	-0.502*	-0.451*
Sig. (2-tailed)	<0.001		<0.001	<0.001	0.003	<0.001	<0.001
PA total score
Spearman’s rho correlation	-0.255*	-0.393*	1	0.320*	0.194 †	0.310*	0.274*
Sig. (2-tailed)	0.006	<0.001		<0.001	0.038	0.001	0.003
Total WEEI
Spearman’s rho correlation	-0.562*	-0.486*	0.320*	1	0.685*	0.916*	0.895*
Sig. (2-tailed)	<0.001	<0.001	<0.001		<0.001	<0.001	<0.001
WEEI Peers
Spearman’s rho correlation	-0.268*	-0.278*	0.194*	0.685*	1	0.439*	0.514*
Sig. (2-tailed)	0.004	0.003	0.038	<0.001		<0.001	<0.001
WEEI Preceptors
Spearman’s rho correlation	-0.535*	-0.502*	0.310*	0.916*	0.439*	1	0.760*
Sig. (2-tailed)	<0.001	<0.001	0.001	<0.001	<0.001		<0.001
WEEI Institution
Spearman’s rho correlation	-0.573*	-0.451*	0.274*	0.895*	0.514*	0.760*	1
Sig. (2-tailed)	<0.001	<0.001	0.003	<0.001	<0.001	<0.001

• EE = Emotional exhaustion; DP = depersonalization; PA = personal accomplishment; WEEI = Work Environment Evaluation Instrument.
• * Correlation is significant at the 0.01 level (2-tailed).
• ^† Correlation is significant at the 0.05 level (2-tailed).

There was a difference in mean WEEI TS between subjects who were positive for EE according to the MBI-HSS and those who were not (22.69±10 versus 31.29±8.4; p < 0.001). Considering that EE is the most consistent dimension of burnout, to the extent that some authors even use it as the only diagnostic criterion, we defined proposed WEEI cutoff points based on mean and SD WEEI scores for these groups, (positive and negative for EE), and performed risk evaluations tests.

The cutoff points tested classified the environment as healthy (> 32 points), risky (between 23 and 31 points), or toxic (< 22 points). In a toxic environment, the odds ratio (OR) for being positive for EE was 6.2 (95%CI: 2.6-15; p < 0.001) and OR for DP was 14.4 (95%CI: 4.2-48.6; p < 0.001). In a healthy environment, the OR for being positive for EE was 0.25 (95%CI 0.11-0.6; p = 0.001) and OR for DP was 0.74 (95%CI 0.64-0.85; p < 0.001). As expected, the results regarding risky environments were not significant, suggesting environments that are nor protective nor consistently associated to burnout risk.

We suggest that the scores should be used as a continuous variable. The cutoff points for toxic and/or healthy environment can be used when appropriate to the objective of the study (^{online-only supplementary material S4}). Nevertheless, the robust association between characterization of the environment as healthy or toxic with EE and DP burnout dimensions is also indicative of the validity of the construct measured by the instrument.

Discussion

We were able to develop a rapid, self-administered instrument that adequately evaluates three work environment dimensions (institution, preceptors, and peers) with high reliability (Cronbach’s alpha = 0.898). This version of the instrument was developed specifically for the medical training community. To our knowledge there is no other instrument like this reported in the literature to date.

Burnout has a huge impact on personal life, work capacity, and the economy and efforts that have been implemented to cope with the problem have not been effective in mitigating the phenomenon. It is also worth noting that other psychiatric conditions for which burnout can be an important risk factor are also more prevalent among doctors, specifically, depression and suicide. As pointed out by several authors, institutional factors must be addressed and the instrument developed in this study can contribute to further research investigating these factors. ², ⁴, ⁵

Limitations of this study include the fact that it was conducted in a specific population (psychiatry residents from Rio Grande do Sul, Brazil). Therefore, studies with different populations and in different settings are needed to confirm our findings and the instrument’s performance. Nevertheless, a more homogenous sample favors internal validity, while the 115 residents in our sample came from 10 different residency programs, representing a range of different institutions. We believe this is an important contribution to the literature that can foster an important and urgent research field.

References

• ¹
  Shanafelt TD, West CP, Sinsky C, Trockel M, Tutty M, Satele DV, et al. Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011 and 2017. Mayo Clin Proc. 2019;94:1681-94.
• ²
  Squiers JJ, Lobdell KW, Fann JI, DiMaio JM. Physician burnout: are we treating the symptoms instead of the disease? Ann Thorac Surg. 2017;104:1117-22.
• ³
  Kane L. Medscape national physician burnout, depression & suicide report 2019 [Internet]. 2019 [cited] https://www.medscape.com/slideshow/2019-lifestyle-burnout-depression-6011056
  » https://www.medscape.com/slideshow/2019-lifestyle-burnout-depression-6011056
• ⁴
  Hauck S, Gabbard GO. Institutional factors in the medical burnout epidemic. Braz J Psychiatry. 2019;41:191-2.
• ⁵
  Slavin SJ. Medical student mental health culture, environment, and the need for change. JAMA. 2016;316:2195-6.
• ⁶
  Dyrbye LN, Thomas MR, Massie FS, Power DV, Eacker A, Harper W, et al. Burnout and suicidal ideation among U.S. medical students. Ann Intern Med. 2008;149:331-41.
• ⁷
  Pacheco JP, Giacomin HT, Tam WW, Ribeiro TB, Arab C, Bezerra IM, et al. Mental health problems among medical students in Brazil: A systematic review and meta-analysis. Braz J Psychiatry. 2017.39:369-78.
• ⁸
  Shanafelt TD, Bradley KA, Wipf JE, Back AL. Burnout and self-reported patient care in an internal medicine residency program. Ann Intern Med. 2002;136:358-67.
• ⁹
  West CP, Shanafelt TD, Kolars JC. Quality of life, burnout, educational debt, and medical knowledge among internal medicine residents. JAMA. 2011;306:952-60.
• ¹⁰
  Brazeau CM, Shanafelt T, Durning SJ, Massie FS, Eacker A, Moutier C, et al. Distress among matriculating medical students relative to the general population. Acad Med. 2014;89:1520-5.
• ¹¹
  Andolsek KM. Physician well-being: organizational strategies for physician burnout. FP Essent. 2018;47:20-4.
• ¹²
  Fahrenkopf AM, Sectish TC, Barger LK, Sharek PJ, Lewin D, Chiang VW, et al. Rates of medication errors among depressed and burnt out residents: prospective cohort study. BMJ. 2008;336:488-91.
• ¹³
  West CP, Huschka MM, Novotny PJ, Sloan JA, Kolars JC, Habermann TM, et al. Association of perceived medical errors with resident distress and empathy: a prospective longitudinal study. JAMA. 2006;296:1071-8.
• ¹⁴
  Monteiro GMC, Cavalcanti IP, Hauck S, Baeza FLC. Burnout in psychiatry residents: the role of psychiatric symptoms, interpersonal and institutional relations. 12th Geneva Conference on Person Centered Medicine Promoting Wellbeing and Overcoming Burnout; 2019 March 25-7; Geneva, Switzerland.
• ¹⁵
  Maslach C, Leiter MP, Jackson SE. The Maslach Burnout Inventory: manual. 4th ed. Menlo Park: Mind Garden; 2016.
• ¹⁶
  Burns N, Grove SK. The practice of nursing research conduct, critique and utilization. 4th ed. Philadelphia: WB Saunders; 2001.

https://minio.scielo.br/documentstore/2238-0019/GHzr5zXXctCfYXHsP9svf7R/370e9ff6fd4f4b6cfea7e5489d6710f77b4c9801.pdf

Publication Dates

• Publication in this collection
  20 Mar 2020
• Date of issue
  Apr-Jun 2020

History

• Received
  14 June 2019
• Accepted
  06 July 2019