Validation of the Perception of Change Scale - Family Version (EMP-F) as a treatment outcome measure in mental health services

Validação da Escala de Percepção de Mudanças pelos familiares como medida de resultado do tratamento em serviços de saúde mental

Validation of the Perception of Change Scale – Family Version (EMP-F) as a treatment outcome measure in mental health services

Validação da Escala de Percepção de Mudanças pelos familiares como medida de resultado do tratamento em serviços de saúde mental

Marina Bandeira^I,II; Cynthia Mara Felicio^III; Luciana Cesari^I

^IUniversidade Federal de São João Del Rei (UFSJ), São João Del Rei, MG, Brazil

^IIPesquisadora do Conselho Nacional de Pesquisa e Desenvolvimento Científico (CNPq), Brazil

^IIIFundação de Amparo à Pesquisa do Estado de Minas Gerais (FAPEMIG), MG, Brazil

^{Correspondence}

ABSTRACT

OBJECTIVE: This study aimed to validate the Perception of Change Scale – Family Version, which evaluates the perception of family caregivers in regard to the treatment outcomes of psychiatric patients in mental health services.

METHOD: Family caregivers (N = 300) of psychiatric patients attending mental health services completed the Perception of Change Scale – Family Version. The scale has 19 items rated in a three-point Likert scale that evaluate changes perceived in the patient's life as a result of treatment.

RESULTS: The factorial analysis revealed a four-factor structure, with the following dimensions: 1) occupation, 2) psychological factors, 3) relationships, and 4) physical health. In the internal consistency analysis, Cronbach's alpha coefficient was 0.85. The test-retest temporal stability analysis yielded a significant intraclass correlation coefficient (r = 0.96; p < 0.005). The convergent validity analysis revealed a positive significant correlation with another scale evaluating a distinct but theoretically related construct of family satisfaction with services (r = 0.41; p < 0.05).

CONCLUSION: The Perception of Change Scale – Family Version has adequate reliability and construct and convergent validity. It can be used to evaluate treatment outcome in mental health services from the perspective of family caregivers, indicating targets to improve treatment.

Descriptors: Outcome assessment (health care); Mental health; Perception; Scale; Patients/psychology

RESUMO

OBJETIVO: O objetivo desta pesquisa foi validar a Escala de Mudança Percebida pelos familiares, que avalia o resultado do tratamento recebido por pacientes psiquiátricos em serviços de saúde mental, baseado na escala canadense Questionnaire of Perceived Changes.

MÉTODOS: Participaram da pesquisa 300 familiares cuidadores de pacientes psiquiátricos, atendidos em serviços de saúde mental. A escala foi aplicada em entrevistas individuais estruturadas. A escala possui 19 itens que avaliam mudanças percebidas na vida dos pacientes em função do tratamento, com alternativas de resposta em escala Likert de três pontos.

RESULTADOS: A escala apresentou estrutura fatorial de quatro fatores, avaliando mudanças nas seguintes dimensões: 1) Ocupação, 2) Dimensão psicológica, 3) Relacionamentos e 4) Saúde física. Possui adequadas consistência interna (alfa = 0,85) e estabilidade temporal teste-reteste (r = 0,96; p < 0,05). A análise da validade convergente indicou correlação positiva significativa com a escala de construto relacionado de satisfação dos familiares com os serviços (r = 0,41; p < 0,05).

CONCLUSÃO: A Escala de Mudança Percebida apresenta qualidades psicométricas adequadas de fidedignidade, validade de construto e validade convergente. Poderá ser usada para avaliar resultados do tratamento, na percepção dos familiares, apontando mudanças positivas na vida dos pacientes e aspectos a serem melhorados no tratamento.

Descritores: Avaliação de resultados (cuidados de saúde); Percepção; Saúde mental; Escala; Pacientes/psicologia

Introduction

The information provided by family caregivers of psychiatric patients is valuable for the evaluation of the outcome of mental health services. As the key providers of daily care to the patients, family members can perceive and report treatment-related changes in several aspects of the patients' life, such as their functioning level, symptoms, interpersonal relationships, and abnormal behaviors.¹ The inclusion of the perspectives of family members in relation to treatment outcomes and the use of this information to improve treatment can also have a positive effect in their level of satisfaction, contributing to decrease burden.² The consideration of family and patient perceptions in the evaluation of treatment services has been highly recommended in the relevant literature.^3,4

In the last three decades, outcome measures have been developed to evaluate patient and family levels of satisfaction with services and their perception of treatment results as having a real impact on their lives, most often assessing the patients' perception.^5,6 According to Donabedian, to obtain service quality, treatment results have to be congruent with the perspectives of users.⁷ User reported outcome measures can provide information to evaluate whether results detected by objective clinical measures are consonant with perceived change in real life.⁸ The perception of the impact of treatment on the patient's life can, therefore, serve as a social validation indicator for the interventions used in services⁹ and family information can be of special help in this regard.

The evaluation of family perception is usually restricted, in the literature, to service-related satisfaction measures. However, these are general measures of service features and do not focus specifically on treatment results.^10,11 Conversely, scales designed to evaluate the perception of change in the patients' life can provide a direct and specific measure of treatment outcome from the family perspective. The aim of this study was to validate the Perception of Change Scale – Family Version (in Portuguese, Escala de Mudança Percebida, henceforth referred to as EMP-F) previously submitted to a pilot study and an expert committee examination.¹² This study is part of a broader research project of the Laboratório de Pesquisa em Saúde Mental (LAPSAM) of the Universidade Federal de São João del-Rei (UFSJ) aimed to validate patient and family measures of perceived changes as a result of treatment in mental health services and to investigate associated variables.¹³

Method

1. Participants

Three-hundred family caregivers of psychiatric patients attending three outpatient mental health services in the state of Minas Gerais for at least one year were enrolled. Patients had a diagnosis of schizophrenia, schizoaffective disorders and delusional disorders, or affective disorders and severe neurotic disorders, and had no alcohol or drug addiction.¹⁴ Family members with a psychiatric diagnosis or with difficulties to understand the questions asked by the interviewer were not included. The sample size was largely superior to the minimum recommended for factorial analysis, which corresponds to 5 to 10 times the number of the scale items^15,16 and a minimum of 200 subjects.¹⁶

Two sub-samples were randomly selected from the total sample in order to accomplish two validation procedures. For the convergent validity study, 50 family members completed a second scale (SATIS-BR) in the same interview, which evaluates family satisfaction with health services. For the scale temporal stability study, 40 subjects participated in a second interview for the reapplication of the EMP-F, with an interval of three weeks.¹⁷

2. Instruments description

1) Perception of Change Scale – Family Version(EMP-F): This scale was elaborated based on an existing scale that assesses the perception of change by patients (Questionnaire of Perceived Changes⁹), after an adaptation procedure. The family scale was designed and tested by an expert committee and submitted to a pilot study to ensure its adequacy and ease of understanding by the target population.¹²

The EMP-F has 19 items evaluating the family perception of changes occurring in several aspects of the patient's life as a result of treatment. For each item, the family member is asked to report if changes were perceived in relation to a specific aspect of the patient's life, after he/she started the treatment in the service. Response alternatives are distributed in a three-point Likert scale (1 = worse than before, 2 = no change, and 3 = better than before). The last item of the scale evaluates the perception of change as a whole. The EPM-F also has an initial open question asking if the treatment is helping the patient to get better, with two response alternatives (yes or no) followed by a field for the inclusion of details.

2) Family Satisfaction Scale (Escala de Satisfação dos Familiares - SATIS-BR): To evaluate the convergent validity of the EMP, a second scale was applied in the interviews, evaluating the family satisfaction with the health service. It was constructed by the World Health Organization in a project aimed to validate mental health evaluation instruments to be applied in 19 countries,¹⁸ including Brazil.¹¹ Its brief version has eight items distributed in three subscales: the first one with three items evaluating satisfaction with treatment; the second with three items evaluating satisfaction with the competence of the health care staff; and the third with two items evaluating satisfaction with privacy and confidentiality in the service. Response alternatives are distributed in a five-point Likert scale, where 1 means "very unsatisfied" and 5 stands for "very satisfied". The scale has adequate psychometric properties, with a Cronbach's alpha coefficient of 0.79 and construct validity assessed by a factorial analysis resulting in four factors that account for 77.14% of the variance.

3) Socio-demographic and clinical questionnaire: A questionnaire previously elaborated and tested in our laboratory was used to evaluate socio-demographic variables in the family sample. It also included items designed to describe the patient's socio-demographic and clinical characteristics.

3. Procedure

Data were collected during structured interviews conducted by previously trained research assistants (undergraduate students in Psychology) in order to achieve standardization in the application procedure, although the scale does not require special clinical training to be applied. The scale was applied individually to the family member either at the health service facilities or at home, depending on the participant's schedule. Interviewers read each question and wrote down the participants' responses. Volunteers signed a consent form after been given information concerning the research objectives and procedures. The research project was approved by the directors of the mental health services involved and by the university ethics committee (011/2009/UFSJ).

4. Data analysis

Construct validity was evaluated by a factorial analysis using the Principal Components Method and Varimax rotation. A minimum loading criterion of 0.40 was adopted for scale items to be maintained in each factor.¹⁹ Hypothesis testing, focused on convergent validity, was performed using Spearman correlation analysis between the EMP-F and the SATIS-BR. Perception of change and satisfaction with services have been shown to be related constructs.^8,20 Significant correlation with a previously validated scale assessing a distinct yet theoretically related concept provides an indication of convergent validity.²¹

Reliability was evaluated by analyzing the internal consistency and temporal stability. Internal consistency was assessed using Cronbach's alpha coefficient analysis, with a minimum criterion of 0.20 item-total correlations for items to be maintained in the scale.²² Temporal stability was evaluated using Intraclass Correlation Analysis (ICC) between test and retest scale scores obtained in two applications of the EMP to a sub-sample of subjects, with an interval of three weeks.¹⁷

Results

1. Sample description

Family caregivers were mostly female (72.3%; n = 217), married (53.3%; n = 160), had a mean age of 48.67 years, and a mean of 5.85 years of education. Most of them had a salary (63.3%; n = 190) with a mean value of R$559.16 per month. Reporting family members were parents (32.0%; n = 96), siblings (26.3%; n = 79), spouses (19.0%; n = 59), and children (12.7%; n = 38) of patients, and 9.3% (n = 28) had another type of relationship with them.

Patients' mean age was 41.78 years and mean education was 5.91 years. Most of them were female (56.3%; n = 169) and had a mean monthly income of R$425.45 (59.0%; n = 177). Diagnoses according to the ICD-10 (1989) criteria were distributed in the categories of schizophrenia, schizoaffective disorders and delusional disorders (57.8%; n = 170), affective disorders (31.6%; n = 93), and neurotic disorders related to stress (10.5%; n = 31). Mean duration of psychiatric illness was 14.69 years and most of the patients (54.0%; n = 162) had been hospitalized 4.68 times on average. The majority of the patients had suffered acute crises during the previous year (54.7%; n = 164) with a mean frequency of 4.18 episodes. A part of the sample also presented a physical condition (41.0%; n = 123). Most of the patients attended the service once a month for medical consultation (90.7%; n = 272) and the remaining patients attended the service facilities three times a week, participating in occupational activities. EMP-F scores had a normal distribution, as assessed with the Kolmogorov-Smirnov test (K-S = 1.02; p = 0.25). EMP-F mean scores and standard deviations are presented in Table 1. SATIS-BR scores were not normally distributed according to the Kolmogorov-Smirnov test (K-S = 0.25; p = 0.39). The mean score for the SATIS-BR was 4.41.

2. Validity and reliability

Factorial analysis resulted in a four-factor solution, with eigenvalues higher than 1.0, accounting for 51.07% of the variance (Table 1). Sample adequacy indices were: KMO = 0.87 and Bartlet = 1358.18 (p < 0.01). Loading coefficients were all above 0.52 in the four factors. The factors and the global scale were all significantly correlated, with coefficients ranging from 0.29 to 0.84. Table 2 shows significant correlation coefficients between the factors and the global scale.

Cronbach's alpha coefficient was 0.85 for the global scale, with item-total correlations ranging from 0.29 to 0.70 (Table 2). The first subscale has five items evaluating changes in the patient's occupational status and energy, with an alpha value of 0.81. The second subscale has six items evaluating changes in psychological aspects, with an alpha value of 0.71. The third subscale has three items evaluating changes in the patients' relationships, with an alpha value of 0.71. The fourth subscale has four items evaluating changes in physical health, with an alpha value of 0.52. The temporal stability analysis for the test and retest scores resulted in positive significant ICC coefficients ranging from 0.83 to 0.96 (Table 2).

The results of the convergent validity analysis using Spearman correlation yielded a significant positive correlation coefficient (0.41; p = 0.003) between the EMP-F and the SATIS-BR.

Discussion

This study investigated the psychometric properties of the EMP-F, which evaluates family-perceived changes occurring in several dimensions of the patients' life as a result of treatment. Both the reliability and the validity analyses indicated adequate psychometric qualities. The factorial analysis resulting in a four-factor solution had all items with loading coefficients above 0.52, which is superior to the minimum recommended value of 0.40.¹⁹ The convergent validity analysis showed a significant positive correlation between the EMP-F and a scale that assesses a distinct but related construct (family satisfaction with services²¹). The correlation coefficient was moderate, as expected, since the scales evaluate different constructs. Positive treatment results as evaluated by improvements in the patient's life, which are assessed by the EMP-F, is one of the dimensions of satisfaction with services, as demonstrated by research in the area.^8,20

The internal consistency alpha coefficient for the global scale (0.85) was high and remained in the ideal range (0.70-0.85) proposed by Vallerand.¹⁷ The alpha values for the subscales fell in this same range in the case of three - out of four - subscales, one of them yielding a value of 0.52, which is still above the minimum of 0.50 considered acceptable by Bowling.²³ The temporal stability was adequate, with results showing significant correlations between test and retest applications, ranging from 0.78 to 0.93, which is superior to the minimum recommended of 0.60.¹⁷

Conclusion

Considering the results of the psychometric analyses, it can be concluded that the EMP-F can be used to evaluate the family perception of changes in the patient's life as a consequence of treatment in mental health services. Family caregivers are in a special position to observe and report changes in specific dimensions of the patients' lives and to indicate those aspects which improved as a result of treatment using this scale. The EMP-F can also provide information about the aspects of the patients' life that remained unchanged or that deteriorated after treatment, indicating specific targets for treatment redesign and improvement. The EMP-F is able, therefore, to contribute in the monitoring of the quality of mental health services, as recommended by the WHO.³

Acknowledgements

This research article is part of a project funded by the Fundação de Amparo à Pesquisa do Estado de Minas Gerais (FAPEMIG) and the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq). The full version of the scale is available on the website of the LAPSAM: www.ufsj.edu.br/lapsam

Submitted: October 3, 2009

Accepted: March 1, 2010

• 1. Bandeira M, Barroso SM. Sobrecarga das famílias de pacientes psiquiátricos. J Bras Psiquiatr 2005;54(1):34-46.
• 2. Tessler RC, Gamache GM. Family experiences with Mental Illness Westport: Auburn House; 2000.
• ³
  WHO. The World Health Report. Mental health: new understanding, New Hope Geneva: World Health Organization; 2001.
• 4. Hamson L. Outcome assessment in psychiatric service evaluation. Soc Psychiatry Psychiatr Epidemiol. 2001;36:244-8.
• 5. FDA Guidance for Industry. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. U.S. Department of Health and Human Services Food and Drug Administration Center for Drug Evaluation and Research (CDER) Center for Biologics Evaluation and Research (CBER) Center for Devices and Radiological Health (CDRH). FDA; 2006.
• 6. McCabe R, Saidi M, Priebe S. Patient-reported outcomes in schizophrenia. Br J Psychiatry Suppl. 2007;191:s21-8.
• 7. Donabedian A. The Lichfield Lecture. Quality assurance in health care: consumers' role. Qual Health Care. 1992;1(4):247-251.
• 8. Hasler G, Moergeli H, Bachman R, Lambreva E, Buddebeg C, Schnyder U. Patient satisfaction with outpatient psychiatric treatment: the role of diagnosis, pharmacotherapy and perceived therapeutic change. Can J Psychiatric 2004;49(5):315-21.
• 9. Mercier L, Landry M, Corbiere M, Perreault M. Measuring clients' perception as outcome measurement. In: Roberts AR, Yeager KR. Evidence-based practice manual: research an outcome measures in health and human services Oxford: University Press; 2004.
• 10. Bandeira M, Pitta AMF, Mercier C. Escalas da OMS de avaliação da satisfação e da sobrecarga em serviços de saúde mental: qualidades psicométricas da versão brasileira. J Bras Psiquiatr 1999;48(6):233-44.
• 11. Bandeira M, Mercier C, Perrault M, Libério M, Pitta AM. Escala de avaliação da satisfação dos familiares em serviços de saúde mental. J Bras Psiquiatr 2002;51(3):153-66.
• 12. Bandeira M, Calzavara MG, Costa CS, Cesari L. Avaliação de serviços de saúde mental: adaptação transcultural de uma medida da percepção dos usuários sobre resultados do tratamento. J Bras Psiquiatr 2009;58(2):107-14.
• 13. Bandeira M. Integrando a percepção dos usuários na avaliação dos serviços de saúde mental. Projeto de pesquisa aprovado pela Fapemig, pelo Programa Pesquisador Mineiro: Processo APQ-0555=506-07; 2007.
• 14. CID-10: Classificação estatística internacional de doenças e problemas relacionados à saúde. 10 ed. São Paulo: USP; 1998.
• 15. Artes R. Aspectos estatísticos da análise fatorial de escalas de avaliação. Rev Psiquiatr Clin 1998;25(5):223-8.
• 16. Pasquali L. Testes referentes a construto: teoria e modelo de construção. In: Pasquali L. Instrumentos psicológicos: manual prático de elaboração. Brasília: LABPAM/IBAPP; 1999. Cap.3, p.56.
• 17. Vallerand RJ. Vers une méthodologie de validation trans-culturelle de questionnaires psychologiques: implications pour la recherché en langue française. Canadian Psychol 1969;30(4):662-80.
• 18. WHO-SATIS. Consumer's and Caregivers' satisfaction with mental health services: a multisite study. Geneva: Division of Mental Health. WHO; 1996.
• 19. Dancey CP, Reidy J. Introdução à análise de fatores. In: Dancey CP, Reidy J. Estatística sem matemática para psicologia. 3a. ed. Porto Alegre: Artmed; 2006.
• 20. Hasler G, Moergeli H, Schnyder U. Outcomes of psychiatric treatment: what is relevant for our patients? Comprehen Psychiatry. 2004;45(3):199-205.
• 21. Pasquali L. Validade dos testes. In: Psicometria: Teoria dos testes na psicologia e na educação Petrópolis, RJ: Editora Vozes; 2003.
• 22. Perreault M, Katerelos TE, Sabourin S, Leichner P, Desmarais J. Information as a distinct dimension for satisfaction assessment of outpatient psychiatric services. Int J Health Care Qual Assur Inc Leadersh Health Serv 2001;14(3):111-20.
• 23. Bowling A. Measuring disease: a review of disease specific quality of life measurement scales. Philadelphia: Open University Press; 1997.

Publication Dates

• Publication in this collection
  11 June 2010
• Date of issue
  Sept 2010

History

• Accepted
  01 Mar 2010
• Received
  03 Oct 2009