Inter-rater reliability and factor analysis of the Brazilian version of the Schedule for the Assessment of Insight: Expanded Version (SAI-E)

Estudo de confiabilidade entre avaliadores e análise fatorial da versão brasileira do Schedule for the Assessment of Insight: Expanded Version (SAI-E)

BRIEF REPORT

Inter-rater reliability and factor analysis of the Brazilian version of the Schedule for the Assessment of Insight - Expanded Version (SAI-E)

Estudo de confiabilidade entre avaliadores e análise fatorial da versão brasileira do Schedule for the Assessment of Insight - Expanded Version (SAI-E)

Clarissa de Rosalmeida Dantas; Cláudio Eduardo Muller Banzato

Department of Psychiatry, Medical School, Universidade Estadual de Campinas (Unicamp), Campinas (SP), Brazil

^{Correspondence}

ABSTRACT

OBJECTIVES: The Schedule for the Assessment of Insight - Expanded Version consists of 11 items that encompass: awareness of having a mental illness, ability to rename psychotic phenomena as abnormal, and compliance with treatment. The objective of the study was to evaluate the inter-rater reliability and to study the factorial structure of the Brazilian version of the instrument.

METHOD: The Brazilian version of the Schedule for the Assessment of Insight - Expanded Version was used for the assessment of insight of 109 psychotic inpatients, 60 of whom had the interview tape-recorded in order to be scored by an independent evaluator. Intraclass correlation coefficient (ICC) was adopted as the inter-rater reliability coefficient. In the factor analysis, principal components analysis and Varimax rotation were adopted.

RESULTS: Inter-rater reliability coefficients from good to excellent were found for the individual items of the Schedule for the Assessment of Insight - Expanded Version with ICC values ranging from 0.54 to 0.82. Regarding the total score, inter-rater reliability was excellent, with ICC = 0.90. A factorial structure similar to the one obtained by the original version of the Schedule for the Assessment of Insight - Expanded Version was found, with 3 factors accounting for 71.72% of variance.

CONCLUSION: In the Brazilian context, the Schedule for the Assessment of Insight - Expanded Version presented good inter-rater reliability and factorial structure compatible to the insight dimensions that are intended to be evaluated.

Descriptors: Psychotic disorders; Awareness; Mental status schedule; Reproducibility of results; Psychometrics

RESUMO

OBJETIVOS:O Schedule for the Assessment of Insight - Expanded Version é constituído por 11 itens que abordam: reconhecimento de se ter um transtorno mental, capacidade de renomear fenômenos psicóticos como anormais e adesão ao tratamento. O objetivo do estudo foi avaliar a confiabilidade entre avaliadores e estudar a estrutura fatorial da versão brasileira do Schedule for the Assessment of Insight - Expanded Version.

MÉTODO: A versão brasileira do Schedule for the Assessment of Insight - Expanded Version foi utilizada na avaliação de 109 pacientes psicóticos internados, dos quais 60 tiveram a entrevista gravada para atribuição de escores por avaliador independente. O coeficiente de correlação intraclasse (ICC) foi utilizado na avaliação da confiabilidade entre avaliadores. Para a análise fatorial foram adotadas análise de componentes principais e rotação varimax.

RESULTADOS: A confiabilidade entre avaliadores para os itens do Schedule for the Assessment of Insight - Expanded Version encontrada esteve entre boa e excelente, com ICC variando de 0,54 a 0,82; para o escore total foi excelente, com ICC = 0,90. Uma estrutura fatorial semelhante à obtida para a versão original do Schedule for the Assessment of Insight - Expanded Version foi encontrada, com três fatores explicando 71,72% da variação.

CONCLUSÃO: No contexto brasileiro, o Schedule for the Assessment of Insight - Expanded Version apresentou boa confiabilidade entre avaliadores e estrutura fatorial compatível com as dimensões do insight que pretende avaliar.

Descritores: Transtornos psicóticos; Conscientização; Entrevista psiquiátrica padronizada; Reprodutibilidade de resultados; Psicometria

Introduction

Although insight impairment is an important acknowledged characteristic of psychotic illnesses, little critical attention had been given to the forms of insight assessment until the late 80's, when efforts were made aiming the development of standardized instruments for that purpose.¹

Among the available instruments, the Scale to Assess Unawareness of Mental Disorder (SUMD),¹ the Schedule for the Assessment of Insight (SAI)² and the Schedule for the Assessment of Insight - Expanded Version (SAI-E)³ have been rated as the most useful and psychometrically sound scales.⁴ Those scales use a continuous approach to insight assessment – in which insight is conceptualized as a continuous process, composed of multiple dimensions and assessed in terms of scores from semi-structured interviews. Comparative studies^5-6 have demonstrated that the scores of SAI, SAI-E, and SUMD are strongly correlated, which indicates that those scales have a good concurrent validity.

The SAI was developed for the assessment of insight in psychotic patients and it is based on a concept of insight that encompasses three distinct but overlapping dimensions:⁷ 1) recognition of mental illness; 2) ability to relabel unusual mental events (e.g. hallucinations) as pathological; and 3) treatment compliance, both expressed and observed. Afterwards, the SAI has been expanded (SAI-E)³ to include items on awareness of core symptoms, emotional/psychological changes and difficulties resulting from the mental condition. The factor analysis performed for the SAI had yielded one single factor.² On the other hand, the factor analysis performed for the SAI-E yielded 3 factors,⁸ which closely correspond to the dimensions of insight proposed by David, suggesting that the SAI-E aptly captures the theoretical construct put forward by the author.

The SAI-E is a semi-structured interview which consists of 11 items, with a standardized mode of rating of the items by the interviewer. The questions to be directed to the interviewee are pointed out, but they allow some flexibility in their formulation. A global score is obtained from the addition of the scores of each item, and it ranges from 0 to 28 (higher scores indicate better insight).

In Brazil, few studies about insight that used standardized instruments have been accomplished thus far. Fiss and Chaves translated and adapted the SUMD to Portuguese.⁹ Even though this instrument has been largely utilized in research, it has some disadvantages such as: 1) it takes longer to be applied when compared to other scales;⁵ 2) the fact that the SUMD is composed of subscales that don't generate a global score sometimes makes the interpretation and comparison of results difficult.⁵

The SAI was translated into Portuguese,¹⁰ however, its psychometric properties within our context have not been studied thus far. The SAI-E was chosen to be adapted – according to the method proposed by Flaherty et al.¹¹ – due to the fact that it is an instrument which has proven to be valid and relatively easy to apply in its original version.³ Furthermore, it deals with aspects relevant to the assessment of insight not properly covered by the SAI and its utilization is easier and faster compared to SUMD's. The complete work with a thorough description of the process of adaptation of the SAI-E and its Brazilian version are available elsewhere.¹² Additionally, a study of the correlation between insight (as measured by the SAI-E and its factor scores) and socio-demographic, clinical and treatment variables was also performed and its results are presented elsewhere.¹³

The objectives of this work were: 1) to test the inter-rater reliability reached through minimum training in the use of the Brazilian version of the SAI-E and 2) to verify the factor structure subjacent to the measures obtained with the application of the Brazilian version of the SAI-E in a sample of psychotic inpatients. The factor structure of a given scale may well change when such an instrument is adapted for use in a different culture. As the three-factor structure is an important feature of the original version of the SAI-E, it would be desirable that the factor analysis of the Brazilian version of SAI-E yield a similar factor structure.

This research project was approved by the Committee of Ethics in Research of the Medical School - UNICAMP (Project number: 288/2005. Date: 06/28/2005). A written informed consent was obtained from all the participants enrolled in the study.

Method

In this study, "Evaluator 1" was one of the authors (C.R.D.), an experienced psychiatrist who, having actively participated in the adaptation process of the SAI-E, acquired familiarity with the instrument. "Evaluator 2" was a psychiatrist who had recently concluded her medical residency training. Regarding the SAI-E training process of "Evaluator 2", only one brief meeting occurred at the beginning of the research when the criteria for score attribution were discussed. Throughout the study, the evaluations were accomplished independently, without any kind of feedback between the evaluators concerning the attributed scores. The authors chose this approach in order to establish what could be considered as the least inter-rater reliability, since reliability is not a stable characteristic of a given instrument of evaluation, but rather is dependent on the interviewers-evaluators, their training and competence in the use of the instrument.¹⁴

Among consecutively admitted inpatients those who had an admission diagnosis of schizophrenia or other psychotic disorders were selected between August 2005 and February 2006, from two different institutions 1) the Ismael Mental Institution – a psychiatric hospital localized in Amparo (SP-Brazil), which treats patients exclusively by the Sistema Único de Saúde (SUS, the Brazilian public health system); 2) the Bairral Institute – a psychiatric hospital localized in Itapira (SP-Brazil), which admits patients covered by SUS, as well as those covered by insurance, and privately-paying patients. We chose such diverse research loci in order to obtain patients from different socio-demographic backgrounds. The exclusion criteria at that point were: patients under 18 or over 65 years old; evidence of brain lesion or severe cognitive impairment; and severe communication difficulties.

Patients who agreed to participate in the research were submitted to a standardized interview for a detailed collection of socio-demographic and clinical data. The Mini International Neuropsychiatry Interview Plus (M.I.N.I. Plus) - Brazilian version¹⁵ was used to confirm the diagnosis. Patients who did not meet the Diagnostic and Statistic Manual – Revised Text (DSM-IV-TR)¹⁶ diagnostic criteria for one of the following disorders: schizophrenia, schizophreniform, schizoaffective, delusional and brief psychotic disorders and psychotic disorder not otherwise specified, were excluded from the study.

From the 121 initially selected inpatients, 11 either did not agree to participate in the research or interrupted the interview and 1 was excluded from the study for not having the psychotic disorder diagnostically confirmed.

All interviews (socio-demographic and clinical data collection, and utilization of M.I.N.I. Plus and the Brazilian version of SAI-E) were conducted by Evaluator 1 (C.R.D.) in the two aforementioned hospitals, during the selected patients' hospitalization. The first sixty patients interviewed had the part of the interview corresponding to the application of the SAI-E tape-recorded for further assessment and scoring by Evaluator 2.

The Intraclass Correlation Coefficient (ICC), with Two-way Random Effect Model and absolute agreement definition, was adopted for the analysis of inter-rater reliability according to the recommendations of Shrout and Fleiss.¹⁷ The unity of analysis adopted was the individual ratings.

Regarding factor analysis, we performed principal components analysis (considering eigenvalues > 1) with VARIMAX rotation and Kaiser Normalization (see Artes and Barroso¹⁸ for statistical considerations concerning factor analysis applied to psychometric scales).

Results

The total sample consisted of 109 patients, 69 (63.3%) males and 40 (36.7%) females, with mean years of education of 7.5 (SD = 4.4, median = 7.5). Schizophrenia was the diagnosis of 74 (67.9%) patients, 23 patients (21.1%) had schizoaffective disorder, 7 patients (6.4%) had psychotic disorder not otherwise specified, 2 patients (1.8%) had delusional disorder, 2 had schizophreniform disorder (1.8%) and 1 patient had brief psychotic disorder (0.9%). Mean age was 37.6 years (SD = 10.5, median = 39.0), and mean duration of illness was 16.7 years (SD = 9.9, median = 17.0).

The mean SAI-E total score was 13.68 (SD = 6.66, median = mode = 13.33). When patients who had the SAI-E interview recorded were compared to those whose interviews were not recorded, no statistical difference was found with respect to either SAI-E total score (Mann-Whitney test, p = 0.28) or socio-demographic and clinical variables.

The ICC for each item of SAI-E ranged from 0.54 to 0.82; for the total score, the ICC was 0.90 (Table 1).

The factor analysis yielded 3 factors accounting for 71.72% of the variance. The factors loadings are shown in Table 2.

Discussion

The maximum value for ICC is 1.0 and occurs if all the raters perfectly agree on the rating for each patient.¹⁷ The ICC values found for the individual items can be considered satisfactory, and for the total score it was excellent (ICC 0.90, p < 0.001). Data on the reliability of individual items and total score of SAI-E have not been published yet. In the original reliability study of SAI, from which SAI-E derives, the ICC found was 0.72.²

Of note, by means of the adopted study method, using the recording of the instrument's administration, it was possible to evaluate only the reliability of score attribution, leaving the question of reliability of interview administration for a further analysis.

The factor analysis performed for the original version of the SAI-E indicated that three factors accounted for 66.5% of variance. Factor 1 (26.4% of variance) consisted of the two items related to the renaming of the symptoms as pathological and the hypothetical contradiction item. Factor 2 (24.2% of variance) consisted of the five items about awareness of the illness; and factor 3 (15.9% of variance), of the three items related to treatment compliance.⁸ In our study using the Brazilian version of the SAI-E, three factors were also found, which explained a greater percentage (71.72%) of the variance.

In addition, the composition of the factors we found was quite similar to that found for the original version of the SAI-E, and in concordance with the dimensions of insight postulated by David.⁷ Factor 1 (30.43% of variance) consisted of the five items about awareness of illness and the item about the awareness of the need for treatment, which corresponds to the "recognition of mental illness" dimension; factor 2 (26.75% of variance) consisted of the two items related to the renaming of symptoms as pathological, the item of hypothetical contradiction and item 3, "attribution of one's condition to a mental disorder", and it corresponds to the "relabeling of symptoms" dimension; and factor 3 (14.54% of variance) consisted of the two items regarding treatment compliance, corresponding to the "treatment compliance". Such a factor structure is in accordance with the concept of insight as a multidimensional construct, and it allows further investigation of the associations between the distinct insight dimensions (represented by the factor scores of SAI-E) and other variables.¹³

In the present study only inpatients were evaluated; future studies including Brazilian psychotic outpatients would extend our knowledge of the factor structure of the Brazilian version of the SAI-E.

Conclusions

The Brazilian version of the SAI-E presented good inter-rater reliability for the individual items and excellent inter-rater reliability for the total score, and featured a three-factor structure similar to the one presented by the original version of the instrument, in close correspondence with the dimensions of insight which the SAI-E is intended to assess.

Acknowledgements

The authors would like to thank Andrea Ferreira Semolini for her statistical advice and Ana Luisa Marques Traballi for her work as research assistant.

Financing: None

Conflicts of interests: None

Submitted: October 10, 2006

Accepted: February 6, 2007

Original article based on the master's dissertation: "Cross-cultural Adaptation of the Schedule for the Assessment of Insight - Expanded Version (SAI-E): Inter-rater Reliability and Factor Analysis of the Brazilian Version of SAI-E". Medical Sciences Graduate Program, Medical School, Universidade Estadual de Campinas (Unicamp), Campinas (SP), Brazil, 2006.

• 1. Amador XF, Strauss DH, Yale SA, Flaum MM, Edincott J, Gorman JM. Assessment of insight in psychosis. Am J Psychiatry. 1993;150(6):873-9.
• 2. David A, Buchanan A, Reed A, Almeida O. The assessment of insight in psychosis. Br J Psychiatry 1992;161:599-602.
• 3. Kemp R, David A. Insight and compliance. In: Blackwell, editor. Compliance and the treatment alliance in serious mental illness Amsterdam: Hardwood Academic Publishers; 1997. p. 61-86.
• 4. Ghaemi SN, Rosenquist KJ. Is insight in mania state-dependent? A meta-analysis. J Nerv Ment Dis. 2004;192(11):771-5.
• 5. Sanz M, Constable G, Lopez-Ibor I, Kemp R, David AS. A comparative study of insight scales and their relationship to psychopathological and clinical variables. Psychol Med. 1998;28(2):437-46.
• 6. Drake RJ, Lewis SW. Insight and neurocognition in schizophrenia. Schizophr Res. 2003;62(1-2):165-73.
• 7. David A. Insight and psychosis. Br J Psychiatry. 1990;156:798-808.
• 8. David AS, Morgan KD, Mallet R, Leff J Murray RM. Insight: unitary or multidimensional phenomenon? Schizophr Res 2003;60(1):14.
• 9. Fiss N, Chaves AC. Translation, adaptation and reliability study of the Scale to Assess Unawareness of Mental Disorder SUMD. Rev Bras Psiquiatr. 2005;27(2):143-5.
• 10. Gigante AD, Castel S. Insight into schizophrenia: a comparative study between patients and family members. Sao Paulo Med J. 2004;122(6):246-51.
• 11. Flaherty JA, Gaviria FM, Pathak D, Mitchell T, Wintrob R, Richman JA, Birz S. Developing instruments for cross-cultural psychiatric research. J Nerv Ment Dis. 1988;176(5):257-63.
• 12. Dantas CR. Adaptação transcultural do "Schedule for the Assessment of Insight - Expanded Version (SAI-E)", estudo de confiabilidade e análise factorial da versão brasileira do SAI-E [dissertation]. Campinas (SP): Universidade Estadual de Campinas - UNICAMP; 2006.
• 13. Dantas CR, Banzato CEM. Predictors of insight in psychotic inpatients. Schizophr Res. 2007;91:263-5.
• 14. Bartko JJ. Measurement and reliability: statistical thinking considerations. Schizophr Bull. 1991;17(3):483-9.
• 15. Amorin P. Mini International Neuropsychiatric Interview (MINI): validação de entrevista breve para diagnóstico de transtornos mentais. Rev Bras Psiquiatr. 2000;22(3):106-15.
• ¹⁶
  American Psychiatric Association. Manual diagnóstico e estatístico de transtornos mentais – texto revisado. Tradução de Cláudia Dornelles. 4a ed. rev. Porto Alegre: Artmed, 2002. 880 p. Original title: Diagnostic and statistical manual of mental disorders – revised text.
• 17. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420-8.
• 18. Artes R, Barroso LP. Aspectos estatísticos da análise fatorial de escalas de avaliação. In: Gorestein C, Andrade LHSG, Zuardi A, editors. Escalas de avaliação clínica em psiquiatria e psicofarmacologia. São Paulo: Lemos Editorial; 2000. p. 35-44.

Publication Dates

• Publication in this collection
  11 May 2007
• Date of issue
  Dec 2007

History

• Accepted
  06 Feb 2007
• Received
  10 Oct 2006