Acessibilidade / Reportar erro

Instrument to screen cases of pervasive developmental disorder: a preliminary indication of validity

Instrumento para rastreamento dos casos de transtorno invasivo do desenvolvimento: estudo preliminar de validação

Abstracts

OBJECTIVE: To translate into Portuguese, back-translate, culturally adapt and validate a screening instrument for pervasive developmental disorder, the Autism Screening Questionnaire, for use in Brazil. METHOD: A sample of 120 patients was selected based on three groups of 40: patients with a clinical diagnosis of pervasive developmental disorder, Down syndrome, or other psychiatric disorders. The self-administered questionnaire was applied to the patients' legal guardians. Psychometric measures of the final version of the translated questionnaire were tested. RESULTS: The score of 15 had sensitivity of 92.5% and specificity of 95.5% as a cut-off point for the diagnosis of pervasive developmental disorder. Internal validity for a total of 40 questions was 0.895 for alpha and 0.896 for KR-20, ranging from 0.6 to 0.8 for both coefficients. Test and retest reliability values showed strong agreement for most questions. CONCLUSIONS: The final version of this instrument, translated into Portuguese and adapted to the Brazilian culture, had satisfactory measurement properties, suggesting preliminary validation proprieties. It was an easy-to-apply, useful tool for the diagnostic screening of individuals with pervasive developmental disorder.

Pervasive development disorders; Autistic disorder; Validation studies; Questionnaires; Diagnosis, clinical


OBJETIVO: Tradução, retro-versão, adaptação cultural e validação do Autism Screening Questionnaire para a língua portuguesa e para o seu uso no Brasil. MÉTODO: Foi selecionada uma amostra inicial de 120 pacientes, encaminhados de duas clínicas privadas e uma pública, divida em três grupos de 40 pacientes distintos: pacientes com diagnóstico clínico de transtornos globais do desenvolvimento ou transtornos invasivos do desenvolvimento; de síndrome de Down e de outros transtornos psiquiátricos. O questionário foi aplicado aos responsáveis legais dos pacientes seguindo os padrões de um questionário auto-aplicável. As medidas psicométricas do questionário traduzido, na sua versão final, foram testadas. RESULTADOS: Valores de sensibilidade de 92,5% e especificidade de 95,5% foram encontrados para uma pontuação de 15, como sendo um valor discriminativo para os sujeitos com características de transtornos globais do desenvolvimento/transtornos invasivos do desenvolvimento. A validade interna para o total das 40 questões foi de 0,895, com uma variação entre 0,6 a 0,8. Os valores de confiabilidade obtidos pelo teste e re-teste demonstraram que a maioria das questões obteve alta concordância. CONCLUSÕES: A versão final do instrumento de pesquisa, traduzido e adaptado à cultura brasileira, apresentou propriedades de medida satisfatórias, sugerindo adequadas propriedades preliminares de validação. É um instrumento de fácil aplicação e uma ferramenta útil para a realização de um screening diagnóstico em indivíduos com transtornos globais do desenvolvimento/transtornos invasivos do desenvolvimento.

Transtornos globais do desenvolvimento; Transtorno autístico; Estudos de validação; Questionários; Diagnóstico clínico


BRIEF REPORT

Instrument to screen cases of pervasive developmental disorder – a preliminary indication of validity

Instrumento para rastreamento dos casos de transtorno invasivo do desenvolvimento – estudo preliminar de validação

Fábio Pinato SatoI; Cristiane Silvestre PaulaII; Rosane LowenthalII; Eduardo Yoshio NakanoIII; Décio BrunoniII; José Salomão SchwartzmanII; Marcos Tomanik MercadanteIV

IGraduate Program, Department of Psychiatry, Universidade de São Paulo (USP), São Paulo (SP), Brazil

IIGraduate Program in Developmental Disorders, Universidade Presbiteriana Mackenzie, São Paulo (SP), Brazil

IIIDepartment of Statistics, Universidade de Brasília (UNB), Brasília (DF), Brazil

IVDepartment of Psychiatry, Child and Adolescent Medical Psychiatric Unit, Universidade Federal de São Paulo (Unifesp), São Paulo (SP), Brazil

Correspondence Correspondence: Fábio Pinato Sato Rua Capote Valente, 432/24 05409-001 São Paulo, SP, Brazil Phone/Fax: (+55 11) 3064-1278 E-mail: fpsato@uol.com.br

ABSTRACT

OBJECTIVE: To translate into Portuguese, back-translate, culturally adapt and validate a screening instrument for pervasive developmental disorder, the Autism Screening Questionnaire, for use in Brazil.

METHOD: A sample of 120 patients was selected based on three groups of 40: patients with a clinical diagnosis of pervasive developmental disorder, Down syndrome, or other psychiatric disorders. The self-administered questionnaire was applied to the patients' legal guardians. Psychometric measures of the final version of the translated questionnaire were tested.

RESULTS: The score of 15 had sensitivity of 92.5% and specificity of 95.5% as a cut-off point for the diagnosis of pervasive developmental disorder. Internal validity for a total of 40 questions was 0.895 for alpha and 0.896 for KR-20, ranging from 0.6 to 0.8 for both coefficients. Test and retest reliability values showed strong agreement for most questions.

CONCLUSIONS: The final version of this instrument, translated into Portuguese and adapted to the Brazilian culture, had satisfactory measurement properties, suggesting preliminary validation proprieties. It was an easy-to-apply, useful tool for the diagnostic screening of individuals with pervasive developmental disorder.

Descriptors: Pervasive development disorders; Autistic disorder; Validation studies; Questionnaires; Diagnosis, clinical

RESUMO

OBJETIVO: Tradução, retro-versão, adaptação cultural e validação do Autism Screening Questionnaire para a língua portuguesa e para o seu uso no Brasil.

MÉTODO: Foi selecionada uma amostra inicial de 120 pacientes, encaminhados de duas clínicas privadas e uma pública, divida em três grupos de 40 pacientes distintos: pacientes com diagnóstico clínico de transtornos globais do desenvolvimento ou transtornos invasivos do desenvolvimento; de síndrome de Down e de outros transtornos psiquiátricos. O questionário foi aplicado aos responsáveis legais dos pacientes seguindo os padrões de um questionário auto-aplicável. As medidas psicométricas do questionário traduzido, na sua versão final, foram testadas.

RESULTADOS: Valores de sensibilidade de 92,5% e especificidade de 95,5% foram encontrados para uma pontuação de 15, como sendo um valor discriminativo para os sujeitos com características de transtornos globais do desenvolvimento/transtornos invasivos do desenvolvimento. A validade interna para o total das 40 questões foi de 0,895, com uma variação entre 0,6 a 0,8. Os valores de confiabilidade obtidos pelo teste e re-teste demonstraram que a maioria das questões obteve alta concordância.

CONCLUSÕES: A versão final do instrumento de pesquisa, traduzido e adaptado à cultura brasileira, apresentou propriedades de medida satisfatórias, sugerindo adequadas propriedades preliminares de validação. É um instrumento de fácil aplicação e uma ferramenta útil para a realização de um screening diagnóstico em indivíduos com transtornos globais do desenvolvimento/transtornos invasivos do desenvolvimento.

Descritores: Transtornos globais do desenvolvimento; Transtorno autístico; Estudos de validação; Questionários; Diagnóstico clínico

Introduction

Pervasive developmental disorders (PDD) are a series of conditions characterized by deficits in the development of sociability and communication and by a restricted behavioral pattern. A prevalence of 1% has been reported when expanded diagnostic criteria were used1. A pilot study conducted in our region suggested a prevalence close to the lower limit of rates found in the literature2.

There are two partially validated instruments for the diagnosis of PDD in Brazil: The Autistic Traits of Evaluation Scale (ATA)3 and Autism Behavior Checklist (ABC)4. The widespread use of ASQ in the literature and its good performance in discriminating individuals with PDD from others5 were the main reasons to choose this instrument. This study was conducted to translate into Portuguese, back-translate, culturally adapt and validate the Autism Screening Questionnaire (ASQ)5 for use in Brazil.

Method

The original ASQ was translated into Portuguese and adapted by specialists in the medical area. After this stage, the questionnaire was back-translated by a bilingual professional. The material was presented to three child and adolescent psychiatrists, who are not participating in the study. At the end of the process, translators and the committee agreed that semantic, idiomatic and cultural equivalence was satisfactory, as previously suggested by others6,7. The semantic structure of ASQ was not altered either in the translation process into Portuguese or in the back-translation into English (exception for minor changes in questions 14 and 33).

After this process, the instrument was applied to the legal guardians of 120 children divided into three groups of three different work settings (two private and one public) in which patients were receiving medical care. Based on the lack of internationally validated instruments for the diagnosis of PDD in Brazil, clinical criteria, according to DSM-IV, were chosen to classify all the subjects. The three groups were comprised of 40 children with a clinical diagnosis of PDD (PDDG), 40 children with Down syndrome (control for cognitive performance – DownG), and 40 children meeting diagnostic criteria for other psychiatric disorders (anxiety, depression, attention deficit disorder with or without hyperactivity, without PDD – PchG). All 120 children were selected based on order of presentation at these three clinics during a 4 week period of time. A pretest was applied to evaluate children with Down syndrome8.

The participants' guardian was contacted by phone and the investigator explained that the study consisted of 40 yes/no questions about the patient's life history. As all selected cases agreed to participate, we sent a written informed consent by mail and an investigator applied the questionnaire (mean time was 20 minutes). The patient's guardian signed the form and mailed it back to the investigator. The study was approved by the Institutional Research Ethics Committee from Universidade de São Paulo Medical School.

After 6 months, the questionnaire was applied again by the same investigator to 10 randomly selected participants in each group.

The Statistical Package for the Social Sciences 11.5 (SPSS) for Windows was used for statistical analyses. Mean scores (MS) for each diagnosis were calculated, and the ANOVA and Tukey post-hoc test were used to compare these means. A ROC (Receiver Operating Characteristics) curve was obtained indicating several cut-off points according to sensitivity and specificity levels. The Kuder-Richardson KR-209 and Cronbach's alpha coefficients10 were used to assess internal consistency of the entire questionnaire and of the three domains individually. Cohen's kappa coefficient11 was used to measure questionnaire reliability. Frequencies were compared using the chi-square test. The level of significance was set at a = 0.05 or 5% for all statistical tests.

Results

Each group had 40 participants, and their mean ages were 11.1 years in the PchG (27 boys), 9.9 years in the DownG (22 boys), and 9.8 years in the PDDG (34 boys).

MS varied according to diagnosis. In patients in the PchG, MS was 7.2 (SD = 4.1); in the DownG, 9.0 (SD = 4.2); and in the PDDG, 21.7 (SD = 5.4). ANOVA revealed a significant difference between means (F = 116.7; p < 0.001). According to the post-hoc test, only PDDG had an MS significantly greater than that of the other groups (p < 0.001 for DownG and p < 0.001 for PchG). The DownG had an MS slightly greater than that of the PchG, but the difference was not statistically significant (p = 0.203).

Sensitivity and specificity analysis showed that the area under the ROC curve was 0.981 (SE = 0.011; p < 0.001). The cut-off point of 14.5 (individuals were classified as having PDD if their score was > 15) had sensitivity of 92.5% and specificity of 95.0%. This cut-off point presented balanced levels of sensitivity and specificity.

1. Analysis of both control groups

In individuals with PDD, MS was 21.7 (SD = 5.4); for the total control group (TCG = DownG + PchG), it was 8.1 (SD = 4.2). The t test confirmed a significant difference between these means (t = 13.85; p < 0.001).

2. Internal validity

The analysis of internal validity showed alpha = 0.895 and KR-20 = 0.896 for the entire questionnaire. Alpha values varied from 0.621 to 0.838 when questions were separated according to domains: language (0.687); behavior (0.621); sociability (0.838), while KR-20 values varied from 0.625 to 0.840: language (0.685); behavior (0.625) and sociability (0.840).

3. Retest reliability (Cohen's kappa coefficient)

The questionnaire was applied again to PchG and PDDG after about 240 days and to DownG at about 730 days after the initial test. No control patients had a score above the cut-off point, and all patients had scores that indicated PDD at retest. Only one patient in the DownG, who had scored 18 in the first ASQ application (the patient was not able to talk then), scored 7 in the retest application. The MS DownG was 7.7 (SD = 4.373; SE = 1.383) at time point 1 and 6.6 (SD = 2.914; SE = 0.921) at time point 2 (t = 0.884; df = 9; p = 0.421); for PchG, 7.2 (SD = 2.898; SE = 0.917) at time point 1 and 5.7 (SD = 3.302; SE = 1.044), at time point 2 (t = 0.183; df = 9; p = 0.859); and for PDDG, 21.7 (SD = 5.971; SE = 1.888) at time point 1 and 24.1 (SD = 6.082; SE = 1.923) at time point 2 (t = 1.500; df = 9; p = 0.168).

Question 5 had the weakest agreement (kappa = 0.374; p = 0.045), and question 16 the strongest (kappa = 0.927; p < 0.001). Only nine questions had kappa below 0.6 (# 5 = 0.374; 9 = 0.426; 22 = 0.426; 7 = 0.485; 25 = 0.533; 11 = 0.561; 31 = 0.571; 21 = 0.583; 24 = 0.598).

The comparison of each question in all three groups showed that questions 1, 5, 7, 13 and 23 had no significant prevalence (or absence) of answer "1" for one of the diagnosis (p > 0.05); they were, therefore, questions with lower classification power.

Discussion

The analysis of ASQ psychometric properties revealed that it has good sensitivity and specificity and that it can discriminate cases of PDD from cases of other psychiatric disorders or mental deficiency. Moreover, the values found in retesting, which showed 100% diagnostic agreement in tests applied at least 8 months after the initial application, suggest excellent agreement12,13.

The ATA3 and the ABC4 were the only scales available in Portuguese before this study. The ATA was applied to a sample of 61 children aged 2-18 years, 30 with autism and 31 with moderate mental deficiency. The analysis of external validity showed that the agreement with DSM-IV criteria was weak (kappa = 0.04); therefore, the data obtained with that instrument could not be generalized for the general population. Conversely, internal validity was 100%, which indicated strong agreement between clinical diagnosis and the diagnosis provided by that scale. That scale had a sensitivity of 0.96 according to DSM-IV criteria, and Cronbach's alpha was 0.7110. The ABC was applied to three groups, 38 mothers of children diagnosed with autism; 43 mothers of children with language disorders other than autism and 52 mothers of children who had no linguistic or behavioral complaints. The ABC correctly identified 81.6% of the autistic children. The cut-off value was 49, sensitivity was 92.1% and specificity was 92.6%. Few studies in the literature used these scales, which limits the comparison of results.

ASQ, conversely, has been widely used. Two studies conducted by our research team demonstrated its applicability. A study that evaluated all patients with Down syndrome in the city of Curitiba, southern Brazil8, showed that ASQ was capable of identifying all cases of PDD in that population. In the first epidemiologic study of PDD in Latin America, a prevalence of 0.3% was found2. In both studies, ASQ was a reliable instrument that discriminated PDD cases from non-PDD cases.

The validation of an instrument with good psychometric properties is essential for the advancement of research and public health programs in Brazil. The translated version of ASQ was appropriate for use in our country. The formerly ASQ, currently Social Communication Questionnaire (SCQ)14, a copyrighted instrument, however, is one of its limitations, since its use is not free of costs.

Conclusions

The current study is a preliminary indication of validity. Other limitations were related to sample size, lack of IQ scores of the subjects, and lack of gold-standard diagnostic instruments. Further studies are required to improve the validation proprieties of the Brazilian version of ASQ.

Acknowledgments

The authors wish to acknowledge the translation and back-translation team, the Revision Committee and the undergraduate Psychology students of Universidade Presbiteriana Mackenzie for all their assistance in this study.

Submitted: April 16, 2008

Accepted: July 14, 2008

  • 1. Fombonne E. Epidemiological surveys of autism and other pervasive developmental disorders: an update. J Autism Dev Disord 2003;33(4):365-82.
  • 2. Ribeiro SH. Prevalência dos transtornos invasivos do desenvolvimento Município de Atibaia: um estudo piloto [dissertação]. São Paulo: Universidade Presbiteriana Mackenzie; 2007.
  • 3. Assumpção FB Jr, Francisco B, Kuczynski E, Gabriel MR, Rocca CC. Validity and reliability of a scale for the assessment of austistic behaviour. Arq Neuro Psiquiatr 1999;57(1):23-9.
  • 4. Marteleto MR, Pedremônico MR. Validity of autism behavior checklist (ABC): preliminary study. Rev Bras Psiquiatr 2005;27(4): 295-301.
  • 5. Berument SK, Rutter M, Lord C, Pickles A, Bailey A. Autism screening questionnaire: diagnostic validity. Br J Psychiatry 1999;175: 444-451.
  • 6. Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol. 1993;46(12):1417-32.
  • 7. Ciconelli RM, Ferraz MB, Santos WS, Meinão IM, Quaresma MR. Tradução para a língua portuguesa e validação do questionário genérico de avaliação de qualidade de vida SF-36 (Brasil SF-36). Rev Bras Reumatol 1999;39(3):143-50.
  • 8. Lowenthal R, Paula CS, Schwartzman JS, Brunoni D, Mercadante MT. Prevalence of pervasive developmental disorder in Down's Syndrome. J Autism Dev Disord. 2007;37(7):1394-5.
  • 9. Kuder GF, Richardson MW. The theory of the estimation of test reliability. Psychometrika. 1937;2:151-60.
  • 10. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297-334.
  • 11. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:3746.
  • 12. Rietveld T, van Hout R. Statistical techniques for the study of language and language behavior Berlin: Mouton De Gruyter; 1993.
  • 13. Di Eugenio B, Glass M. The Kappa statistics: a second look. Computational Linguistics. 2004;30(1):95-101.
  • 14. Rutter M, Bailey A, Lord C. SCQ: The Social Communication Questionnaire 2nd ed. Los Angeles (CA): Western Psychological Services; 2003.
  • Correspondence:

    Fábio Pinato Sato
    Rua Capote Valente, 432/24
    05409-001 São Paulo, SP, Brazil
    Phone/Fax: (+55 11) 3064-1278
    E-mail:
  • Publication Dates

    • Publication in this collection
      23 Mar 2009
    • Date of issue
      Mar 2009

    History

    • Accepted
      14 July 2008
    • Received
      16 Apr 2008
    Associação Brasileira de Psiquiatria Rua Pedro de Toledo, 967 - casa 1, 04039-032 São Paulo SP Brazil, Tel.: +55 11 5081-6799, Fax: +55 11 3384-6799, Fax: +55 11 5579-6210 - São Paulo - SP - Brazil
    E-mail: editorial@abp.org.br