Validity and reliability of a teacher's scale developed in Brazil for assessment of hyperactive: impulsive behavior and inattention in children and adolescents

Validade e confiabilidade de uma escala para professores desenvolvida no Brasil para avaliação de comportamento hiperativo: impulsivo e desatenção em crianças e adolescentes

ARTICLES

Validity and reliability of a teacher's scale developed in Brazil for assessment of hyperactive: impulsive behavior and inattention in children and adolescents

Validade e confiabilidade de uma escala para professores desenvolvida no Brasil para avaliação de comportamento hiperativo: impulsivo e desatenção em crianças e adolescentes

Ana Lúcia Novais Carvalho; Alex Christian Manhães; Sérgio Luis Schmidt

Universidade do Estado do Rio de Janeiro, Centro Biomédico, Laboratório de Neurofisiologia e Avaliação Neurocomportamental. Av. Prof. Manoel de Abreu, 444, 5º andar, Vila Isabel, 20550-170, Rio de Janeiro, RJ, Brasil. Correspondência para/Correspondence to: S.L. SCHMIDT. E-mails: <[e-mail]>; <[e-mail]>

ABSTRACT

This study was designed to develop and assess the internal consistency and factor structure of a teacher's scale for the assessment of hyperactive/impulsive and inattentive behavior in a sample of 221 children from different cities in the State of Rio de Janeiro, Brazil. Forty-five teachers rated the children. The internal consistency of the scale was evaluated using Cronbach's alpha. An orthogonal varimax rotation that maximizes the variance of the squared loadings for each factor was used to find the simplest possible factor structure. There is no internal discrepancy in the item content of the scale. Exploratory factor analysis showed four primary factors (hyperactivity/impulsivity; inattention; social isolation; self-confidence) that are psychologically meaningful. When factor analysis was carried out there was no substantial difference from other studies when compared with the results of other western and oriental countries. It was concluded that this teacher's scale can be a useful aid to clinicians in the identification of children with hyperactivity/impulsivity and inattention problems.

Uniterms: Attention deficit disorder with hyperactivity. Mental disorders. Psychological assessment.

RESUMO

Este estudo foi elaborado para avaliar a consistência interna e a estrutura fatorial de uma escala para professores, destinada à avaliação de comportamento hiperativo/impulsivo e desatenção em uma amostra de 221 crianças de diferentes cidades do Estado do Rio de Janeiro. Quarenta e cinco professores avaliaram as crianças. A consistência interna da escala foi avaliada usando o coeficiente alfa de Cronbach. Uma rotação Varimax (ortogonal), que maximiza a variância ao quadrado das cargas para cada fator, foi usada para encontrar a estrutura fatorial mais simples. Não houve discrepâncias no conteúdo dos itens da escala. A análise fatorial exploratória indicou quatro fatores primários (hiperatividade/impulsividade; desatenção; isolamento social; autoconfiança) com significado psicológico. Os resultados obtidos neste estudo não diferem daqueles realizados em outros países ocidentais e orientais. Conclui-se que esta escala para professores pode auxiliar na identificação de crianças com queixas de hiperatividade/impulsividade e desatenção.

Unitermos: Transtorno do deficit de atenção com hiperatividade. Transtornos mentais. Avaliação psicológica.

Previous studies have shown that physicians and psychologists who rely only on their clinical judgment may experience difficulties in identifying children with behavioral problems (Barkely, 1990; Schmidt, Snyder, Rouget & Gray, 2000). The use of checklists completed by teachers or parents allows for a greater level of accuracy, even when considering the limitations of the instruments (Greenhill, 1998; Schatz, Ballantyne & Trauner, 2001; Ulloa, Narváez, Arroyo, del Bosque & de la Peña, 2009). Studies have demonstrated that in some cases, behavior rating scales are even more effective in classifying children with learning and cognitive disorders than traditional psychological tests. These data were obtained by Dewey, Crawford and Kaplan (2003) in a study with 159 children using a parent's scale (Parent Ratings of Everyday Cognitive and Academic Abilities) and psychometrics measures, such as the Woodcock-Johnson Psycho-Educational Battery-Revised, the Bruininks-Oseretsky Test of Motor Proficiency-Short Form, the Wechsler Intelligence Scale for Children, 3^rd edition (Vocabulary and Block Design subtests), and the Developmental Test of Visual-Motor Integration. The results demonstrated that the scale was more efficient in classifying children with problems (reading disabilities, Attention Deficit/Hyperactivity Disorder - ADHD and combined) than the psychometric measures alone. These limitations of neuropsychological tests, specifically in the diagnosis of ADHD, were also recently highlighted in an article by Andrade and Flores-Mendonza (2010), which confirms the usefulness of scales for parents and teachers.

The presence of primary inattention disorder and/or hyperactivity/impulsivity is a diagnostic criterion for ADHD, the most common childhood psychiatric disorder, affecting 5% of the school population. According to the Diagnostic and Statistical Manual of Mental Disorders - 4^th Edition Text Revision - DSM-IV-RT (American Psychiatric Association, 2000) these behavioral changes must be present in more than one venue frequented by children, and their behavior at home and at school is commonly rated by parents and teacher. As suggested by Mares, McLuckie, Schwartz and Saini (2007) and Tripp, Schaughency and Clarke (2006) it is essential to collect data from these two sources, considering that parents and teachers emphasize different aspects during the evaluation. Such differences are commonly described in the literature (Mitsis, McKay, Schultz, Newcorn & Halperin, 2000). Cho et al. (2011), in a recent study with the aim of comparing data obtained from a scale for teachers and parents, on the assessment of cognitive performance in children, confirm these data. These authors found that teachers are more skilled in assessing hyperactive/impulsive behavior, which facilitates the identification of children with low cognitive performance. This facility by professionals is probably related to the fact that teachers know about child development and deal with a large number of children. This finding does not exclude the need for data collection though scales filled out by parents. The study makes it clear that parents and teachers point out different but important aspects of child behavior.

With further regard to the use of scales, an American research with 401 pediatricians and family practice physicians, in a prospective cohort study of 22,059 children from 4 to 14 years, showed that 53.5% of them used school reports to diagnose attentional or hyperactivity problems (Carey, 1999). Although there was no data available as regards the use of questionnaires by Brazilian physicians and psychologists, it is believed that the use of rating scales to identify children with behavioral and learning problems needs to be emphasized. The socio-economic aspects of Brazil make the development of teacher behavior rating scales particularly relevant. The low educational level of the population is still an important social problem. At the end of the twentieth century, the average Brazilian over 25 had only 6 years of formal education. In 2001, 13.6% of the population over 15 years of age was illiterate (Henriques, 2001). Considering these facts, it is common to find parents who have a low educational level or who are even illiterate, and for these people written scales may not be a good way of providing information. Therefore, it was decided to develop a scale to assess hyperactive/impulsive behavior and inattention to be filled out by teachers. The present study, in particular, was conducted to provide information about the psychometric properties of this scale, including the analysis of its reliability and validity, which are essential characteristics of an instrument that is to be used clinically (Glascoe, Martin & Humphrey, 1990; Meisels, 1989).

Three specific objectives and respective hypotheses were investigated. First, the checklist was verified to see whether presented high internal reliability. This involves the relationship of the items comprising the scale. Second, an evaluation was made to establish whether the checklist items could be expressed by a small number of factors. To be valid, the resulting factors of the rating scale must be psychologically interpretable and meaningful (Nunnally, 1978). Finally we wished to know whether the factors derived from the exploratory factor analysis differed from those of other studies conducted in different western and oriental countries.

Method

Participants

The participants were 45 teachers who rated 221 scales from children drawn from elementary schools located in four different cities in the State of Rio de Janeiro. The sample consisted of 108 boys (mean age=8.91 years, SD=2.08 years, minimum=6 years, maximum=15 years) and 113 girls (mean age=8.88 years, SD=2.41 years, minimum=6 years, maximum=15 years).

Considering socio-economic characteristics (particularly educational level), a heterogeneous population can be found in the State of Rio de Janeiro (Lago, 1999). The Human Development Index (HDI), from the United Nations Development Program (2000), was used to characterize the cities. The HDI is a composite index and three variables are considered for its composite: life expectancy at birth, educational attainment and Gross National Income (GNI) per capita (PPP US$). Here, the HDI of each city and the number of participants are described (Table 1).

The checklist was answered by the teacher who spent most of the time with the child in the classroom, and for each child only one teacher answered the questionnaire. It should be mentioned that in the Brazilian educational system, during the first segment of elementary school, in which children are between 6 and 10 years old, schoolchildren normally have only one teacher, whereas those in the second segment (aged between 11 and 14) have more than one teacher.

The participation of the child and the teacher were voluntary. Parents and school principals signed terms of informed consent as regards the voluntary participation of the children in the research. This study was approved by the Ethics Committees of the Universidade do Estado do Rio de Janeiro (UERJ), Protocol 004.2004, and all participants signed an informed consent before their inclusion in the sample.

Measures

This scale was developed from data derived from academic meetings held with clinicians experienced in the area of infant diagnosis. Data published in national and international articles and diagnostic criteria of DSM-IV RT (American Psychiatric Association, 2000) for ADHD were taken into account.

The scale is divided into two parts: qualitative and quantitative. The qualitative part is intended to describe the child's main problems, report school performance, describe whether the child needs or receives special care or help to develop his/her school activities, and give information about the child's family. Considering this part, the teacher has the opportunity to say whether the child has attention or behavior problems, and to describe his/her academic performance. The quantitative part is composed of 58 questions with information about behavior in the classroom, peer group, and disruptive behavior. These questions are rated on an oral assessment scale ranging from 1 to 3 reflecting problem prevalence (1=not true; 2=somewhat or sometimes true; 3=very true or often true).

Analytical methods

The analysis of the qualitative part of the scale was performed by means of the content analysis technique, through which the observations described by the teacher were classified into learning complaint, attention, behavior or individual without complaint. We used the quantitative approach to content analysis, whereby the content of the information was analyzed through systematic and objective rules of categorization. For each of the three questions concerning the qualitative part of the scale, categories were established and analyzed in the light of referential coding. Each unit of text could be classified according to the categories raised (Bauer, 2002).

The internal consistency of the 58 scorable-item scale was evaluated using Cronbach's alpha. Alpha is a correlation coefficient and ranges in value from -1 to 1. If Cronbach's alpha drops below 0.7, the hypothesis of internal reliability cannot be accepted (Barret & Kline, 1981; Cattell, 1966). To evaluate how each individual item affects the reliability of the scale, alpha was calculated removing each of the items of the scale.

Raw data from the questionnaire were factor analyzed. To find the simplest possible factor structure, an orthogonal varimax rotation that maximizes the variance of the squared loadings for each factor was used. Only factors that account for variances greater than 1 (eigenvalues [λ] greater than 1) were included. With regard to the factor loadings, only the most highly correlated variables (greater than 0.50) for each factor (Table 2) were considered.

To verify the relationship between quantitative and qualitative parts of the scale, standard scores were constructed for each factor, considering gender and age.

Results

Internal reliability

For the 221 participants, the checklist presented high internal reliability as indicated by Cronbach's alpha coefficients (α= 0.96). Elimination of any of the items from the scale caused little change in a.

Factor analyses

Four factors were extracted (Table 2): Factor 1- Hyperactivity/Impulsivity (explained variance = 36.3%); Factor 2- Inattention (explained variance = 11.4%); Factor 3- Social isolation (explained variance = 5.2%); Factor 4- Self-confidence (explained variance = 3.3%)

Correlation between quantitative and qualitative questions

In the analysis of the standardized data, it was seen that the individuals who presented complaints, considering the qualitative part of the scale, produced higher scores in factors 1, 2 and 3: F1 (t=- 8.643; gl=157; p<0.001); F2 (t=-12.598; gl=143; p<0.001); F3 (t=-5.468; gl=169; p<0.001); F4 (t=-2.250; gl=218; p=0.025).

Discussion

Internal reliability

For internal consistency analyses, the samples should contain at least 100 subjects (Guilford, 1956; Nunnally, 1978). Barret and Kline (1981) found that a minimum of 100 subjects and a variable-subject ratio of 2:1 are needed for a reliable factor analysis. Thus, the sample of 221 subjects used, with a variable-subject ratio of at least 2:1 supports the stability of the reliability analysis.

The components of the scale can be viewed as an internally reliable instrument for obtaining quantified information from teachers. Empirical analysis demonstrated that there is no internal discrepancy in the item content of the checklist. High internal consistency is a prerequisite of validity (Guilford, 1956; Nunnally, 1978).

Factor analyses

Arrindel and Endle (1985) claimed that stable factors required the sample to be approximately 20 times larger than the number of factors. According to these authors, variable-subject ratio is not important when compared with the ratio of sample size to factors. Therefore if only reliable factors are to be considered, a conservative approach requires that the number of factors in the study should never be greater than 11. As the study showed four factors, the hypothesis that the factors are reliable is confirmed.

In support of the hypothesis of construct validity, the number of items of each checklist could be expressed by a smaller number of factors, which are?/were psychologically meaningful.

Correlation between quantitative and qualitative questions

The comparison between subjects with and without complaints, identified through the analysis of the qualitative questions of the scale, demonstrated that the individuals with complaints presented higher scores than those without complaints, as described by the teacher. These differences were statistically significant for factors 1, 2 e 3. In conclusion, the present checklist can be considered a valid instrument in helping to identify children with learning and behavior problems.

Comparison with other western and oriental studies

When the primary factors were considered there was no substantial difference from studies carried out in other western countries (Conners, Sitarenios, Parker & Epstein, 1998; Pineda et al., 2005) as well as in many oriental countries (Aramaki & Uno, 2004; Liu et al., 2000; Luk, Leung & Lee, 1988 ; Yao, Solanto & Wender, 1988). These studies support ADHD as an universally valid syndrome. An important example occurred in Japan, where studies on ADHD symptoms were uncommon before 2005. In that country, the results of a large scale survey of school teachers showed 6.3% of children with developmental disorder, and 2.5% of them with ADHD (Tani, Okada, Ohnishi, Nakajima & Tsujii, 2010). A checklist to be completed by parents and teachers has been validated for use in assessment of Japanese children (Davis, Cheung, Takahashi, Shinoda & Lindstrom, 2011) and ADHD has become the focus of many researches. These studies support that ADHD symptoms are comparable across cultures. Inattentiveness and impulsivity can be viewed as factors that emerge in different cultures when behavioral problems of children are analyzed by instruments conveniently adapted to the local language and culture.

Limitations of the study and directions for future research

In this study, the question/issue of whether the scale can be reliably used as successive measures (test-retest reliability) of child behavior was not evaluated. While construct validity was demonstrated, there are also other types of validity, such as predictive validity. Therefore, further study is necessary to evaluate other types of validity and reliability.

As previous investigators have reported that parents are a useful source of information on children (Barkley, 1990; Connors et al., 1998; Schatz et al., 2001), systematic oral information from parents with a low level of formal education could be collected by clinicians and the results compared with the written information provided by teachers.

Although the primary method used by psychologists to test the validity of constructs has been factor analysis (Cronbach & Meehl, 1955), recent studies have addressed the problems that can arise from using shared-variance techniques to evaluate construct validity. In particular, it has been proposed that a correlation between two or more variables should not represent a unitary cognitive construct (Delis, Jacobson, Bondi, Hamiton & Salmon, 2003). However, previous studies have demonstrated the stability of the conceptual relationship across a number of homogeneous clinical populations, and thus, confirmatory factor analysis can extend construct validity analysis based on shared-variance procedures (Larrabee, 2001). Therefore, confirmatory factor analysis studies need to be conducted on specific cognitive constructs in selected homogeneous patient populations.

Conclusion

The components of the scale are quantifiable. The checklist presents high internal reliability and can be reduced to a smaller number of factors that are psychologically meaningful. Thus, it will be possible to develop abbreviated forms of this scale. Rating scales do not have to be long instruments; they have to be able to evaluate the interest construct. In fact, clinical scales can be refined through factor-analytic techniques.

Considering the need to improve the efficiency of our educational system, especially for those children with treatable disabilities, it is suggested that the results of this study may contribute to integrate the activities of pediatricians, psychologists and teachers, thereby improving the actions that are directed towards children with behavioral and educational problems. This checklist may be a useful aid in the identification of children with behavioral and attention problems.

References

American Psychiatric Association. (2000). Diagnostic and statistical manual for mental disorders (4^th ed., text rev.). Washington, DC: Author.

Andrade, A. C., & Flores-Mendoza, C. (2010). Transtorno do déficit de atenção/hiperatividade: o que nos informa a investigação dimensional? Estudos de Psicologia (Natal), 15(1),17-24.

Aramaki, Y., & Uno, H. (2004). Impulsive behaviors in attention-deficit/hyperactivity disorder. No To Hattatsu, 36,467-472.

Arrindel, W. A., & Endle, J. (1985). An empirical test of the utility of the observation to variables ratio in factor and components analysis. Applied Psychological Measurement, 9,165-178.

Barkley, R. A. (1990). Attention deficit hyperactivity disorder: a handbook for diagnosis and treatment. New York: The Guilford Press.

Barret, P., & Kline, P. (1981). The observation to variable ratio in factor analyses. Journal of Personality and Group Behavior, 2,94-98.

Bauer, M. W. (2002). Análise de conteúdo clássica: uma revisão. In M. W. Bauer & G. Gaskell (Org.), Pesquisa qualitativa com texto, imagem e som: um manual prático (pp.189-217). Petrópolis: Vozes.

Carey, W. B. (1999). Problems in diagnosing attention and activity. Pediatrics, 103(3),664-667.

Cattell, R. B. (1966). The screen test for the number of factors. Multivariate Behavioral Research, 1,141-161.

Cho, S. C., Kim, H. W., Kim, B. N., Shin, M. S., Yoo, H. J., Kim, J. W., et al. (2011). Are teacher ratings and parents ratings differently associated with children's intelligence and cognitive performance. Psychiatry Investigation, 8,15 -21.

Conners, C. K., Sitarenios, G., Parker, J. D. A., & Epstein, J. N. (1998). The revised Conners' Parent Rating Scale (CPRS-R): factor structure, reliability, and criterion validity. Journal of Abnormal Child Psychology, 26(4),257-268.

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52,281-302.

Davis, J. M., Cheung, S. F., Takahashi, T., Shinoda, H. & Lindstrom, W. A. (2011). Cross-national invariance of Attention-Deficit/Hyperactivity Disorder factors in Japanese an U. S. university students. Research in Developmental Disabilities, 32(6),2972-2980.

Delis, D. C., Jacobson, M., Bondi, M. W., Hamilton, J. M., & Salmon, D. P. (2003). The myth of testing construct validity using factor analysis or correlations with normal or mixed clinical populations: lessons from memory assessment. Journal of the International Neuropsychological Society, 9(6),936-946.

Dewey, D., Crawford, S. G., & Kaplan, B. J. (2003). Clinical importance of parent ratings of everyday cognitive abilities in children with learning and attentional problems. Journal of Learning Disabilities, 36,87-95.

Glascoe, F. P., Martin, E. D., & Humphrey, S. (1990). A comparative review of developmental screening tests. Pediatrics, 86(4),547-554.

Greenhill, L. L. (1998). Diagnosing attention-deficit/hyperactivity disorder in children. Journal of Clinical Psychiatry, 59(Suppl.7),31-41.

Guilford, J. P. (1956). Psychometric methods. New York: McGraw-Hill.

Henriques, R. (2001). Desigualdade racial no Brasil: evolução das condições de vida na década de 90(Texto para discussão 807). Rio de Janeiro: IPEEI/MPOG.

Lago, L. C. (1999). Desigualdade socioespacial e mobilidade residencial: a metrópole do Rio de Janeiro nos anos 80. In L. M. M. Bógus & L. C. Q. Ribeiro (Eds.), Cadernos metró-pole 2 (pp.49-189). São Paulo: Fapesp.

Larrabee, G. J. (2003). Lessons on measuring construct validity. Journal of the International Neuropsychological Society, 9(6)947-953.

Liu, X., Kurita, H., Guo, C., Tachimori, H., Ze, J., & Okawa, M. (2000). Behavioral and emotional problems in Chinese children: teacher reports for ages 6 to 11. The Journal of Child Psychology and Psychiatry, 41(2),253-260.

Luk, S. L., Leung, P. W., & Lee, P. L. M. (1988). Conners' teacher rating scale in chinese children in Hong Kong. The Journal of Child Psychology and Psychiatry, 29(2),165-174.

Mares, D., McLuckie, A., Schwartz, M., & Saini, M. (2007). Executive function impairments in children with attention-deficit hyperactivity disorder: do they differ between school and home environments. Canadian Journal of Psychiatry, 52(8),527-534.

Meisels, S. J. (1989). Can developmental screening tests identify children who are developmentally at risk? Pediatrics, 83,578-585.

Mitsis, E. M., McKay, K. E., Schulz, K. P., Newcorn, J. H., & Halperin, J. M. (2000). Parent-teacher concordance for DSM-IV attention-deficit/hyperactivity disorder in a clinic-referred sample. Journal of the American Academy of Child and Adolescent Psychiatry, 39(3),308-313.

Nunnally, J. O. (1978). Psychometric theory. New York: McGraw-Hill.

Pineda, D. A., Aguirre, D. C., Garcia, M. A., Lopera, F. J., Palacio, L. G., & Kamphaus, R. W. (2005). Validation of two rating scales for attention-deficit hyperactivity disorder diagnosis in Colombian children. Pediatric Neurology, 33(1),15-25.

Schatz, A. M., Ballantyne, A. O., & Trauner, D. A. (2001). Sensitivity and specificity of a computerized test of attention in the diagnosis of attention-deficit/hyperactivity disorder. Assessment, 8(4),357-365.

Schmidt, S. L., Snyder, T. J., Rouget, A. C., & Gray, E. (2000). Empirical analysis of the selective attention and associated behavior checklists of the aggregate neurobehavioral student health and educational review. Journal of Developmental & Behavioral Pediatrics, 21(3),165-171.

Tani, I., Okada, R., Ohnishi, M., Nakajima, S., & Tsujii, M. (2010). Japanese version of the ADHD-RS: an evaluation of its reliability and validity. Research in Developmental Disabilities, 31(6),1426-1433.

Tripp, G., Shaughency, E. A., & Clarke, B. (2006). Parent and teacher rating scales in the evaluation of attention-deficit hyperactivity disorder: contribution to diagnosis and differential diagnosis in clinically referred children. Journal of Developmental & Behavioral Pediatrics, 27(3),209-218.

Ulloa, R. E., Narváez, M. R., Arroyo, E., del Bosque, J., & de la Peña, F. (2009). Estudio de validez del cuestionario para trastorno por déficit de atención y trastorno de conducta. Actas Espanolas de Psiquiatria, 37(3):153-157.

United Nations Development Programme. (2000). Human development report. New York: Orford University Press.

Yao, K. N., Solanto, M. V., & Wender, E. H. (1988). Prevalence of hyperactivity among newly immigrated Chinese-American children. Journal of Developmental & Behavioral Pediatrics, 9(6),367-373.

Received on: 25/8/2010

Final version on: 3/2/2012

Approved on: 13/2/2012

Publication Dates

• Publication in this collection
  05 Dec 2012
• Date of issue
  Dec 2012

History

• Received
  25 Aug 2010
• Accepted
  13 Feb 2012
• Reviewed
  03 Feb 2012