Validity and reliability of a teacher ́s scale developed in Brazil for assessment of hyperactive: impulsive behavior and inattention in children and adolescents

This study was designed to develop and assess the internal consistency and factor structure of a teacher’s scale for the assessment of hyperactive/impulsive and inattentive behavior in a sample of 221 children from different cities in the State of Rio de Janeiro, Brazil. Forty-five teachers rated the children. The internal consistency of the scale was evaluated using Cronbach’s alpha. An orthogonal varimax rotation that maximizes the variance of the squared loadings for each factor was used to find the simplest possible factor structure. There is no internal discrepancy in the item content of the scale. Exploratory factor analysis showed four primary factors (hyperactivity/impulsivity; inattention; social isolation; self-confidence) that are psychologically meaningful. When factor analysis was carried out there was no substantial difference from other studies when compared with the results of other western and oriental countries. It was concluded that this teacher’s scale can be a useful aid to clinicians in the identification of children with hyperactivity/impulsivity and inattention problems. Uniterms: Attention deficit disorder with hyperactivity. Mental disorders. Psychological assessment.

Previous studies have shown that physicians and psychologists who rely only on their clinical judgment may experience difficulties in identifying children with behavioral problems (Barkely, 1990;Schmidt, Snyder, Rouget & Gray, 2000).The use of checklists completed by teachers or parents allows for a greater level of accuracy, even when considering the limitations of the instruments (Greenhill, 1998;Schatz, Ballantyne & Trauner, 2001;Ulloa, Narváez, Arroyo, del Bosque & de la Peña, 2009).Studies have demonstrated that in some cases, behavior rating scales are even more effective in classifying children with learning and cognitive disorders than traditional psychological tests.These data were obtained by Dewey, Crawford and Kaplan (2003) in a study with 159 children using a parent´s scale (Parent Ratings of Everyday Cognitive and Academic Abilities) and psychometrics measures, such as the Woodcock-Johnson Psycho-Educational Battery-Revised, the Bruininks-Oseretsky Test of Motor Proficiency-Short Form, the Wechsler Intelligence Scale for Children, 3 rd edition (Vocabulary and Block Design subtests), and the Developmental Test of Visual-Motor Integration.The results demonstrated that the scale was more efficient in classifying children with problems (reading disabilities, Attention Deficit/Hyperactivity Disorder -ADHD and combined) than the psychometric measures alone.These limitations of neuropsychological tests, specifically in the diagnosis of ADHD, were also recently highlighted in an article by Andrade and Flores-Mendonza (2010), which confirms the usefulness of scales for parents and teachers.
The presence of primary inattention disorder and/or hyperactivity/impulsivity is a diagnostic criterion for ADHD, the most common childhood psychiatric disorder, affecting 5% of the school population.According to the Diagnostic and Statistical Manual of Mental Disorders -4 th Edition Text Revision -DSM-IV-RT (American Psychiatric Association, 2000) these behavioral changes must be present in more than one venue frequented by children, and their behavior at home and at school is commonly rated by parents and teacher.As suggested by Mares, McLuckie, Schwartz and Saini (2007) and Tripp, Schaughency and Clarke (2006) it is essential to collect data from these two sources, considering that parents and teachers emphasize different aspects during the evaluation.Such differences are commonly described in the literature (Mitsis, McKay, Schultz, Newcorn & Halperin, 2000).Cho et al. (2011), in a recent study with the aim of comparing data obtained from a scale for teachers and parents, on the assessment of cognitive performance in children, confirm these data.These authors found that teachers are more skilled in assessing hyperactive/impulsive behavior, which facilitates the identification of children with low cognitive performance.This facility by professionals is probably related to the fact that teachers know about child development and deal with a large number of children.This finding does not exclude the need for data collection though scales filled out by parents.The study makes it clear that parents and teachers point out different but important aspects of child behavior.
With further regard to the use of scales, an American research with 401 pediatricians and family practice physicians, in a prospective cohort study of 22,059 children from 4 to 14 years, showed that 53.5% of them used school reports to diagnose attentional or hyperactivity problems (Carey, 1999).Although there was no data available as regards the use of questionnaires by Brazilian physicians and psychologists, it is believed that the use of rating scales to identify children with behavioral and learning problems needs to be emphasized.The socio-economic aspects of Brazil make the development of teacher behavior rating scales particularly relevant.The low educational level of the population is still an important social problem.At the end of the twentieth century, the average Brazilian over 25 had only 6 years of formal education.In 2001, 13.6% of the population over 15 years of age was illiterate (Henriques, 2001).Considering these facts, it is common to find parents who have a low educational level or who are even illiterate, and for these people written scales may not be a good way of providing information.Therefore, it was decided to develop a scale to assess hyperactive/impulsive behavior and inattention to be filled out by teachers.The present study, in particular, was conducted to provide information about the psychometric properties of this scale, including the analysis of its reliability and validity, which are essential characteristics of an instrument that is to be used clinically (Glascoe, Martin & Humphrey, 1990;Meisels, 1989).
Three specific objectives and respective hypotheses were investigated.First, the checklist was Estudos de Psicologia I Campinas I 29(4) I 481-488 I outubro -dezembro 2012 verified to see whether presented high internal reliability.This involves the relationship of the items comprising the scale.Second, an evaluation was made to establish whether the checklist items could be expressed by a small number of factors.To be valid, the resulting factors of the rating scale must be psychologically interpretable and meaningful (Nunnally, 1978).Finally we wished to know whether the factors derived from the exploratory factor analysis differed from those of other studies conducted in different western and oriental countries.

Method Participants
The participants were 45 teachers who rated 221 scales from children drawn from elementary schools located in four different cities in the State of Rio de Janeiro.The sample consisted of 108 boys (mean age=8.91 years, SD=2.08 years, minimum=6 years, maximum=15 years) and 113 girls (mean age=8.88years, SD=2.41 years, minimum=6 years, maximum=15 years).
Considering socio-economic characteristics (particularly educational level), a heterogeneous population can be found in the State of Rio de Janeiro (Lago, 1999).The Human Development Index (HDI), from the United Nations Development Program (2000), was used to characterize the cities.The HDI is a composite index and three variables are considered for its composite: life expectancy at birth, educational attainment and Gross National Income (GNI) per capita (PPP US$).Here, the HDI of each city and the number of participants are described (Table 1).
The checklist was answered by the teacher who spent most of the time with the child in the classroom, and for each child only one teacher answered the questionnaire.It should be mentioned that in the Brazilian educational system, during the first segment of elementary school, in which children are between 6 and 10 years old, schoolchildren normally have only one teacher, whereas those in the second segment (aged between 11 and 14) have more than one teacher.The participation of the child and the teacher were voluntary.Parents and school principals signed terms of informed consent as regards the voluntary participation of the children in the research.This study was approved by the Ethics Committees of the Universidade do Estado do Rio de Janeiro (UERJ), Protocol 004.2004, and all participants signed an informed consent before their inclusion in the sample.

Measures
This scale was developed from data derived from academic meetings held with clinicians experienced in the area of infant diagnosis.Data published in national and international articles and diagnostic criteria of DSM-IV RT (American Psychiatric Association, 2000) for ADHD were taken into account.
The scale is divided into two parts: qualitative and quantitative.The qualitative part is intended to describe the child's main problems, report school performance, describe whether the child needs or receives special care or help to develop his/her school activities, and give information about the child's family.Considering this part, the teacher has the opportunity to say whether the child has attention or behavior problems, and to describe his/her academic performance.The quantitative part is composed of 58 questions with information about behavior in the classroom, peer group, and disruptive behavior.These questions are rated on an oral assessment scale ranging from 1 to 3 reflecting problem prevalence (1=not true; 2=somewhat or sometimes true; 3=very true or often true).

Analytical methods
The analysis of the qualitative part of the scale was performed by means of the content analysis technique, through which the observations described by the teacher were classified into learning complaint, attention, behavior or individual without complaint.We used the quantitative approach to content analysis, Loading whereby the content of the information was analyzed through systematic and objective rules of categorization.For each of the three questions concerning the qualitative part of the scale, categories were established and analyzed in the light of referential coding.Each unit of text could be classified according to the categories raised (Bauer, 2002).
The internal consistency of the 58 scorable-item scale was evaluated using Cronbach's alpha.Alpha is a correlation coefficient and ranges in value from -1 to 1.If Cronbach's alpha drops below 0.7, the hypothesis of internal reliability cannot be accepted (Barret & Kline, 1981;Cattell, 1966).To evaluate how each individual item affects the reliability of the scale, alpha was calculated removing each of the items of the scale.
Raw data from the questionnaire were factor analyzed.To find the simplest possible factor structure, an orthogonal varimax rotation that maximizes the variance of the squared loadings for each factor was used.Only factors that account for variances greater than 1 (eigenvalues [ ] greater than 1) were included.With regard to the factor loadings, only the most highly correlated variables (greater than 0.50) for each factor (Table 2) were considered.
To verify the relationship between quantitative and qualitative parts of the scale, standard scores were constructed for each factor, considering gender and age.

Internal reliability
For the 221 participants, the checklist presented high internal reliability as indicated by Cronbach's alpha coefficients ( = 0.96).Elimination of any of the items from the scale caused little change in a.

Internal reliability
For internal consistency analyses, the samples should contain at least 100 subjects (Guilford, 1956;Nunnally, 1978).Barret and Kline (1981) found that a minimum of 100 subjects and a variable-subject ratio of 2:1 are needed for a reliable factor analysis.Thus, the sample of 221 subjects used, with a variable-subject ratio of at least 2:1 supports the stability of the reliability analysis.
The components of the scale can be viewed as an internally reliable instrument for obtaining quantified information from teachers.Empirical analysis demonstrated that there is no internal discrepancy in the item content of the checklist.High internal consistency is a prerequisite of validity (Guilford, 1956;Nunnally, 1978).
Factor analyses Arrindel and Endle (1985) claimed that stable factors required the sample to be approximately 20 times larger than the number of factors.According to these authors, variable-subject ratio is not important when compared with the ratio of sample size to factors.Therefore if only reliable factors are to be considered, a conservative approach requires that the number of factors in the study should never be greater than 11.As the study showed four factors, the hypothesis that the factors are reliable is confirmed.
In support of the hypothesis of construct validity, the number of items of each checklist could be expressed by a smaller number of factors, which are?/were psychologically meaningful.

Correlation between quantitative and qualitative questions
The comparison between subjects with and without complaints, identified through the analysis of the qualitative questions of the scale, demonstrated that the individuals with complaints presented higher scores than those without complaints, as described by the teacher.These differences were statistically significant for factors 1, 2 e 3. In conclusion, the present checklist can be considered a valid instrument in helping to identify children with learning and behavior problems.

Comparison with other western and oriental studies
When the primary factors were considered there was no substantial difference from studies carried out in other western countries (Conners, Sitarenios, Parker & Epstein, 1998;Pineda et al., 2005) as well as in many oriental countries (Aramaki & Uno, 2004;Liu et al., 2000;Luk, Leung & Lee, 1988 ;Yao, Solanto & Wender, 1988).These studies support ADHD as an universally valid syndrome.An important example occurred in Japan, where studies on ADHD symptoms were uncommon before 2005.In that country, the results of a large scale survey of school teachers showed 6.3% of children with developmental disorder, and 2.5% of them with ADHD (Tani, Okada, Ohnishi, Nakajima & Tsujii, 2010).A checklist to be completed by parents and teachers has been validated for use in assessment of Japanese children (Davis, Cheung, Takahashi, Shinoda & Lindstrom, 2011) and ADHD has become the focus of many researches.These studies support that ADHD symptoms are comparable across cultures.Inattentiveness and impulsivity can be viewed as factors that emerge in different cultures when behavioral problems of children are analyzed by instruments conveniently adapted to the local language and culture.

Limitations of the study and directions for future research
In this study, the question/issue of whether the scale can be reliably used as successive measures (testretest reliability) of child behavior was not evaluated.While construct validity was demonstrated, there are also other types of validity, such as predictive validity.Therefore, further study is necessary to evaluate other types of validity and reliability.
As previous investigators have reported that parents are a useful source of information on children (Barkley, 1990;Connors et al., 1998;Schatz et al., 2001), systematic oral information from parents with a low level of formal education could be collected by clinicians and the results compared with the written information provided by teachers.
Although the primary method used by psychologists to test the validity of constructs has been factor analysis (Cronbach & Meehl, 1955), recent studies have addressed the problems that can arise from using shared-variance techniques to evaluate construct validity.In particular, it has been proposed that a correlation between two or more variables should not represent a unitary cognitive construct (Delis, Jacobson, Bondi, Hamiton & Salmon, 2003).However, previous studies have demonstrated the stability of the conceptual relationship across a number of homogeneous clinical populations, and thus, confirmatory factor analysis can extend construct validity analysis based on sharedvariance procedures (Larrabee, 2001).Therefore, confirmatory factor analysis studies need to be conducted on specific cognitive constructs in selected homogeneous patient populations.

Conclusion
The components of the scale are quantifiable.The checklist presents high internal reliability and can be reduced to a smaller number of factors that are psychologically meaningful.Thus, it will be possible to develop abbreviated forms of this scale.Rating scales do not have to be long instruments; they have to be able to evaluate the interest construct.In fact, clinical scales can be refined through factor-analytic techniques.
Considering the need to improve the efficiency of our educational system, especially for those children with treatable disabilities, it is suggested that the results of this study may contribute to integrate the activities of pediatricians, psychologists and teachers, thereby improving the actions that are directed towards children with behavioral and educational problems.This checklist may be a useful aid in the identification of children with behavioral and attention problems.
HDI: Human Development Index.