Evaluation of the Pediatric Symptom Checklist as a screening tool for the identification of emotional and psychosocial problems

Evaluación del uso de la Lista de Síntomas Pediátricos como instrumento de rastreo para identificar problemas emocionales y psicosociales

Abstracts

OBJECTIVE

To investigate the Brazilian version of Pediatric Symptom Checklist (PSC) as a screening tool to identify psychosocial and emotional problems in schoolchildren from six to 12 years old.

METHODS

Diagnostic test conducted in a public school of Curitiba, Paraná (Southern Brazil), to evaluate the PSC accuracy and consistency, considering the Child Behavior Checklist (CBCL) as the gold standard. Among 415 parents invited for the study, 145 responded to both PSC and CBCL. The results of the two instruments were compared. PSC and CBCL were considered positive if scores ≥28 and >70 respectively.

RESULTS

Among the 145 cases, 49 (33.8%) were positive for both PSC and CBCL. The ROC curve showed the PSC score of 21 as the best cutoff point for screening psychosocial and emotional problems, with a sensitivity of 96.8% and a specificity of 86.7%. Regarding the reference cutoff (score ≥28 points), the sensitivity was 64.5% and the specificity, 100.0%, similar to those found in the original version of the tool.

CONCLUSIONS

The Portuguese version of PSC was effective for early identification of emotional and/or psychosocial problems in a schoolchildren group and may be useful for pediatricians.

child; triage; mental disorders

OBJETIVO

Investigar la Lista de Síntomas Pediátricos (LSP) como instrumento de rastreo para identificar problemas emocionales y psicosociales en escolares de 6 a 12 años.

MÉTODOS

Estudio de tipo prueba diagnóstica, realizado en una escuela pública de Curitiba, Paraná (Brasil), para evaluar la precisión y la consistencia de la LSP, considerando el Child Behavior Checklist (CBCL) como patrón oro. De los 415 padres invitados a participar del estudio, 145 contestaron a la LSP y al CBCL, constituyendo la muestra, y los resultados de los dos instrumentos fueron comparados. Se consideró como positivo el escore de LSP ≥28 y del CBCL >70 puntos.

RESULTADOS

De los 145 cuestionarios, 49 (33,8%) tanto de la LSP como del CBCL se mostraron positivos. La curva ROC indicó 21 como el mejor punto de corte para rastreo de problemas emocionales y psicosociales, con sensibilidad del 96,8% y especificidad del 86,7%. En el punto de corte de referencia (≥28), se observó la sensibilidad del 64,5% y especificidad del 100%, similares a las en contradas en la versión americana del instrumento.

CONCLUSIONES

La LSP se mostró eficaz para la identificación temprana de problemas emocionales y/o psicosociales en un grupo de escolares, pudiendo ser una herramienta útil al pediatra.

niño; rastreo; trastornos mentales

OBJETIVO

Investigar a Lista de Sintomas Pediátricos (LSP) como instrumento de triagem para identificar problemas emocionais e psicossociais em escolares de seis a 12 anos.

MÉTODOS

Estudo do tipo teste diagnóstico, realizado em uma escola pública de Curitiba, Paraná, para avaliar a acurácia e a consistência da LSP, considerando-se o Child Behavior Checklist (CBCL) como padrão-ouro. Dos 415 pais convidados a participar do estudo, 145 responderam à LSP e ao CBCL, constituindo a amostra. Os resultados dos dois instrumentos foram comparados. Considerou-se como positivo o escore da LSP ≥28 e o do CBCL >70 pontos.

RESULTADOS

Dos 145 questionários, 49 (33,8%) tanto da LSP como do CBCL mostraram-se positivos. A curva ROC indicou 21 como o melhor ponto de corte para triagem de problemas emocionais e psicossociais, com sensibilidade de 96,8% e especificidade de 86,7%. No ponto de corte de referência (≥28), observou-se sensibilidade de 64,5% e especificidade de 100,0%, similares às encontradas na versão americana do instrumento.

CONCLUSÕES

A LSP mostrou-se eficaz para a identifição precoce de problemas emocionais e/ou psicossociais em um grupo de escolares, podendo ser uma ferramenta útil ao pediatra.

criança; triagem; transtornos mentais

Introduction

The identification of psychosocial and emotional problems should be part of pediatric practice. Studies have documented the increased prevalence of mental disorders (MD) in children and adolescents. In the United States, epidemiological studies indicate a prevalence of 17 to 27% of some type of MD in this age range^{( 1 , 2 )}. In Taubaté, state of São Paulo, a study with the same group found a prevalence of 12.5% individuals with at least one MD, and, in poor urban areas, where there are sociocultural risk factors, the rate was 13.7%^{( 3 )}.

The recognition, intervention, and preventive care of mental disorders in children and adolescents help define and reduce the effects on personality, not yet fully formed, contributing to a healthy development. The lack of early recognition, the persistence and worsening of symptoms in the later stages of development can cause damage not only to the childâ€(tm)s daily life but also to the family and the society.

The relationship between genetic and environmental factors in the origin of MDs is dynamic and cumulative in its capacity to influence development and change their future conditions. The negative result of situations experienced in childhood, the genetic, biologic, psychological, and environmental risk factors, as well as the complex interactions between them, can lead to the appearance of one or more MDs. Among the most prevalent are depressive disorder (DD), anxiety disorder (AD), oppositional defiant disorder (ODD), and attention deficit disorder/hyperactivity (ADHD), with frequent presence of more than one disorder in the same individual^{( 4 )}. These inter-relations may hinder diagnosis and, consequently, an early intervention by health professionals who are not familiar with these clinical features. However, the indicative of a mental disorder in childhood and adolescence should not be confused with the normal characteristics of developmental stages, which may present similar pathological symptoms.

The high prevalence of MD in childhood and adolescence demands the use of assistance techniques to identify the need for psychological and/or psychiatric assessment and treatment. Most of these professionals, many of them pediatricians, do not feel adequately trained or do not have time for the identification and subsequent referral of children and/or adolescents with indicatives of MD. An alternative is the use of rating scales already validated, such as the Child Behavior Checklist (CBCL)^{( 5 , 6 )} and the Pediatric Symptom Checklist (PSC)^{( 7 )}.

The present study evaluated the accuracy of the Brazilian version of the PSC - Pediatric Symptom Checklist - as a screening instrument for emotional and psychosocial problems in a population of schoolchildren aged from 6 to 12 years, comparing the results with those obtained by the CBCL.

Method

This was a diagnostic test study to assess the accuracy of PSC in the diagnosis of emotional and/or psychosocial problems, considering the CBCL as the gold standard. The PSC is a screening instrument for emotional and/or psychosocial problems for children and adolescents from 6 to 16 years. It is a questionnaire with 35 items, with a fast fill-in process, which reflects the impression of parents on the behavior and development of the child. The cutoff point for positive risk must be ≥28 points^{( 8 )}. The CBCL, in turn, is a questionnaire also responded by parents, that assesses emotional and behavior problems, being considered a standard instrument in child and adolescent psychology.

Initially, all parents of children who met the inclusion criteria were invited: students of both genders, aged from 6 to 12 years of age, enrolled in a public school in Curitiba, state of Paraná (n=601). A total of 415 agreed to participate and responded to the first assessment instrument, the PSC, reporting the results to the participants. These 415 parents were asked to respond to a second assessment instrument, the CBCL. Among these, 145 agreed, constituting, thus, the study sample. The results of the administration of the two instruments in this sample were compared, considering the gold standard set by the CBCL. We sought to further evaluate the internal consistency of the PSC, as well as its sensitivity and specificity for the diagnosis of screening of emotional and/or psychosocial problems.

The PSC is a screening tool to identify children or adolescents with emotional and/or psychosocial problems. It consists of 35 items, valid for the age group from 6 to 16 years. It is easy to administer, to understand, and to interpret the results. The score attributed by the respondent, who indicates how often the situation proposed in the question occurs, is zero for "never" one for "sometimes" and two for "often". The cutoff point established for the American score is ≥28 points, in which case the result is considered positive, i.e., the respondent would have the indication to be referred for a mental health evaluation (Chart 1).

Chart 1
Pediatric Symptom Checklist

The CBCL was adapted and translated to the Portuguese language by Bordin, Mari and Caeiro, in 1995^{( 6 )}. It is a comprehensive instrument considered as gold standard, answered by the parents. It reports the skills and problems in children or adolescents between 6 and 18 years and provides an analysis of the emotional, social, and behavioral profile of the individual, divided in two parts. The first refers to the Social Competence, i.e., involvement and performance of the child/adolescent in sports activities, games, hobbies, jobs, and daily tasks, whose scores grow according to the greater social competence of the assessed. The second part refers to the assessment of the existence of emotional and behavioral problems. It consists of 118 items and the answers of the individual are "not true/absent" (equivalent to score 0); "somewhat/ sometimes true" (equivalent to score 1) and "often true" (equivalent to score 2). The sum of the scores is converted into T-scores, according to appropriate analyzes for age and gender. In this case, it is presented in scale and syndromes, that is, problems that usually occur together. A T-score ≥70 is considered clinical; between 64-69, borderline; and <63, normal. For scales of internalizing and externalizing problems, a T-score ≤60 is considered normal, between 60-63, borderline, and >63 it is considered clinical.

In statistical analysis were applied: a) Pearsonâ€(tm)s chi-square test with Yates correction, to assess the frequency of responses obtained in each category (normal, borderline, and clinical) in the two assessment instruments; b) Cronbachâ€(tm)s coefficient to assess the internal consistency of the PSC; c) Kappa concordance coefficient to estimate the degree of agreement of the results of the two evaluation instruments, and d) ROC curves, to evaluate the performance of the PSC as a screening tool for the diagnosis of emotional and/or psychosocial problems and the cutoff, i.e., the score value with higher sensitivity and specificity^{( 7 )}. The sample was calculated in order to verify the degree of agreement between the two assessment tools for the diagnosis of screening for emotional and/or psychosocial problems to compare the frequency obtained in the sample with the estimated prevalence of mental disorders in the population (13-27%)^{( 1 - 3 )}. The level of significance was established at 5%, the type II error at 10%, and the power test at 90%, and there was a minimum calculation of 120 subjects^{( 7 )}.

This study was approved by the Research Ethics Committee of Hospital de Clínicas da Universidade Federal do Paraná. Parents or guardians signed an informed consent form.

Results

In the analysis of internal consistency of the PSC, a high Cronbachâ€(tm)s coefficient (0.89) was obtained, which showed its quality and reliability.

Among 415 parents and/or guardians who responded only to the PSC, 58 (14%) presented a positive result, indicating risk for emotional and/or psychosocial problems. Among the 145 cases in which the evaluation by the PSC and the CBCL were conducted, the frequency was of 33.8% (49 cases). The frequency of PSC ≥28 has higher in males (p<0.01) and similar between the age groups (p=0.49).

In the CBCL, it was observed that 62 children/adolescents (42.7%) had a score greater than 70; 23 (15.9%), between 65-69; and 60 (41.4%), a score <60. When studying in detail the 62 children (42.7%) with score rated as clinical by CBCL, it was observed that, among the most common syndromes were anxiety/depression, somatic complaints, and aggressive behavior (Table 1). Among these, 29 presented externalizing and internalizing problems; 17, only externalizing; 13, only internalizing; and three, other problems.

Table 1
Frequency distribution of syndromes presented by the Child Behavior Checklist

Syndromes	Clinic (>70)		Borderline (65–69)		Normal (< 65)
	n	%	n	%	n	%
Anxiety/depression	25	17.2	18	12.4	102	70.3
Isolation/depression	17	11.7	17	11.7	111	76.5
Somatic complaints	24	16.5	21	14.5	100	69
Social problems	21	14.5	21	14.5	103	71
Thinking Problems	18	12.4	11	7.6	116	80
Attention Problems	04	2.7	15	10.3	126	86.9
Breaking the rules	16	11	10	6.9	119	82
Aggressive Behavior	35	24.1	29	20	81	55.9

There was good agreement between the responses of the PSC and the CBCL for results classified as normal and clinical (Kappa=0.69). Borderline cases were excluded from this analysis (Table 2).

Table 2
Concordance between the values of the Pediatric Symptom Checklist and the Child Behavior Checklist, for "normal" and "clinic" results

		CBCL
	score	≥70	< 65	Total
PSC	≥28 < 28 Total	43 (35.2%) 19 (15.6%) 62 (50.8%)	0 (0%) 60 (49.2%) 60 (49.2%)	43 (35.2%) 79 (64.7%) 122 (100%)

• The borderline cases of this analysis were excluded to compare only clinical and normal cases of the LSP and the CBCL. Kappa Coefficient=0.69

Considering the frequencies obtained, the result of the PSC showed a sensitivity of 64.5%, specificity of 100%, positive predictive value (PPV) of 100%, negative predictive value of (NPV) of 75.5%, accuracy of 84.4%, false positive rate (FP) of zero and false negative of (FN) 24% for the diagnosis of emotional and psychosocial problems established by the CBCL, with a cutoff ≥28 points established in the literature. There was a strong correlation between the total score of the PSC and the CBCL (r=0.84), with no significant difference regarding gender or age range.

By observing the gender distribution in the sample, it was found that in males, mean scores for externalizing problems were greater than in females (p<0.01). This difference was due to the breaking rules syndrome and aggressive behavior, more common in males.

In the ROC curve, considering as a dependent variable the total clinical versus normal problems, and as an independent variable, the score of the PSC, it was obtained 21 as the cutoff score, with sensitivity of 96.8% and specificity of 86.7%. The area under the curve was 0.96, indicating high sensitivity and specificity for identifying psychosocial and emotional problems (Graph 1).

Graph 1
ROC curve, considering the total of normal and clinical problems

Discussion

In developed countries, one in ten children presents at least one MD. Pediatricians should be prepared and alert to detect emotional problems and/or psychosocial interventions, since early interventions and treatment may improve outcome, prognosis, and damages to the child, the family and the society^{( 8 , 9 )}. A study showed that over 50% of children with MD in school age already demonstrated characteristics of the problems in the preschool period^{( 10 )}. Similarly, in a retrospective study on antisocial behavior in adolescence^{( 11 )}, it was noted that adolescents with problems already presented psychological signs at 2-3 years old, demonstrating, thus, that characteristic symptoms of internalizing and externalizing problems may be often predictive^{( 12 , 13 )}. An increase in referrals to mental health professionals was effective in reducing and improving symptoms as well as in the score of the PSC in a second reassessment^{( 14 )}.

Among the 415 parents who responded only to the first assessment instrument (PSC), 58 (14%) presented positive risk for emotional or psychosocial problems. In the evaluation of the 145 children and/or adolescents, it was observed that the PSC was positive in 33.8%. Among the 270 who did not agree in answering the CBCL, only nine (3.3%) had a PSC score ≥28 points, i.e., risk for emotional and/or psychosocial problems. The remaining 261 presented a score <28 points (97.7%). The frequency of 14% obtained among the 415 participants seems to reflect more faithfully the prevalence of emotional and/or psychosocial problems pointed in the literature, since the frequency of 33.8% obtained among the 145 parents who agreed to responding also to the CBCL may be overestimated because of greater adherence to the second stage of the study, as a result of the positive outcome of the PSC applied previously.

The frequency observed in this study is similar to that of the U.S. population^{( 15 )}, of 13%, on a broad approach to the pediatric population in 44 U.S. states, from Porto Rico, and four Canadian provinces. In a Hispanic population^{( 16 )} living in the state of Texas, there was a similar prevalence, of 16.5% of positive PSC. The results of the PSC which indicate the degree of reliability and validity in relation to the CBCL, used in this study as the gold-standard, are very similar to those found in studies conducted in other countries^{( 2 , 15 - 17 )}.

The questions chosen to compose the PSC reflect emotional and behavioral characteristics that are important for the identification of these problems in schoolchildren^{( 16 )}. By observing the frequency of responses to the grouping of questions (1-3, 10, 11, 13, 15, 19, 20-22, 27 and 30; Chart 1), the pediatricians should consider, in their assessment, characteristics of internalizing problems, i.e., children or adolescents with signs of anxiety, isolation, depression, and somatic complaints. Considering that younger children have more difficult to identify, describe, and demonstrate their feelings, due to emotional immaturity, the pediatricianâ€(tm)s attention should be redoubled in this age group.

Questions 12, 16, 17, 25, 26, 29, 31, 32-35 (Chart 1) are characteristic of externalizing problems, such as aggression and rule breaking. Not following the rules is a behavior that is part of the development of children, especially from 2-3 years of age and in adolescence. However, when this kind of behavior occurs often, consistently, and with damages to adaptive functioning, it may indicate oppositional defiant disorder.

Both in the 415 participants that responded only to the PSC and the 145 who responded to both instruments, it was observed a predominance of positivity in the male gender. Another study assessed a population of 21,065 children and adolescents and showed scores with positive responses in significantly higher in males compared to females (17 and 9% for schoolchildren and 12 and 9% for preschoolers)^{( 15 )}.

The results of the PSC showed good internal consistency and reliability of measurement. Considering the outcome of risk of emotional and psychosocial problems established by the CBCL, the ROC curve showed that the cutoff for this population was 21 points, with a sensitivity of 96.8% and specificity of 86.7%. When observing how the curve would behave with the cutoff point proposed for the U.S. population^{( 8 , 15 , 18 )} (28 points), a sensitivity of 64.5% and specificity of 100% were found. This reveals that the Brazilian sample behaves similarly to the U.S. sample, when considering the cutoff point of 28 points. In a study in the Netherlands^{( 2 )}, a cutoff of 22 points was proposed, with sensibility of 71.7% and specificity of 93% for the outcome risk of emotional and psychosocial problems. The choice of the best cutoff in PSC^{( 18 )} considers the ability to detect the largest number of children and adolescents with indicatives of MD, in order to identify those who truly need evaluation and/or treatment by mental health professionals. High cutoffs select a smaller number of people, probably covering the most affected, which, in fact, require specialized evaluation. Lower cutoffs select a larger number of subjects, and among them, those with little or no real problems, creating a great demand for specialized assessments and perhaps unnecessary concern for the family.

Thus, the identification of children and adolescents with MDs cannot be restricted to a single assessment instrument and to numeric accuracy. One should not oversimplify this complex task. The role of the pediatrician is essential, both in clinical assessment and in the assessment of the results of the screening instrument, in this case, the PSC. This assessment may help the professional to decide for the orientation of the family and the next reassessment or, if not sure or capable, he may refer the family to an appointment with a specialist. In this sample, a PSC with a score ≥28 presented specificity of 100% for emotional and psychosocial problems, being essential the referral for specialized evaluation. Between 21 and 27 points, there are cases in which the clinical evaluation is crucial to decide for the referral or close observation by the pediatrician.

It is the role of the pediatrician, a professional who has a great bond with families, to know how to recognize the different MDs for early interventions. The increasing emphasis on productivity and profitability adds more pressure on this professional, further limiting attention to the emotional and psychosocial problems. The use of screening tools for MDs as part of routine health care of children and adolescents may facilitate recognition and early referral, as well as information on the clinical symptoms of physical and organic origin. The PSC is a screening test for children and adolescents aged from zero to 16 years, of easy administration, understanding, and interpretation, besides the great degree of respect, reliability, and validity in relation to the CBCL (gold standard). The pediatrician who already uses other screening tools will find it easy to handle it. Parents can answer the questions while they wait for the pediatric consultation of their children. The PSC is a first step for screening MDs in childhood. The second is up to the pediatrician, together with parents, so that a final decision regarding the referral is made.

Acknowledgements

To the board of Direction, Coordination, Faculty Members and, in particular, to parents and families of students of Escola Estadual Aline Picheth, Curitiba, state of Paraná.

References

• ¹
  Stoppelbein L, Greening L, Jordan SS, Elkin TD, Moll G, Pullen J. Factor analysis of the Pediatric Symptom Checklist with a chronically ill pediatric population. J Dev Behav Pediatr 2005;26:349-55.
• ²
  Reijneveld SA, Vogels AG, Hoekstra F, Crone MR. Use of the Pediatric Sympton Checklist for the detection of psychosocial problems in preventive child healthcare. BMC Public Health 2006;6:197.
• ³
  Goodman R, Neves dos Santos D, Robatto Nunes AP, Pereira de Miranda D, Fleitlich-Bilyk B, Almeida Filho N. The Ilha de Maré study: a survey of child mental health problems in a predominantly African-Brazilian rural community. Soc Psychiatry Psychiatr Epidemiol 2005;40:11-7.
• ⁴
  Canino G, Shrout PE, Rubio-Stipec M, Bird HR, Bravo M, Ramirez R et al. The DSM-IV rates of child and adolescent disorders in Puerto Rico: prevalence, correlates, service use, and the effects of impairment. Arch Gen Psychiatry 2004;61:85-93.
• ⁵
  Achenbach TM. Manual for the Child Behavior Checklist/4-18 and 1991 profile. Burlington: University of Vermont; 1991.
• ⁶
  Bordin IA, Mari JJ, Caeiro MF. Validação da versão brasileira do Child Behavior Checklist (CBCL) (Inventário de Comportamentos da Infância e Adolescência): dados preliminares. Rev ABP-APAL 1995;17:55-66.
• ⁷
  Altman DG. Practical statistics for medical research. Londres: Chapman & Hall/CRC; 1999.
• ⁸
  Jellinek MS, Murphy JM, Burns BJ. Brief psychosocial screening in outpatient pediatric practice. J Pediatr 1986;109:371-8.
• ⁹
  Hacker KA, Myagmarjav E, Harris V, Suglia SF, Weidner D, Link D. Mental health screening in pediatric practice: factors related to positive screens and the contribution of parental/personal concern. Pediatrics 2006;118:1896-906.
• ¹⁰
  Lavigne JV, Arend R, Rosenbaum D, Binns HJ, Christoffel KK, Burns A et al. Mental health service use among young children receiving pediatric primary care. J Am Acad Child Adolesc Psychiatry 1998;37:1175-83.
• ¹¹
  Moffitt TE, Caspi A. Childhood predictors differentiate life-course persistent and adolescence-limited antisocial pathways among males and females. Dev Psychopathol 2001;13:355-75.
• ¹²
  Williams J, Klinepeter K, Palmes G, Pulley A, Foy JM. Diagnosis and treatment of behavioral health disorders in pediatric practice. Pediatrics 2004;114:601-6.
• ¹³
  Mesman J, Koot HM. Early preschool predictors of preadolescent internalizing and externalizing DSM-IV diagnoses. J Am Acad Child Adolesc Psychiatry 2001;40:1029-36.
• ¹⁴
  Bird HR, Canino G, Rubio-Stipec M, Gould MS, Ribera J, Sesman M et al. Estimates of the prevalence of child maladjustment in a community survey in Puerto Rico. The use of combined measures. Arch Gen Psychiatry 1988;45:1120-6.
• ¹⁵
  Jellinek MS, Murphy JM, Little M, Pagano ME, Comer DM, Kelleher KJ. Use of the Pediatric Symptom Checklist to screen for psychosocial problems in pediatric primary care: a national feasibility study. Arch Pediatr Adolesc Med 1999;153:254-60.
• ¹⁶
  Leiner MA, Balcazar H, Straus DC, Shirsat P, Handal G. Screening Mexicans for psychosocial and behavioral problems during pediatric consultation. Rev Invest Clin 2007;59:116-23.
• ¹⁷
  Thun-Hohenstein L, Herzog S. The predictive value of the pediatric symptom checklist in 5-year-old Austrian children. Eur J Pediatr 2008:167:323-9.
• ¹⁸
  Murphy JM, Arnett HL, Bishop SJ, Jellinek MS, Reede JY. Screening for psychosocial dysfunction in pediatric practice. A naturalistic study of the Pediatric Symptom Checklist. Clin Pediatr (Phila) 1992;31:660-7.

• Instituição: Universidade Federal do Paraná (UFPR), Curitiba, PR, Brasil

Publication Dates

• Publication in this collection
  Sept 2013

History

• Received
  09 Oct 2012
• Accepted
  29 Apr 2013