Systematic review on the use and description of measures to evaluate psychotherapeutic interventions with children and adolescents in Brazil

INTRODUCTION
Psychotherapeutic interventions in childhood and adolescence are recognized as a mental treatment and as a tool to reduce psychological disorders in youth and adulthood. Therefore, there is an increasing concern about evidence of effectiveness of mental disorder interventions and adequacy of measurement. The objectives of this systematic review were to investigate predominant research designs and instruments used in Brazilian studies describing psychotherapeutic interventions with children and adolescents and to examine how these instruments are described regarding evidence of validity and reliability.


METHOD
Five databases were surveyed, and for each one two independent judges performed the selection of records and applied the inclusion and exclusion criteria.


RESULTS
The final database comprised 28 papers, in which 92 instruments were referred. Fifty-seven instruments cited did not have descriptions of evidence of validity for the Brazilian population; for 31 instruments, validity evidence was mentioned, but the study did not detail which validity parameter was used; three studies described content validity evidence for their instruments. Furthermore, information about reliability was described for only two instruments.


CONCLUSION
A lack of studies in the field of child and adolescent psychotherapy was found in Brazil. There is a significant need for the field to attend both the psychometric properties and the quality of description of research instruments. The scientific production of studies focused on the evaluation of psychotherapeutic interventions may promote evidence-based psychotherapy and justify the offer of mental treatment in different contexts.


Introduction
Most psychological disorders that are typically discovered when maturity is reached have an early beginning in childhood or adolescence. 1Mental disorders are prevalent in up to 20% of children and adolescents worldwide, and 50% of mental disorders affecting adults begin during childhood. 2The most commonly reported childhood psychiatric disorders -oppositional defiant disorder, attention deficit/hyperactivity disorder (ADHD), anxiety and depression -are associated with psychosocial damage such as school dropout, psychoactive substance use, risky behavior and other dysfunctions in adult life.Thus, the issue is not only individual but also social. 2 Despite the impact of childhood psychiatric disorders on the individuals' overall health and their lifetime effects, psychiatric interventions in adolescence and childhood are often neglected, especially in underdeveloped countries. 2,3ccording to the World Health Organization, detecting and treating mental disorders in children and adolescents decreases the likelihood of long-term illness and reduces individual, family, community and health system stress. 4Therefore, it is relevant to highlight the importance of investing governmental funds in research into these early development stages with the aim of reducing the duration of untreated mental illnesses in childhood and adolescence, which may last for the entire adult life. 2 Generally, this type of government investment ends up being targeted toward treatments that provide evidence of results.Therefore, interest in evaluating the effectiveness of available therapeutic models and psychotherapeutic interventions has increased and has given strength to evidence-based therapies.This is especially relevant considering the growing prevalence of behavioral therapies and the need for evidence in support of recognized therapeutic models. 5factor that has a major impact on these types of outcome-focused studies is the adoption of appropriate measurements, given that instrument selection can help to assure the trustworthiness and consistency of studies.In terms of clinical recognition, the degree of applicability and social utility that a study might achieve is based upon the adequacy of the measure to the researcher's objectives and design. 6It is not uncommon for studies to fail to report information regarding the validity, reliability and responsiveness of the measures employed.Some instruments are often used and assumed to be appropriate even though they may be of poor quality. 7Failure to properly describe the instruments used not only hinders the possibility of replicating the study but also weakens the credibility of the results.
In countries in which English is not the native language, such as Brazil, cases of instrument adaptation and translation are more frequent than the development and creation of original measurements. 8The choice to adapt an existing instrument instead of developing a new one may be common in Brazil due to the possibility of comparing data in different samples and contexts, thereby allowing greater equity in the evaluation. 9For this reason, it is important to respect the standardization steps in the instrument adaptation process [10][11][12]  they can be applied in clinical settings in Brazil. 13n considering evidence-based psychotherapy in Brazil, one of the most remarkable aspects is the lack of psychotherapy evaluation studies and the great variety of psychotherapeutic techniques and models that lack scientific validation but are still being offered in the country. 14 change this situation, it is essential to clarify how the results of psychotherapeutic interventions are measured and how the effectiveness and efficacy of treatments are identified among different psychopathologies. 14,15However, it is unclear how psychologists use knowledge garnered by research in the field to make decisions regarding the selection of an evidence-based treatment, especially given the complexity of scientific literature, which may be difficult to understand for professionals outside the academy. 16This is even more true in Brazil, where there is little debate about evidence-based practice and a dearth of literature on the subject in Portuguese. 17 Some aspects of the instruments employed were also identified (e.g., evaluation method, construct to be measured, respondent).In addition, we conducted a critical analysis of how the instruments, which are the core of effective studies, are described in the research.
More specifically, we examined whether the researchers provided information (particularly regarding validity) about the instruments employed, thereby supporting their choice of measures.The purpose is not to find out whether the instrument has already been the object of previous validation studies in Brazil, but rather to determine whether studies in this field seek and describe scientific support, especially regarding psychometric properties, for the selection of the instruments.
Given this objective, the method chosen was a systematic review.The goal was to gather, recognize and synthesize different studies to gauge whether there are gaps in knowledge, inform proposals for improvement 18 and provide a critical and reflexive analysis of the research results that goes beyond summarizing the data found. 19,20

Method
To conduct this systematic review of instruments used in Brazilian studies where psychotherapeutic interventions with children and adolescents were developed, we followed the steps described by the checklist Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA). 21All studies included in the final database were led by at least one Brazilian researcher, and the samples in all the papers comprised only Brazilian participants.Three resources were used for this search.First, the following databases were used to search for articles: 1) PubMed; 2) Scientific Electronic Library Online (SciELO); 3) Biblioteca Virtual em Saúde (BVS; affiliated with the Brazilian Ministry of Health); 4) Literatura Latino-Americana e do Caribe em Ciências da Saúde (LILACS); and 5) Periódicos Eletrônicos de Psicologia (PePSIC) -Latin America.

The choice of keywords was based on Science Health
Descriptors (Descritores em Ciências da Saúde -DeCS).
Additionally, keywords not catalogued but recurrently used by psychologists and psychotherapists were added to maximize the search potential.The second resource used was an interlibrary loan system, to obtain the complete copies of papers.Third, paper authors and journal editors were contacted when the previously listed retrieval methods were unsuccessful.

Results
The first search yielded 1,179 records (SciELO [28], After applying the inclusion criteria, only 62 records remained in the database (SciELO [7], BVS [5], Lilacs [22], PePSIC [15], PubMed [13]).The exclusion criteria removed a further 34 articles from the selection, leaving only 28 records remaining in the final analysis (SciELO [5], BVS [2], Lilacs [7], PePSIC [7], PubMed [7]).Among the 28 studies that comprised the final database, only two required assistance of the third judge to determine their inclusion.The process is shown in Figure 1, which illustrates the records found, denoting results obtained using the Portuguese and English strings.Half of the studies adopted measures specific to the pathology observed or provided as the reason for the consultation, such as depression, anxiety, intelligence, sexual abuse or eating disorder scales.Regarding the overall assessment of patient functioning (behavior, feelings, patient experience) or treatment outcome or progress, studies employed the following instruments to measure intervention effectiveness: five studies used The Child Behavior Checklist (CBCL), 23 three the Children's Global Assessment Scale (CGAS), 24 and three used the 36-Item Short Form Health Survey (SF-36). 25Two studies employed the Clinical Global Impression (CGI), 26 and another two employed the Child Psychotherapy Q-Set (CPQ). 27Other instruments were employed only once among the identified articles, namely: Questionário Parental de Sintomas (Parental Symptom Questionnaire; without pointed reference), Profile of Mood States (POMS), 28 Development and Well-Being Assessment, 29 and Youth Quality of Life Instrument-Research Version (YQOL-R). 30e quality of the description of the instruments used in the studies was also analyzed, or, more specifically, whether the validity or reliability of evidence within the Brazilian population was mentioned, taking into account that all studies had a Brazilian sample.Of the 92 instruments referenced in the 28 articles, 57 (61.9%) did not have descriptions about evidence of validity within the Brazilian population; for 31 instruments (33.6%), validity evidence was mentioned, but the study did not detail which validity parameter was used.
Only three studies described content validity evidence for the instruments used.Furthermore, information about reliability was described for only two instruments.
All results are shown in Table 1.

Discussion and considerations
Studies that seek to evaluate the results of psychotherapeutic interventions with children and adolescents are extremely relevant for indicating the possibilities, benefits and limits of each modality of intervention for different pathologies or psychological structures.That type of study, which aims to evaluate results in the health field, should seek the most accurate, reliable and cautious way to measure the phenomenon observed.The decision regarding the preferred treatment for a particular pathology should be based on the most accurate data available.Therefore, information on the validity of the instruments used for this purpose is extremely important.Considering that this systematic review was performed in five databases in September 2017 without time limitations, the outcome -that only 28 studies focusing on psychotherapeutic intervention with children and adolescents in Brazil used some type of instrument or assessment to measure their own resultindicates a significant lack of research into mental health in this population.These findings reinforce the fact that the field of quantitative measurement of psychotherapy effects remains insufficiently explored (and perhaps inadequately valued) by Brazilian researchers. 59struments are an essential tool for research designs that aim for external validity and generally applicable results. 60  a more thorough method for psychological evaluations. 67riangulating data among measurements that assess the perspective of the patient, close relatives, teachers, patient's friends and the therapist could make research designs substantially more robust. 64The wide use of

Limitations
The objective of the present review was to offer an overview of how psychotherapeutic intervention studies on children and adolescents have been methodologically developed with a particular emphasis on the measures used and their description by researchers.The limitations of this study are due to language and the number of databases involved; both limitations may have excluded papers that would have fit this systematic review's inclusion criterion.As a suggestion, other studies should widen their searches to other countries to identify whether the lack of data on child and adolescent psychotherapy is only a Brazilian issue or if it is part of a larger dilemma.Another limitation of this review is that it does not cover whether the studies that were described as providing validity and reliability evidence for their instruments (and those that were not) actually used valid and reliable instruments for the Brazilian context and target population.Although this was not the objective of the present review and would exceed our initial scope, we suggest future reviews on this topic.

Implications
There is an ample scope for the development of studies in this area, as demonstrated by the limited number of studies in the field of child and adolescent psychotherapy that use instruments that provide evidence of validity to evaluate results and that take place in Brazil.The American Psychological Association has encouraged the development of standardized measures to help therapists evaluate the diagnostic process and therapeutic progress. 68o qualify mental health interventions and increase their applicability to different populations, we suggest that research should focus on the development of instruments to assess these interventions and that the benefits of such research should be shared beyond academic boundaries.
Therefore, studies should seek to systematize the body of evidence related to psychotherapeutic processes and assessments in Brazil to qualify the interventions employed in the country and to make this body of knowledge available to clinical professionals The objective of this study was to evaluate how Brazilian studies that researched psychotherapeutic interventions with children and adolescents based on different theoretical approaches have evaluated and measured their own results.For this purpose, we described the study designs as well as the different types of psychotherapeutic interventions and theoretical approaches adopted.In addition, we sought to investigate which instruments were used to assess interventions.

Four
judges conducted the searches.Two of the authors of this study reviewed the PubMed, SciELO and BVS databases, and the other two examined the LILACS and PePSIC databases.The authors conducted the searches individually, applying the inclusion and exclusion criteria and then comparing the results with their working partner.In cases where their opinions diverged regarding the inclusion or exclusion of a study, a third judge who was not working in that database assessed the case to make the final decision.The search was performed in September 2017, with no date limitations in the databases.The following inclusion criteria were applied: 1) research that described group, individual, brief or single psychotherapeutic interventions with children and adolescents based on any theoretical approach; 2) articles in English, Spanish or Portuguese; 3) research that included a Brazilian sample and/or was led by at least one Brazilian researcher; and 4) research that used instruments and assessments to evaluate the interventions performed.After applying the inclusion criteria, the following exclusion criteria were observed: 1) papers duplicated across databases; and 2) studies for which we could not retrieve the full text or abstracts (i.e., when the editor or author did not return contact or when the interlibrary loan system could not provide the required material).In all databases, we used Portuguese and English descriptors and strings.The model string for all surveys was: "(youth OR adolescent OR child) AND (instrument OR scale OR test OR inventory OR assessment OR survey OR questionnaire OR measure) AND (psychotherapy)" in English; the same string in Portuguese was also employed.Since the databases have different search logics, the descriptor string was adapted for each database, although coherence and homogeneity were maintained among all strings.To summarize, in PubMed, we used the advanced search, inserting the complete string in the open field "Advanced Search Builder."Considering that this database has worldwide reach, the descriptor "Brazil OR Brasil" was added to the string in hopes of refining the results.The SciELO database has two search options: free entry field, which requires the selection of just one logical operator (AND or OR) and therefore does not support the use of the model string, and advanced search, which has three separate fields for the search.Thus, in the first field, we entered the descriptor "youth OR adolescent OR child," in the second "instrument OR scale OR test OR inventory OR assessment OR survey OR questionnaire OR measure," and in the third we entered the descriptor "psychotherapy."The BVS database, in contrast, provides only separate fields for the search, and each descriptor within the complete string must be added separately.Finally, Lilacs allows the complete string to be entered into the search field, whereas PePSIC does not return results when the complete string is used in the free entry field.Therefore, to amplify the search potential, we selected the broader descriptors from the string and applied them in the basic form, which contains only three fields.In the first, we entered the descriptor child, in the second adolescent, and in the third psychotherapy.With regard to the term psychotherapeutic intervention, the present review used Kazdin's 22 conceptual perspective, which describes it as having a focus on improving the individual's adaptation and helping to reduce harmful behavior, psychological symptoms or complaints through psychological processes such as social support, interpersonal influence, and learning.The author's concept ranges from classic psychotherapeutic intervention, such as individual or group therapy derived from largely widespread approaches such as psychoanalysis and cognitive behavioral therapy, to non-traditional interventions with theoretical and scientific bases, such as boot camps where children are responsible for an animal or activity.Retrospective studies were not considered.Descriptive statistical techniques were performed using the Statistical Package for the Social Sciences version 19.0 to identify frequencies within the studies' designs, types of psychotherapeutic intervention, theoretical approaches and instrument evaluation method (whether projective or psychometric) of the psychotherapeutic intervention.
Qualitative research designs, which are widely employed in the psychotherapy research field, do not contribute to the generalization of the results of interventions or evaluations performed.The scarcity of research on psychotherapeutic interventions can also obscure the identification of variables that promote positive and negative treatment outcomes and that may contribute substantially to the development of evidence-based and effective practices and techniques. 61Moreover, this gap restricts the offer of psychological interventions in places where only treatments that show evidence of effectiveness are available; for example, the difficulty in demonstrating instruments answered only by patients is also a limiting factor when the participants are pre-verbal or nonliterate children.Because self-assessment through psychometric measures is very difficult or impossible in these cases, these patients need the evaluation of an external observer who can help to check the treatment progress.Considering that the evaluation of treatment effectiveness in pre-verbal children depends almost exclusively on their guardians' or therapist's assessment, we found a surprising dearth of studies that use other perspectives to evaluate treatment outcomes.

Table 1 -
Instrument description regarding evidence of validity and reliability of validity or reliability was not described for the Brazilian population; ADHD = attention deficit/hyperactivity disorder; CV = content validity; EV = evidence of validity was mentioned, but studies did not detail which validity parameter was used; HTP = The House-Tree-Person Test; OCD = obsessivecompulsive disorder; SF-36 = 36-Item Short Form Health Survey; SNAP-IV = Swanson, Nolan and Pelham Questionnaire.