Implementation of evidence-based practices for children in four countries: a project of the World Psychiatric Association

Implementação de práticas baseadas em evidências para o tratamento de crianças em quatro países: um projeto da Associação Mundial de Psiquiatria

ORIGINAL ARTICLE

Implementation of evidence-based practices for children in four countries: a project of the World Psychiatric Association

Implementação de práticas baseadas em evidências para o tratamento de crianças em quatro países: um projeto da Associação Mundial de Psiquiatria

Kimberly E Hoagwood; Kelly Kelleher; Laura K Murray; Peter S Jensen; Integrated Services Program Task Force^*

Columbia University, New York (NY), EUA

^{Correspondence}

ABSTRACT

OBJETIVE: The present study examined implementation issues in adopting cognitive-behavioral therapies in routine clinical settings in four countries reflecting diverse cultures, languages, settings, and traditions.

METHOD: A Director's Systems Survey was administered prior to program implementation and one year later. Therapist ratings on attitudes about evidence-based practices and satisfaction were also gathered.

RESULTS: All sites reported successful adoption of the program, although significant variations existed in fiscal support, family involvement, prior experience with cognitive-behavioral therapies, and plans for sustainability. Therapists' ratings indicated overall satisfaction with the implementation of the project. Findings from the Director's Systems Survey pointed to five factors facilitating implementation: 1) early adoption and guidance by innovative leaders (i.e., the Directors); 2) attention to the "fit" between the intervention model and local practices; 3) attention to front-end implementation processes (e.g., cultural adaptation, translation, training, fiscal issues); 4) attention to back-end processes early in the project (e.g., sustainability); and 5) establishing strong relationships with multiple stakeholders within the program setting.

CONCLUSIONS: The implementation issues here mirror those identified in other studies of evidence-based practices uptake. Some of the obstacles to implementation of evidence-based practices may be generic, whereas issues such as the impact of political/economic instability, availability of translated materials, constitute unique stressors that differentially affect implementation efforts within specific countries.

Keywords: Evidence-based medicine; Cognitive therapy; Behavior; Feasibility studies; Child

RESUMO

OBJETIVO: Este estudo visa a examinar problemas na implementação de técnicas psicoterápicas cognitivo-comportamentais em ambientes clínicos de atendimento primário em quatro países que refletem diversas culturas, línguas, ambientes e tradições.

MÉTODO: Uma pesquisa foi aplicada aos diretores de Sistemas Clínicos antes da implementação do programa e um ano após. Também foram coletados dados sobre como os terapeutas avaliavam as ações relativas à prática baseada em evidências e qual seu grau de satisfação com essa prática.

RESULTADOS: Todos os locais de implementação relataram a adoção bem sucedida do programa, ainda que com significativas variações no apoio fiscal, envolvimento familiar, experiência prévia com terapias cognitivo-comportamentais e planos de sustentação em longo prazo. As avaliações dos terapeutas indicaram uma satisfação generalizada com a implementação do projeto. Achados da pesquisa com os diretores dos Sistemas Clínicos apontaram cinco fatores que facilitaram a implementação: 1) rápida adoção e orientação por líderes inovadores (i.e., os diretores); 2) atenção à adequação entre o modelo de intervenção e as práticas locais; 3) atenção desde o início à relação entre os processos de implementação e os usuários finais (e.g., adaptação cultural, tradução, treinamento, problemas fiscais); 4) atenção precoce aos processos de retaguarda do projeto (e.g., sustentação); e 5) estabelecimento de relações estreitas com múltiplos financiadores da instalação do programa.

CONCLUSÕES: Os problemas de implementação encontrados neste estudo se assemelham aos identificados em outros estudos sobre a aquisição de práticas baseadas em evidência. Alguns dos obstáculos para a implementação de práticas baseadas em evidência podem ser generalizados, ao passo que problemas como o impacto da instabilidade político-econômica e disponibilidade de materiais traduzidos constituem estressores peculiares que afetam de forma diferenciada os esforços de implementação em cada país.

Descritores: Medicina baseada em evidência; Terapia cognitivo-comportamental; Comportamento; Estudos de viabilidade; Criança

Introduction

International epidemiological studies emphasize the pandemic of behavioral and emotional disorders among youth throughout the world and the high social and economic toll they exact.^1-2 High rates of disorders, coupled with insufficient clinical capacity for treating children's psychiatric disorders, has created an international sense of urgency. For example, an international survey of 66 countries conducted by the World Health Organization (WHO) in 2003 -The ATLAS Child and Adolescent Mental Health Resources Survey (2005) - found that in less than 1/3 of all countries was it possible to identify an individual or governmental entity with sole responsibility for child mental health programming,³ and funding for child mental health services was rarely identifiable in the country's budgets at all.

In response to these capacity issues, in 2003 the World Psychiatric Association (WPA) Presidential Program on Child Mental Health was created to identify action steps that could be undertaken to improve child and adolescent mental health internationally. The Integrated Services Program Taskforce on Mental Health Services identified three activities that could be undertaken in to improve delivery of effective treatments for youth: 1) development, translation, and publication of a set of manualized psychosocial therapies for internalizing and externalizing problems in youth, derived from an extensive review of the research base on effective treatments; 2) training in the use of these manuals among international teams of 30 clinicians in four countries (Brazil, Egypt, Israel, and Lebanon) who volunteered to participate; and 3) an evaluation of the implementation effort itself, with specific attention to factors that either impeded or facilitated implementation of these manualized psychotherapies. This paper describes the findings from this evaluation.

Our focus is specifically on implementation, defined as the extent to which the proposed interventions were feasibly carried out and instituted in the four pilot sites. We contrast this with adoption, which is the decision to engage in the new interventions. All of our sites were 'early adopters' i.e., the Directors of the sites, leaders in their respective countries, had already decided to adopt the interventions. In our study, therefore, we were less concerned about the decision to participate than the details of how the intervention was put into place after the decision to adopt had been made.

As with other implementation studies,^4-6 many factors influence the extent to which service organizations fully institute particular interventions. The factors include: 1) organizational capacity, infrastructure, and culture; 2) regulations, financing and competition; 3) characteristics of the intervention itself; and 4) clinical issues (e.g., the" fit" of the intervention with already existing programs within the site; perceived relevance of the intervention). The particular intervention is described in detail in the other papers in this volume. Prior studies suggest that knowledge about how to promote use of psychotherapy manuals is weak⁷ and that the perspectives of the implementing clinicians as well as system factors may contribute to the success or failure of implementation efforts.⁸ Research on factors that influence implementation have largely been conducted in the United States or Great Britain.³ The evaluation aimed to examine factors associated with implementation of manualized CBT therapies in either community outpatient clinics, schools, or hospitals located in four countries. A complete description of the training, the manual development, and the clinical sites has been presented elsewhere.^9-12

Method

1. Data collection

Directors of the sites were interviewed by the primary authors at the beginning and the end of the project, using a survey instrument designed to assess barriers to and facilitators of implementation of the manualized psychotherapies. This survey instrument, called the Clinic Systems Director Survey for WHO-WPA, was derived from an instrument designed by the MacArthur Foundation Research Network on Youth Mental Health (J. Weisz, PI) for a national survey of Clinic Directors in the USA. That survey was based on organizational and policy research findings on implementation processes associated with adoption of evidence-based practices.^4,6,13 Semi-structured interviews completed with clinic directors from a national sample of 197 mental health clinics in the MacArthur Foundation survey across 38 states is being used to develop profiles of the variability of these system factors.

The purpose of the WPA survey was to examine specific administrative, fiscal, and clinical issues that previous studies^5,14 indicated are important to the uptake of new clinical services. The WPA survey grouped the issues into 4 domains: 1) governance, financing and regulatory processes; 2) family or consumer involvement; 3) clinical and organizational implementation; and 4) sustainability. Open-ended questions were used to elicit information about the implementation process. In addition, the survey included a standardized scale, developed by New York State and by the MacArthur Foundation, to rate the importance of specific factors in the implementation of this project. In pilot testing of this scale, factor analysis yielded 7 dimensions (financing, training, EBP characteristics, research support, client characteristics, support for EBP, and leadership) and alphas ranged from .66 to .86.¹⁵

In addition, therapists at each of the four sites completed two measures to assess the acceptability and satisfaction with the treatments. The National Survey Questionnaire (adapted from D. Kolko) assessed clinicians' attitudes and knowledge about manualized treatments and supervision processes.^7,16 Clinicians also completed the Therapist Satisfaction Index (TSI) (adapted from Addis & Krasnow, 2000) to examine attitudes about treatment manuals and their role in clinical practice.⁷ See Table 1 for specific items.

Results

1. Director's System Survey

1) Governance, regulatory issues, and financing

a) Structural and economic factors prior to implementation

The responses were characterized by diversity. The site for the pilot study in Brazil was a large urban university clinic. As part of the University, the site employs numerous mental health professionals who are salaried employees of the University. These positions are funded under both the Ministry of Health and the Ministry of Education. Workers have a portion of their income withheld by the government to fund health and mental health care. Thus, all therapists and supervisors at the pilot site were either salaried employees or worked on a voluntary basis, and patients seeking care there had no out of pocket costs for care.

In contrast, the pilot site in Lebanon was a fee-for-service clinic staffed by psychiatrists and psychologists. While there is health insurance for the Lebanese, mental health care is not included, and almost no patients there had any form of mental health insurance coverage. Additionally, while the government backed National Social Security Fund covers visits to physicians (psychiatrists included) it does not cover psychotherapy sessions provided by psychologists or social workers. Therefore, patients paid for care completely out of pocket. These financial arrangements requiring out of pocket payments appear to make services less accessible in Lebanon.

The pilot site in Egypt was a school where children received care through the government funded national health insurance program. Mental health staffs were salaried employees of the mental health agency that received its support directly from the Ministry of Health to provide care at the school. No out of pocket payments were required from children or their families. Supervision and training were provided as part of the national health insurance plan.

In Israel, the pilot site was a new location designed to provide services to the Bedouin community. However, in general, Israelis are insured through a national health insurance program as well. Many mental health services are not covered by this program so the Department of Health provides for modest program funding for mental health care. This fixed program funding does not require patients to pay for services but limits the amount that mental health clinicians and agencies can provide. So many services are funded by private grants or research funds. Patients do not have out of pocket costs but the extent of specialty involvement is limited because of the requirements for obtaining external funds.

b) Structural and economic factors during and after the implementation

As expected, the diverse economic and structural features of the pilot sites resulted in marked differences in implementation and the likelihood of sustaining the program after the end of the grant period. Brazil used the small amount of grant money for incidental expenses mostly associated with training. Overall, their costs of providing services and receiving income were not altered by participation and implementation of the treatment manuals. In fact, the Director observed that since the manuals' sessions ended sooner than some of the psychodynamic treatments typically provided, the sessions may have actually reduced average treatment costs for the site. He felt that the likelihood of continued use of the manuals was high as were the chances for ongoing supervision and collection of patient specific data.

Although a very different environment, the Director from Lebanon also felt strongly that the manuals had been implemented effectively and would likely be sustained as ongoing treatments. The modest grant monies had been employed to assist with training and incidental expenses, but the manuals and their supervision were accommodated in ongoing supervision experiences and were similar to other treatment activities underway. Thus, the change from routine practice was minimal. Moreover, no additional financial costs were incurred for the clinic since the interventions represented little change from usual care.

The Egyptian experience was similar. Grant money was used to pay for training time, translation of materials, and for transportation of therapists to reach the school sites where the intervention was delivered. However, the treatment manuals and supervision were similar to practices already underway in the school mental health clinic. This allowed the clinicians to implement the manualized interventions in a relatively seamless fashion. The Director had confidence that therapists would be able to continue using the manuals and that supervision from the originally trained supervisors would continue. There was less clarity about how new training and dissemination might occur.

The Israeli site was in a new clinic with no previous services. As such, there was no disruption in ongoing activities and the implementation was considered to be relatively smooth. In contrast to the other three sites though, the long term sustainability of the intervention and possibly even the site is unclear. Because the site was supported by grant funds only, the sustainability of the model will be highly dependent on the success of the agency, Director and clinicians in obtaining additional grants.

2) Family involvement

A recent scientific review of studies on family involvement identified several critical care processes (e.g., family education, support, engagement, empowerment) associated with improved family or youth outcomes.^17-18 As a consequence, the Director's Survey included questions about nature, degree and extent of family involvement in mental health services prior to implementing this WPA intervention program and subsequent to it.

a) Level, type, and extent of family involvement before implementation of the WHO-WPA Program

Family involvement in clinical care varied across the sites. In Brazil, all families typically received a baseline assessment (e.g., K-SADS, baseline scales for emotional/behavioral problems and for ADHD) as well as psychoeducational conferences open to family members and to community and school personnel related to child mental health issues. There was a concerted focus on psychoeducational programs in schools and family members also participated in treatment. In general the level of involvement was considered to be strong.

In Egypt, a different pattern emerged. Family involvement was described as minimal because the program itself was housed in schools and in general the families did not like to go to schools, as it is viewed negatively (families are only invited to schools when there is a problem). However, the Director reported that some of the homework assignments and activities connected to the program improved communication with families.

In Israel, services were also provided at schools serving Bedouin communities although the clinical staff operated from a hospital. Family involvement, particularly by fathers, was especially low, and the staff had to engage in concerted outreach to involve family members. The program itself changed the level and extent of family involvement.

In Lebanon, as in Brazil, family involvement was a pre-requisite of clinical services. If a parent was not available, the clinic sought grandparents or other extended family members to participate in at least the intake. Family participation was essential to clinical care.

b) Family involvement subsequent to the WPA program

The Directors in Brazil and Lebanon where family involvement was already strong agreed that involvement in the WPA program did not change the level of family participation. In Egypt and Israel, the program was reported to have improved the level and extent of family participation. For example, when children began to improve clinically or academically, the relationship between clinical staff and families improved.

3) Program implementation

a) Pre-existing factors that facilitated implementation

In Brazil, the Director indicated that prior experience and familiarity among his staff with CBT, positive attitudes towards CBT, secure financing of services, and extensive prior experience with data collection facilitated implementation. In Egypt, the presence of a strong child psychiatry clinic, experience in providing regular trainings in the clinic, and availability of local clinical staff who were motivated to participate in the program facilitated implementation. In addition, strong connections and support from larger organizations in Egypt, such as the Ministry of Education, helped to improve adoption of the intervention program.

Pre-existing and positive relationships and connections with the local community were noted as key factors improving implementation in Israel. Staff had already been working in the particular rural Bedouin village in which the project was launched, creating strong connections with the mayor of the village, the Ministry of Education, and the school principal. Being accepted by the local people themselves was also noted as an important factor.

In Lebanon, the Director indicated that the existence of a pre-existing multidisciplinary clinic within their site with a robust infrastructure for conducting the project was critical to its implementation. In addition, the clinical staff in this site already had experience in using a variety of rating scales to measure outcomes in their patients. The site had psychologists and social workers on staff, many of whom where familiar with parent-training concepts and many of whom had already integrated these techniques into their clinical work.

b) Key facilitators of implementation

In Brazil the prior commitment among clinical staff in the use of CBT interventions was a key facilitator of the uptake. Although the location of this site, Porto Alegre, is well-known for its CBT training for adults, they were eager to train clinical staff to provide this type of intervention to children.

In Egypt, one of the primary clinical facilitators was availability of the manual translated into Arabic. This was reported as an important factor by the Directors within Israel and Lebanon as well. In addition, in Egypt, the Director reported that the contents of the manuals themselves contributed to successful implementation in that they allowed for fidelity and flexibility with the components.

In Israel one of the major facilitators of implementation was the ability to strengthen the dialogue between the academic teaching hospital and the staff in the rural village where the program was implemented. The CBT treatment itself was seen as practical and short-term, and this facilitated implementation. The principles and techniques were found to be useful by the practicing therapists. Another facilitator was the choice of the sites for delivery - Israel selected a place where there are limited, if any, experienced therapists. Thus this program was seen as a welcome addition to a resource-poor environment.

In Lebanon, the Director indicated that the key facilitators of implementation included the practical usefulness of the parent sessions. In addition, the overall training itself was seen as contributing to success. Finally, the combination of inexperienced with experienced therapists enabled the therapeutic team to collaborate well and contributed to the overall success of the program.

c) Key impediments to implementation

The Director from Brazil reported relatively few impediments, other than those that are common to mental health service delivery in general, such as lack of capacity, lack of a sustained infrastructure, and lack of training.^19-20 One specific challenge was difficulty in access for children because of the inconsistency of adult involvement in some families. During the summer months, for example, access often drops because families go on vacation, making recruitment and maintenance challenging.

In Egypt, the Director reported several challenges consistent with school-based implementation projects²¹ including academic scheduling within schools making it difficult to find time to meet for therapy (eventually all sessions had to be delivered at the end of the school day). Another challenge included family beliefs such as children not needing mental health services. Some families refused to do homework in between sessions. Finally, some of the sessions were too long and some required considerable investment of time by family members.

In Israel the most significant impediments were financial. For example, the site experienced difficulty recruiting therapists because of inadequate funds to compensate them, and to provide adequate compensation for supervision. This site reported challenges associated with quality assurance, especially with respect to monitoring therapists' adherence to the treatment protocols. Challenges were also noted in engaging fathers in treatment services. Finally, the large number of breaks in the school year created added challenges to the continuity of the services.

In Lebanon, the major challenges were due to the impact of significant political and socioeconomic turmoil at the time of the program's implementation. These events made logistical implementation (e.g., patient recruitment and attendance) as well as continuity of the intervention program very difficult. The Director explained that since their site was fiscally supported through a fee for service mechanism, psychotherapy was seen as a luxury to be afforded in times of stability. Some families would agree to participate in the treatment, but would drop out after attending a few sessions when they saw some improvement. When political turmoil and questionable safety was added to this pattern, the difficulties associated with recruitment and attendance in the program were amplified. The clinic received referrals only for very severe clinical cases during these times of turmoil, and these cases were usually deemed inappropriate for the treatment models within the WPA treatment intervention program.

4) Sustainability

All Directors expressed an intention to continue the interventions in their country, but the optimism of doing so varied, based largely on fiscal issues. In Brazil, the Director reported that one of the psychologists trained in this program is now collaborating in supervision for residents on CBT for child patients, thus ensuring sustainability. This site is planning to deploy these interventions in community outpatient settings. Plans are also underway to conduct an efficacy trial of these interventions in three cities within Brazil.

In Egypt plans have been made to maintain the program, although the location would likely be not in schools but rather in community outpatient settings. This site has made active plans to sustain the treatments through a trainer-of-trainers model, as well as to continue use of them within guidance and counseling centers.

In Israel, continuation of the project is highly dependent on external funding. Specific plans have been made to expand the project to the Palestinian Authority, and to provide these treatments in both schools and clinics. This site plans to use existing staff to provide the services and to expand the number of staff trained on the model.

In Lebanon plans have been made to continue the use of the WPA manuals in the hospital clinic, using the same staff and training additional staff. Training, especially parent training, was expanded to social workers and nurses who work at health clinics or community centers in poor areas of the country. This training has been endorsed by the Lebanese Ministries of Public Health and Social Affairs that manage the health and community centers respectively. Parent management techniques were viewed as especially useful, with the potential to be readily adopted in a variety of community settings. The likelihood of continued sustainability of the program is improved by a focus on one particular component of it, and garnering of governmental support for it.

Importantly, Directors of all sites had crafted specific plans to attempt to maintain sustainability of the program by the end of the brief (one-year) timeframe.

5) Director's Survey Scale of Factors Affecting Implementation

A scale to assess the relative importance of a variety of factors hypothesized to influence implementation of new therapies was administered to the Directors. The scale was adapted from the Dimensions of Organizational Readiness, developed by New York State, and by the MacArthur Foundation (see Table 2). The diversity of the implementation experiences described above were mirrored in the results from this scale. All sites agreed that leadership support, clinical and management staff support, and the fit between the program and the philosophy or mission of the clinic were important in improving implementation. The role of outside agencies in implementation processes was rated as unimportant by all sites except Egypt where it was rated as very important. The importance of an intervention being "evidence-based" was considered important to Lebanon and Brazil and only of average to low significance to Egypt and Israel. The fit of the program with current training practices was rated as having very low importance by Israel in contrast to the other sites, where it was rated as very important.

2. Clinician reports

1) National Survey Questionnaire

Six questions were adapted and used from the National Survey Questionnaire,^7,16 to assess therapists or clinicians perspective on treatment manuals, and training and consultation practices. Results indicate that clinicians in only one site (Lebanon) had a very clear idea of what a treatment manual is, but they also reported that they only thought about its use to a minimal extent. In contrast, Israel's clinicians reported a somewhat unclear idea of what a treatment manual is, but indicated that they think about its use often. Clinicians in all sites indicated that manuals are important to good clinical practice and will enhance outcomes. Clinicians in all sites agreed or strongly agreed that training and consultation are important for good clinical practice.

2) The Therapist Satisfaction Index

The TSI, adapted from Addis & Krasnow, 2000, was administered to assess clinician's perspectives on the acceptability of manualized treatments and the likelihood of using it in the future.⁷ We added satisfaction questions to assess whether the therapist liked the particular therapeutic model used in this WPA program and planned to use it in the future. Brazil and Egypt completed the TSI for every completed case for the externalizing manual, resulting in an N of 10 and 34, respectively. Lebanon and Israel completed one TSI per clinician, each with an N of 2 (See Table 1 for mean ratings.) The items with the strongest agreement were: 1) approach will be helpful for other children in the future; 2) approach will enhance average outcomes of children; 3) I liked using this approach; 4) approach allowed me to use my best clinical skills; and 5) I plan to use this approach with other children in the future. No item mean fell below a neutral response indicating overall satisfaction, acceptability, and plan to use this therapy in the future.

T-tests were conducted to detect differences between responses of the two sites with the largest number of clinician responses (Brazil and Egypt). Significant differences were obtained for two of the items on the scale (i.e., "the approach I used allowed for authenticity of the therapeutic interaction" and "the approach I used will enhance average outcomes of children seen in psychotherapy"; Brazil: mean = 4.2 vs. Egypt: mean = 3.6 and Brazil: mean = 4.4 vs. Egypt: mean = 3.9; p < .029 for each of the tests). While this is only a preliminary consideration of differences, it implies that clinicians in sites with greater experience in adopting evidence-based practices (i.e., Brazil), may be more positively disposed towards the impact of them on their own clinical work.

Discussion

Efforts to integrate research-based practices into routine clinical settings are increasingly common^13,22-24 and they share a goal of improving access to and quality of mental health services. Despite the breadth of these efforts, however, research on implementation processes remain thin and there are farmore questions about how to do so than answers. This is particularly true in the international arena where child mental health services are most severely under-resourced.

The Presidential Program of the WPA on Child Mental Health is the first international effort to systematically deploy specific evidence-based psychotherapies for children into routine clinical practice, and to study the implementation process. There were several important findings from this study that we believe may be relevant to implementation of research-based clinical practices elsewhere. First, all four of the sites were, by definition, "early adopters" who came to this project voluntarily, with commitment and motivation to participate despite numerous anticipated obstacles. As has been found in other studies of innovation diffusion,^25-27 early adopters tend to be seen as "local missionaries" within their sites, able to speed the diffusion process because they serve as role models in the discrete application of new ideas. The fact that the sites were guided by innovative leaders who embodied these qualities no doubt assisted the implementation effort. In addition, the Directors had a breadth of experience in clinical practice, leadership, research, and governance. Because of the range of these experiences, the management of the projects at the site level was very strong and the Directors were able to exert high quality leadership and guidance to the project, even when formidable obstacles (such as the political upheaval that arose in Lebanon) occurred.

Second, all of the Directors underscored the importance of attending to the "fit" or match between the new clinical practice and existing structures or practices already functional in their sites. For example, there was agreement among the Directors that the fit of the program with existing clinical supports and with the philosophy or mission of the clinic contributed significantly to the ease of implementation within their sites.

Third, once appropriate cultural (i.e., translation of materials, adaptation for local variation), training, and fiscal modifications were made, the ability of the sites to implement these relatively complicated clinical practices was highly successful. Furthermore, the variations across the sites, once the initial obstacles were overcome, appears to have been minimal. This suggests that attention to implementation processes at the front end may be especially important and that once processes have become routinized, additional implementation activities may be less complicated.

Fourth, significant variations exist in the extent to which and the way in which this therapy program is anticipated to be sustained over time. However all of the Directors had crafted specific plans for sustainability and had done so prior to the project's termination. These plans seem to depend largely upon the local environment and the consistency of the intervention with other practices that are already in place. For example, in Brazil plans are underway for a community-based extension of the program, along with a systematic research effort. In Egypt, extension of the program is likely to be housed outside of the schools, given new needs. In Israel, the program is likely to be deployed into other communities if funding is secured. In Lebanon, parent training is believed to be most transportable and thus will be applied in new settings.

Fifth, factors related to the ease with which implementation was effected included extensive prior connections and relationships with multiple stakeholders within the settings in which the program was implemented. For example, in Israel, Egypt, and Lebanon relationships were forged with both large systems providers (i.e., Ministry of Education, Ministries of Public Health and Social Affairs) as well as with local providers (i.e., school principals, mayors). The project was thus fitted into existing structures and relationships and this prior experience appeared to facilitate the integration of the CBT therapies into these settings.

Overall, it appears that some of the issues faced by the sites in implementing this project mirror efforts in other countries (notably the USA) to incorporate evidence-based practices for children into community practices. For example, issues around leadership support, training and support by and for clinicians, cultural adaptations of manuals, and the "fit" of new clinical practices within existing routines - all of these have been identified as critical ingredients in facilitating adoption of new clinical services in other studies.^6,20,28 Thus it appears that some obstacles to scaling up research based practices may be generic both domestically and internationally. However, some issues, such as the impact of political instability, availability of translated materials, or serious fiscal crises, are unique and perhaps unpredictable obstacles that may differentially affect implementation efforts.

Differences among the sites in assigning importance to implementation factors may reflect a range of pre-existing or culturally specific conditions. For example, in Lebanon the fit of the model with current clinical training practices was ranked as very important yet in Israel this factor was rated unimportant, indicative perhaps of differences in the context within which this project was integrated. Similarly, levels of family involvement varied dramatically across sites, with some sites (i.e., Brazil and Lebanon) indicating significant family involvement as a pre-requisite of clinical care irrespective of this program and other sites (i.e., Egypt and Israel, both school-based sites) indicating lower levels of family involvement. Cultural differences in perspectives about parental roles and responsibility for children's healthcare may have influenced these differences across sites or they may have reflected setting variations (clinics vs. schools). Cross-cultural variations in implementation processes will be especially important to examine if new clinical services are to be implemented efficiently.

Conclusions

The findings from this evaluation point to the feasibility of collecting data from clinicians and directors in an international context. However doing so requires considerable investment of time to ensure that data collection procedures are well coordinated, clearly communicated, and that adequate infrastructure support exists. The use of simple measures, translations of all materials, frequent and consistent communications (through conference calls, e-mails, web-based systems) to ensure the integrity of data collection procedures, and, use of existing fiscal, clinical or technical infrastructures facilitated the uptake of this program.

One of the contributions of this project consistently cited as valuable was the provision of training on the CBT model. Despite concerns that treatments developed in some cultures may not be accepted by other cultures, we found the opposite: an openness to learning how to deliver manualized treatments even in settings where no prior experience with them existed. Neither the Directors nor the therapists viewed the implementation of the manualized therapies as hindering their clinical skills or flexibility, but rather enhancing them. These findings suggests that effective transportability of manualized clinical treatments may require more attention on how to do so - on the implementation processes themselves-- rather than on whether to do so.

Acknowledgements

See complete acknowledgements in the Introduction to the Special Section, this issue. Also, the authors would like to acknowledge the efforts of the clinicians and therapists who contributed their time and expertise to this project, including providing comments on this paper: Lynn Farah and Youmna Cassir, Beirut (Lebanon); and Mary Azer and Doa Habib (Alexandria, Egypt). We wish also to acknowledge the contributions of Marleen Radigan, New York State Office of Mental Health, in data analysis consultation and Maura Crowe for her superb editorial assistance.

15. Hoagwood K, Pezzulo, JXJ. Dimensions of organizational readiness (The DOOR): the development of a state-planning tool for implementing evidence-based practices in children's mental health systems. Under review.

28. CATS Consortium. Child and Adolescent Trauma Treatments and Services (CATS): An Implementation Study of Evidence-Based Treatments for Youth Trauma. J Clin Child Adolesc Psychol, in press.

Submitted: 21 September 2005

Accepted: 20 December 2005

Financing: World Psychiatric Association (WPA)

Conflict of interests: None

• 1. Murray C, Lopez AD. The global burden of disease: A comprehensive assessment of mortality and disability from disease, injuries, and risk factors in 1990 and projected to 2020. Published by the Harvard School of Public Health on behal of the World Health Organization and the World Bank. Cambridge, MA: Harvard University Press; 1996.
• 2. Kessler RC, Berglund P, Demler O, Jin R, Koretz D, Merikangas KR, Rush AJ, Walters EE, Wang PS. The epidemiology of major depressive disorder: results from the National Comorbidity Survey Replication (NCS-R). JAMA 2003;289(23):3095-105.
• ³
  ATLAS: Child and adolescent mental health resources: Global concerns: Implications for the future. Geneva: WHO Press; 2005. [cited 15 nov 2005] Available from: URL: http://www.who.int/mental_health/resources/Child_ado_atlas.pdf
• 4. Glisson C, Schoenwald SK. The ARC organizational and community development strategy for implementing evidence-based children's mental health treatments. Ment Health Serv Res. 2005;7(4):243-59.
• 5. Glisson C. The organizational context of children's mental health services. Clin Child Fam Psychol Rev. 2002;5(4):233-53.
• 6. Schoenwald SK, Henggeler SW. Special series: current strategies for moving evidence-based interventions from the lab into clinical practice introductory comments. Cogn Behav Prac. 2003;10(4):275-8.
• 7. Addis ME, Krasnow AD. A national survey of practicing psychologists' attitudes toward psychotherapy treatment manuals. J Consult Clin Psychol. 2000;68(2):331-9.
• 8. Schoenwald SK, Hoagwood K. Effectiveness, transportability, and dissemination of interventions: what matters when? Psychiatr Serv. 2001;52(9):1190-7.
• 9. Bauermeister JJ, So CY, Jensen PS, Integrated Services Program (ISP) Task Force. Development of adaptable and flexible treatment manuals for externalizing and internalizing disorders in children and adolescents. Rev Bras Psiquiatr. 2006;28(1):67-71.
• 10. Murray LK, Fayyad J, Jensen PJ, Hoagwood KH, Azer M, Integrated Services Program (ISP) Task Force. An examination of cross-cultural systems implementing evidence-based assessment and intervention approaches. Rev Bras Psiquiatr. 2006;28(1):76-9.
• 11. So CYC, Hung JSF, Bauermeister JJ, Jensen PS, Integrated Services Program (ISP) Task Force. Training of evidence-based assessment and intervention approaches in cross-cultural contexts: Challenges and solutions. Rev Bras Psiquiatr. 2006;28(1):72-5.
• 12. Integrated Services Program (ISP) Task Force. Disseminating child & adolescent mental health treatment methods: an international feasibility study. Rev Bras Psiquiatr. 2006;28(1):1-2.
• 13. Drake RE, Goldman HH. Evidence-based practices in mental health care. Washington, DC: American Psychiatric Association Press; 2003.
• 14. Kelleher KJ. Use of services and costs for youth with ADHD and related conditions. In: Jensen PS, Cooper JR, editors. Attention deficit hyperactivity disorder: state of the science-best practices. Kingston, NJ: Civic Research Institute; 2002. Ch 27:1-12.
• 16. Kolko DJ. Clinical monitoring of treatment course in child physical abuse: psychometric characteristics and treatment comparisons. Child Abuse Negl. 1996;20(1):23-43.
• 17. Friesen BJ, Koroloff NM. Family-centered services: implications for mental health administration and research. J Ment Health Adm. 1990;17(1):13-25.
• 18. Hoagwood KE. Family-based services in children's mental health: a research review and synthesis. J Child Psychol Psychiatry. 2005;46(7):690-713.
• 19. Burns BJ, Costello EJ, Angold A, Tweed D, Stangl D, Farmer EM, Erkanli A. Children's mental health service use across service sectors. Health Aff (Millwood) 1995;14(3):147-59.
• 20. Hoagwood K, Burns BJ, Kiser L, Ringeisen H, Schoenwald SK. Evidence-based practice in child and adolescent mental health services. Psychiatr Serv. 2001;52(9):1179-89.
• 21. Mufson LH, Dorta KP, Olfson M, Weissman MM, Hoagwood K. Effectiveness research: transporting interpersonal psychotherapy for depressed adolescents (IPT-A) from the lab to school-based health clinics. Clin Child Fam Psychol Rev. 2004;7(4):251-61.
• 22. Burns BJ, Hoagwood K. Evidence-based practice, part I: research update. Child Adolesc Psychiatr Clin N Am. 2004;13(4):xi-xiii.
• 23. Hoagwood KE, Burns BJ. Evidence-based practice, part II: effecting change. Child Adolesc Psychiatr Clin N Am. 2005;14(2):xv-xvii.
• 24. Goldman HH. Deinstitutionalization and community care: social welfare policy as mental health policy. Harv Rev Psychiatry. 1998;6(4):219-22.
• 25. Rogers EM. Diffusion of innovations. 4th ed. New York, NY: Free Press; 1995.
• 26. Van de Ven AH, Polley DE, Garud R, Ventkataraman S. The innovative journey. Oxford, New York: Oxford University Press; 1999.
• 27. Greenhalgh T, Robert G, MacFarland F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581-629.

*

Members of the ISP Task Force are listed in the Introduction to the Special Section: Peter S Jensen, The ISP Task Force. Disseminating child & adolescent mental health treatment methods: an international feasibility study.

Publication Dates

• Publication in this collection
  24 Mar 2006
• Date of issue
  Mar 2006

History

• Accepted
  20 Dec 2005
• Received
  21 Sept 2005