On-line version ISSN 1809-452X
Rev. Bras. Psiquiatr. vol.28 no.1 São Paulo Mar. 2006
Exame dos sistemas transculturais que implementam estratégias de intervenção e avaliação baseadas em evidências
Laura K MurrayI; John FayyadII; Peter S JensenIII; Kimberly HoagwoodIV; Mary AzerV; Integrated Services Program Task Force*
IBoston University School of Public Health, Applied Mental Health Research Group, Boston, MA, USA
IIDepartment of Psychiatry and Clinical Psychology, St George Hospital University Medical Center, Balamand University Faculty of Medicine, Beirut, Lebanon
IIICenter for the Advancement of Children's Mental Health,Columbia University, New York, (NY), USA
IVState Office of Mental Health, Columbia University, New York (NY), USA
VChild Mental Health Clinic, Alexandria University, Alexandria, Egypt
Implementation of evidence-based assessment and intervention approaches for youth with behavioral and/or emotional problems is rising to recognition worldwide. Feasibility research is critical to examine what characteristics of systems allow for success or barriers to the implementation of evidence-based practices into real-world settings, especially when working cross-culturally. This paper briefly reviews the experience of 4 international sites to understand how the overall structure and specific site variables directed the implementation of the World Health Organization and the World Psychiatry Association project. Discussion includes a thematic summary of the successes and challenges experienced by the sites, and future directions of feasibility studies.
Keywords: Evidence-based medicine; Outcome and process assessment (Health care); Feasibility studies; Cross-cultural comparison; Case-control study
A implementação de estratégias de intervenção e avaliação baseadas em evidências para jovens com problemas comportamentais e/ou emocionais está assumindo reconhecimento internacional. Estudos de praticabilidade são críticos para examinar quais características dos sistemas permitem êxito ou se constituem em barreiras para a implementação de práticas baseadas em evidências em instituições do mundo real, especialmente quando trabalhando de forma transcultural. Este estudo revisa brevemente a experiência de quatro localidades internacionais para compreender como a estrutura geral e as variáveis específicas da localidade orientaram a implementação do projeto da Organização Mundial de Saúde e da Associação Mundial de Psiquiatria. A discussão inclui um resumo temático dos êxitos e desafios vivenciados pelas localidades e os caminhos futuros de estudos de praticabilidade.
Descritores: Medicina baseada em evidências; Avaliação de processos e resultados (Cuidados da saúde); Estudos de praticabilidade; Comparação transcultural; Estudo de caso controle
The large disparities between the mounting evidence of scientifically-based mental health treatments ("evidence-based treatments" or EBTs) and actual practice are well-documented.1-4 Increasingly, researchers and clinicians alike must grapple with questions about the applicability and feasibility of implementing EBTs across widely varied sites.5-6 Some of the most pressing questions pertain to what type of organizations are "ready" for EBT implementation,7 and relatedly, how cultural factors may affect the generalizability and implementation of EBTs.5-6 Few studies have sought to evaluate these questions internationally, where the challenges may be multiplied by a host of logistic, cultural, and organizational variables. Feasibility studies, particularly when working cross-culturally, are an important first step,8-9 and can lend valuable information on the types of organizational adaptations necessary to enable successful implementation of EBTs into real-world settings.7,10
As described in other papers in this edition,11-14 the Integrated Services Program (ISP) Task Force of the World Psychiatric Association's Presidential Global Program on Child Mental Health sought to implement a feasible yet comprehensive clinical service delivery program for children with internalizing and externalizing disorders, applicable across different countries and cultures. Specific therapies were chosen due to the evidence base for their reducing youth symptomatology,15 and were adapted for this project.13 Four international sites were chosen to participate based on their willingness to pilot the treatments and help investigate the implementation process. These sites include Brazil, Egypt, Israel, and Lebanon, each distinct in culture, population, setting, and staffing. As these organizational variables are likely to significantly affect the feasibility of implementing EBTs, this study represents an innovative effort to examine program implementation across varied international sites.
This paper briefly reviews each site's overall site structure, services available, implementation design, and specific challenges and successes, followed by discussion of cross-site themes relevant to international feasibility efforts.
Site case studies
This site includes the Child and Adolescent Psychiatric Division of the Hospital de Clínicas de Porto Alegre, located within the teaching hospital of the Universidade Federal do Rio Grande do Sul. The staff is highly trained with four professors and three clinical instructors of child psychiatry. Also, as a teaching hospital with a residency program, more than 10 child psychiatrists work as supervisors for residents. A strong commitment to research exists within the division.
The Department of Psychiatry and the Division of Child Psychiatry made available any additional infrastructure needed to conduct this work. A team was formed consisting of a senior psychiatrist, a supervisor (senior psychologist), and four psychologists. This project was implemented within an outpatient Attention Deficit Hyperactivity Disorder (ADHD) clinic. This clinic serves both children and adults, and uses standardized assessment measures as a routine practice. Participants were recruited from all patients seen at that clinic. Cases were assigned to therapists with extensive training in CBT (n = 2) or therapists with no formal training (n = 2) to test feasibility according to the therapist's experience. The following patient profile was chosen: age between 8 and 14 years-old, IQ³ 80, ADHD combined type, no comorbidity other than anxiety disorders or ODD, already receiving ADHD medication, and having at least one available parent or guardian) (Total n = 6). All four psychologists applied the externalizing manual to groups of 3 patients.
3) Challenges and successes
Recruitment was easy given the ADHD clinic's capacity and the high demand for services. This site reported good acceptability of psychosocial interventions for children in their culture.This site was already successfully using one questionnaire, the Strengths and Difficulties Questionnaire (SDQ). Given the existing infrastructure and commitment to research, no significant challenges with implementation of this EBT were noted.
This site is embedded within Alexandria University and its Child Mental Health Clinic. This Clinic is dedicated to training of local pediatricians and psychiatrists, who go on to work in the clinic itself as well as in schools. Staff works closely with the Minister of Education to train teachers in early detection and screening of children's mental health issues. Within this setting, the national school health program provides free mental health services to all school children, resulting in large numbers of youth seen in schools. Any child deemed as needing additional services, such as psychopharmacology, family therapy, day treatment or medical care is referred to the clinic. The Clinic regularly utilizes standardized assessment measures in intelligence, personality, behavior, and mood.
A meeting was held with the Ministry of Education and the Ethical Committee of the Child Mental Health Association to explain the aim of the intervention program, as well as the school where the ISP was to be implemented. Meetings involved administrators and teachers. Project staff included a professor of child psychiatry and mental health and two senior psychologists.
One school site in an overpopulated poor district of Alexandria was selected to implement the EBT. School staff were trained on how to identify children and administer the SDQ for all 6th graders (n = 205). Those with elevated scores were divided into groups of 4-6 children each: four groups for ADHD, two groups for Conduct Disorder (CD), and three groups for internalizing problems (depression and anxiety).
3) Successes and challenges
Many of the challenges experienced by the Egypt site are common among school-based implementation projects, including difficulties in recruiting children due to the heavy academic curriculum and lack of time in the school day. At times, therapists had to wait until the end of the school day to see participants. A greater challenge was recruitment of families. Since children were seen during the school day or immediately after, families had little to no incentive to attend, or were not available due to other obligations.
A key observation made by this site was the importance of "buy-in" from the teachers who had to be coached in using the materials and behavioral classroom techniques, before fully gaining their trust and cooperation.
This site is at the Feinberg Department of Psychiatry at Schneider Children's Medical Center of Israel, a funded tertiary children's hospital affiliated with Sackler School of Medicine at Tel Aviv University. The department has many projects at community satellite locations, including a Bedouin Village in northern Israel with a population of 6,000 people where this project was conducted. The psychological services in this village are limited, consisting of only two school psychologists who must cover twelve nurseries, two elementary, and one high school.
Meetings were held with the supervisor of Bedouin education in the Ministry of Education in Israel, to explain the project and to obtain permission from school administrators and teachers. Additional meetings were offered to parents to explain the intervention. The Psychiatry department staff worked collaboratively with the onsite team, which was coordinated by an Arab-Israeli psychologist of Bedouin origin and was implemented by him, a school psychologist, and a school counselor.
Twelve children were identified with the help of local teachers and the two psychologists. Of these, six children were chosen due to the presence of low-to-moderate externalizing behavior problems and expected motivation from their families. Group therapy sessions with children were conducted at school, and parent sessions at the community center. No additional services such as psychopharmacology were available locally, so more serious cases were referred to the Children's Hospital.
3) Successes and challenges
A previously established relationship with the village leaders and their inclusion in the project planning and implementation appeared to lead to successful implementation. However, the site experienced initial challenges in recruitment of mental health professionals to work on the project, a difficulty partly related to budget constraints and site distance. Another problem was noted in working with teachers, suggesting that future efforts might increase their focus on teachers to raise their involvement.
One significant difficulty concerned parental involvement, primarily concerning the length of intervention. Parents from this village preferred longer but fewer sessions. In addition, in this type of traditional society, psychological services were not trusted, and stigma, such as parental failure and shame was attached to participation. More difficulties in recruiting fathers than mothers were also noted, possibly a function of the traditional role of men within this culture.
This site is at the Department of Psychiatry and Clinical Psychology, St George Hospital University Medical Center, Balamand University Faculty of Medicine in Beirut. The program took place at a specialized child psychiatry and psychology clinic within the department, possessing the capability to treat severe cases, including treatment failures from other settings. The team implementing child services consists of one child psychiatrist, 2 child psychologists, and one social worker. Validated assessment measures are routinely used within the clinic for research, diagnosis, and treatment planning. There is a strong commitment to research within the department.
The project was implemented in two outpatient clinic settings in Beirut. The protocols required by the program were easily incorporated within the existing clinical structure where evaluations, individual and family therapy were already being provided.
Participants were recruited from children presenting to the clinic for evaluation and treatment. Sessions were administered individually by a psychologist and a social worker to nine mild or moderate cases of ADHD, combined type; five had comorbid ODD. Medication use was allowed and family participation was mandatory.
3) Challenges and successes
No administrative barriers were noted to incorporating the program at this site, but delays in recruitment took place due to unexpected circumstances. Following the assassination of a leading prime minister and series of random bombs, there were grave concerns about security and the economy. Families were afraid to come to the clinic or to commit to regular therapy, thus only severe cases came to the clinic, leading to difficulty finding cases initially. In order not to lose therapeutic gains, contact with enrolled families who interrupted therapy due to security concerns was continued by maintaining phone contact. Once the political situation stabilized it became easier to recruit cases and enrollment into the program continued.
Another challenge encountered at this site was the fee-for service structure of the health care system In Lebanon insurance does not cover psychiatric and psychological treatment so families paid for service out of pocket. The site initiated discounted therapy rates for this program to help mitigate this.
The therapy program was very well accepted by parents and children and was successfully implemented despite the challenges.
This paper reviewed four diverse international sites with varying systems and processes of implementing the ISP project. Across sites, successful implementation within systems was aided by prior positive relations with stakeholders, staff, and/or administrators within the site location. Also, the availability of consistent fiscal support for the project was related to ease of recruitment and general acceptability of the project among staff.
Common themes also emerged within school-based sites. One challenge of implementing EBTs in schools was parental involvement and teacher "buy-in". Both school-based sites (Tel Aviv and Alexandria) suggested that future implementation studies should provide additional training to teachers to facilitate their involvement, perhaps augmented by giving them some other tangible benefits. Another shared theme was difficulty in securing parental participation due to setting and/or cultural roles.
Much can also be learned by the differences of implementation across systems. Possibly the largest contrast can be made between Brazil and the other three sites, wherein Brazil had an existing funded research infrastructure staffed with professionals which allowed for a more rigorous design. Thirdly, the EBTs were delivered either individually or in groups, indicating that implementation may need to be flexible in its delivery. Recruitment in these four different settings was not a major challenge, except for Lebanon in the initial stages due to security problems, highlighting the need to pay attention to socio-political aspects of systems.
This study, although small, represents an effort to examine implementation work across diverse systems and cultures. An overview of the sites participating in this feasibility study sheds light on some of the multiple factors that need to be considered in cross-cultural implementation research. Organizational structures are diverse and may include socio-political issues and variations in the availability of trained professionals. Cultural factors, such as mistrust of psychological services or the roles of certain family members, also have to be considered when such factors might constitute obstacles.
Given the uniqueness of sites' implementation process, general guidelines (e.g., rather than a step-by-step) of how to work through implementation challenges may be especially important for international work on implementing EBTs, and further emphasizes the importance of initial feasibility work.8-9
See complete acknowledgements in the Introduction to the Special Section, this issue. The authors would also like to acknowledge the efforts and dedication of the clinicians and therapists who contributed to this project, including providing comments on this paper: Lynn Farah and Youmna Cassir, Beirut (Lebanon); and Doa Habib (Alexandria, Egypt).
1. Drake RE, Rosenberg SD, Teague GB, Bartels SJ, Torrey WC. Fundamental principles of evidence-based medicine applied to mental health care. Psychiatr Clin North Am. 2003;26(4):811-20, vii. [ Links ]
2. Presidents' New Freedom Commission on Mental Health. Achieving the promise: transforming mental health care in America. [cited 15 oct 2005] Washington (DC): Department of Health and Human Services; 2003. Available from: URL: http://www.mentalhealthcommission.gov/reports/FinalReport/toc.html. [ Links ]
3. National Advisory Mental Health Council. Bridging science and service: a Report by the National Advisory Mental Health Council's Clinical Treatment and Services Research Workgroup. [cited 15 oct 2005] NIH Publication No. 99-4353, 1999. Available from: URL: http://www.nimh.nih.gov/publicat/mhservices.cfm. [ Links ]
4. US Public Health Service. Report of the Surgeon General's Conference on Children's Mental Health: A National Action Agenda. [cited 20 oct 2005]. Washington, DC: Department of Health and Human Services; 2000. Available from: URL: http://www.hhs.gov/surgeongeneral/topics/cmh/cmhreport.pdf. [ Links ]
5. Drake RE, Goldman HH, Leff HS, Lehman AF, Dixon L, Mueser KT, Torrey WC. Implementing evidence-based practices in routine mental health service settings. Psychiatr Ser. 2001;52(2):179-82. [ Links ]
6. Essock SM, Goldman HH, Van Tosh L, Anthony WA, Appell CR, Bond GR, Dixon LB, Dunakin LK, Ganju V, Gorman PG, Ralph RO, Rapp CA, Teague GB, Drake RE. Evidence-based practices: setting the context and responding to concerns. Psychiatr Clin North Am. 2003;26(4):919-38, ix. [ Links ]
7. Schoenwald SK, Hoagwood K. Effectiveness, transportability, and dissemination of interventions: what matters when? Psychiatr Serv. 2001;52(9):1190-7. [ Links ]
8. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321(7262):694-6. [ Links ]
9. Power R, Langhaug LF, Nyamurera T, Wilson D, Bassett MT, Cowan FM. Developing complex interventions for rigorous evaluation - a case study from rural Zimbabwe. Health Educ Res. 2004;19(5):570-5. [ Links ]
10. Hoagwood K, Hibbs E, Brent D, Jensen P. Introduction to the special section: Efficacy and effectiveness in studies of child and adolescent psychotherapy. J Consult Clin Psychology. 1995;63:683-7. [ Links ]
11. Jensen PS, The Integrated Services Program Taskforce. Disseminating child & adolescent mental health treatment methods: an international feasibility study. Rev Bras Psiquiatr. 2006;28(1):1-2. [ Links ]
12. Hoagwood KH, Kelleher K, Murray LK, The Integrated Services Program Taskforce. Implementation of evidence-based practices for children in four countries: a project of the World Psychiatric Association. Rev Bras Psiquiatr. 2006;28(1):59-66. [ Links ]
13. Bauermeister JJ, So C, Jensen PJ, Krispin O, Seif El Din A, The Integrated Services Program Taskforce. Development of adaptable and flexible treatment manuals for externalizing and internalizing disorders in children and adolescents. Rev Bras Psiquiatr. 2006;28(1):67-71. [ Links ]
14. So C, Hung JSF, Bauermeister JJ, Jensen PJ, Habib D, Knapp P, Krispin O, The Integrated Services Program Taskforce. Training of evidence-based assessment and intervention approaches in cross-cultural contexts: challenges and solutions. Rev Bras Psiquiatr. 2006;28(1):72-5. [ Links ]
15. Butler AC, Chapman JE, Forman EM, Beck AT. The empirical status of cognitive-behavioral therapy: a review of meta-analyses. Clin Psychol Rev. 2005;26(1):17-31. Epub 2005 Sep. 30. [ Links ]
Laura K. Murray
Boston University School of Public Health
Center for International Health and Development
85 E. Concord Street, 5th Floor
Boston, MA 02118
Phone: 617-414-1154 Fax: 617-414-1261
Submitted: 21 September 2005
Accepted: 20 December 2005
Financing: World Psychiatric Association (WPA)
Conflict of interests: None
* Members of the ISP Task Force are listed in the Introduction to the Special Section: Peter S Jensen, The ISP Task Force. Disseminating child & adolescent mental health treatment methods.