Acessibilidade / Reportar erro

Evaluation of the Implementation Process of the #Tamojunto2.0 Prevention Program in Brazilian Schools

Avaliação do Processo de Implementação do Programa Preventivo #Tamojunto2.0 em Escolas Brasileiras

Evaluación del Proceso de Implementación del Programa Preventivo #Tamojunto2.0 en Escuelas Brasileñas

Abstract

The #Tamojunto2.0 program (Unplugged) has been evaluated in Brazil in order to support its implementation as a national public policy. The aim of this study was to evaluate the implementation process of the #Tamojunto2.0 school-based program, disseminated by the Ministry of Health, in the prevention of alcohol and other drug use among adolescents. This is a mixed-methods study. The sample consisted of 13 teachers and eight managers of 8th grade classes of elementary education public schools located in two cities, and a training provider professional for the program’s application. Observation, evaluation and semi-structured interview forms were used. Descriptive and thematic analyses were performed. The results indicate low implementation fidelity, good quality in the application of the program, and high student absenteeism. Aspects which facilitate or hinder implementation of the program are also presented. Such results are relevant for the large-scale implementation of #Tamojunto2.0.

Keywords:
implementation science; program evaluation; drug abuse prevention; adolescents; schools

Resumo

O programa #Tamojunto2.0 (Unplugged) tem sido avaliado no Brasil com vistas a subsidiar sua implantação como política pública nacional. O objetivo deste estudo foi avaliar o processo de implementação do programa escolar #Tamojunto2.0, disseminado pelo Ministério da Saúde, na prevenção do uso de álcool e outras drogas entre adolescentes. Trata-se de estudo de métodos mistos. A amostra constituiu-se de 13 professores e oito gestores de turmas de 8º ano do ensino fundamental de escolas públicas localizadas em duas cidades, e uma profissional ministrante da formação para aplicação do programa. Foram utilizados formulários de observação, avaliação e roteiro de entrevista semiestruturada, os dados coletados foram submetidos às análises descritiva e temática. Os resultados indicam baixa fidelidade de implementação, boa qualidade na aplicação do programa, alto absenteísmo dos estudantes. São também apresentados aspectos que facilitam ou dificultam sua implementação. Tais resultados são relevantes para a implementação do #Tamojunto2.0 em larga-escala.

Palavras-chave:
ciência da implementação; avaliação de programa; prevenção do abuso de drogas; adolescentes; escolas

Resumen

El programa # Tamojunto2.0 (Unplugged) ha sido evaluado en Brasil con miras a subsidiar su implementación como política pública nacional. El objetivo del estudio fue evaluar el proceso de implementación del programa escolar #Tamojunto2.0, difundido por el Ministerio de Salud, en la prevención del consumo de drogas en adolescentes. Es un estudio de métodos mixtos. La muestra se constituyó de 13 maestros y ocho directores de clases del 8º grado de Enseñanza Fundamental en escuelas públicas de dos ciudades, y un profesional provedor de capacitación para la aplicación del programa. Se utilizaron formularios de observación, evaluación y entrevista semiestructurada. Los datos se sometieron a análisis descriptivo y temático. Los resultados indican baja fidelidad en la implementación, buena calidad en la aplicación del programa, alto ausentismo estudiantil. También se presentan aspectos que facilitan o dificultan la implementación del programa. Estos resultados son relevantes para la implementación a gran escala de #Tamojunto2.0.

Palabras clave:
ciencia de la implementación; evaluación de programa; prevención en el abuso de drogas; adolescentes; escuelas

The implementation of psychosocial preventive programs or interventions based on scientific evidence has been an important way to address social and health issues and support public policies worldwide. Evidence on the efficacy and effectiveness of school programs has been produced since the 1980s, with notable methodological advances reported.

Several researchers (Bradshaw & Pas, 2011Bradshaw, C. P., & Pas, E. T. (2011). A statewide scale up of positive behavioral interventions and supports: A description of the development of systems of support and analysis of adoption and implementation. School Psychology Review, 40(4), 530-548. doi:10.1080/02796015.2011.12087528
https://doi.org/10.1080/02796015.2011.12...
; Moullin, Dickson, Stadnick, Rabin, & Aarons, 2019Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019). Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implementation Science , 14, 1. doi:10.1186/s13012-018-0842-6
https://doi.org/10.1186/s13012-018-0842-...
) have documented that there are more evidence-based programs being developed and evaluated than being implemented with due fidelity in schools for at least two decades. Thus, a necessary approximation between research and practice is evidenced in a movement in which both affect each other, or at least they should.

In a move to elucidate this mismatch, studies which evaluate the program implementation process have sought to understand how programs work in the real world (Mihalic, Fagan, & Argamaso, 2008Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Implementing the LifeSkills Training drug prevention program: Factors related to implementation fidelity. Implementation Science , 3, 5. doi:10.1186/1748-5908-3-5
https://doi.org/10.1186/1748-5908-3-5...
; Morrison, Newman, & Erickson, 2021Morrison, J. Q., Newman, D. S., & Erickson, A. G. (2021). Process evaluation of literacy practices within a multi-tiered system of supports framework. Journal of Applied School Psychology, 37(2), 140-164. doi:10.1080/15377903.2020.1804030
https://doi.org/10.1080/15377903.2020.18...
). Research for this purpose includes a feasibility assessment, which precedes the dissemination stage, and has the function of identifying the capacity of the parties involved and the environment to host the program (Bowen et al., 2009Bowen, D. J., Kreuter, M., Spring, B., Cofta-Woerpel, L., Linnan, L., Weiner, D., … Fernandez, M. (2009). How we design feasibility studies. American Journal of Preventive Medicine, 36(5), 452-457. doi:10.1016/j.amepre.2009.02.002
https://doi.org/10.1016/j.amepre.2009.02...
). It is known that integrating the program with infrastructure and school dynamics (for example) is an essential component for its choice, implementation, dissemination and sustainability by the community. In this sense, when the school team recognizes that a program will bring benefits to their institution, that they believe in it and that it is viable, they then engage in its implementation (Mercer, McIntosh, & Hoselton, 2017Mercer, S. H., McIntosh, K., & Hoselton, R. (2017). Comparability of fidelity measures for assessing tier 1 school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions, 19(4), 195-204. doi:10.1177/1098300717693384
https://doi.org/10.1177/1098300717693384...
).

In turn, studies on the fidelity in applying programs (Horner, Sugai, & Fixsen, 2017Horner, R. H., Sugai, G., & Fixsen, D. L. (2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20(1), 25-35. doi:10.1007/s10567-017-0224-7
https://doi.org/10.1007/s10567-017-0224-...
; Throuvala, Griffiths, Rennoldson, & Kuss, 2019Throuvala, M. A., Griffiths, M. D., Rennoldson, M., & Kuss, D. J.(2019). School-based prevention for adolescent internet addiction: Prevention is the key: A systematic literature review. Current Neuropharmacology, 17(6), 507-525.doi:10.2174/1570159X16666180813153806
https://doi.org/10.2174/1570159X16666180...
) suggest that it is necessary for the school management to develop a collective culture which is in tune with the core values of the program being implemented in order to ensure implementation as planned in its essential aspects. The authors also point out that specific measures could be planned to overcome difficulties with low-fidelity implementation, such as intensive support from local coordination or providing additional evidence of program effectiveness. Other aspects such as the program scope, dose, complexity and structure also influence implementation of the intervention by the school community (Dusenbury & Hansen, 2004Dusenbury, L., & Hansen, W. B. (2004). Pursuing the course from research to practice. Prevention Science, 5(1), 55-59.doi:10.1023/B:PREV.0000013982.20860.19
https://doi.org/10.1023/B:PREV.000001398...
; Leeman, Birken, Powell, Rohweder, & Shea, 2017Leeman, J., Birken, S. A., Powell, B. J., Rohweder, C., & Shea, C. M. (2017). Beyond implementation strategies: Classifying the full range of strategies used in implementation science and practice. Implementation Science, 12, 125. doi:10.1186/s13012-017-0657-x
https://doi.org/10.1186/s13012-017-0657-...
).

A study conducted by Horner et al. (2017Horner, R. H., Sugai, G., & Fixsen, D. L. (2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20(1), 25-35. doi:10.1007/s10567-017-0224-7
https://doi.org/10.1007/s10567-017-0224-...
) identified variables relevant to the implementation process (exploration [initial adoption], installation, initial implementation, full implementation, innovation and sustainability) of the SWPBIS Program in seven US states. Each state follows an implementation schedule and its schedule depends on the local capacity to provide training, monitoring/follow-up, evaluation, qualified technical support and availability of its own resources, independently of state funding or subsidies. Building local capacity has repeatedly been seen as an efficiency-enhancing mechanism for implementing and sustaining interventions over time. In this direction, it is essential that the training of implementers be evaluated in depth, as there is evidence that failures in this process compromise the program implementation (Bartelink et al., 2019Bartelink, N. H. M., van Assema, P., Jansen, M. W. J., Savelberg, H. H. C. M., Moore, G. F., Hawkins, J., & Kremers, S. P. J. (2019). Process evaluation of the healthy primary school of the future: The key learning points. BMC Public Health, 19(1), 698. doi:10.1186/s12889-019-6947-2
https://doi.org/10.1186/s12889-019-6947-...
; Kok et al., 2016Kok, G., Gottlieb, N. H., Peters, G. J. Y., Mullen, P. D., Parcel, G. S., Ruiter, R. A. C., … Bartholomew, L. K. (2016). A taxonomy of behaviour change methods: An Intervention Mapping approach. Health Psychology Review, 10(3), 297-312. doi:10.1080/17437199.2015.1077155
https://doi.org/10.1080/17437199.2015.10...
; Nelson & Madiba, 2020Nelson, C., & Madiba, S. (2020). Barriers to the implementation of the ward-based outreach team program in Mpumalanga Province: Results from process evaluation. Journal of Primary Care and Community Health, 11(1), 1-13. doi:10.1177/2150132720975552
https://doi.org/10.1177/2150132720975552...
).

Horner et al. (2017Horner, R. H., Sugai, G., & Fixsen, D. L. (2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20(1), 25-35. doi:10.1007/s10567-017-0224-7
https://doi.org/10.1007/s10567-017-0224-...
) state that documenting the effectiveness of practices aimed at schools is necessary, but insufficient for their large-scale implementation given the disproportion between the number of practices/programs/interventions that have been shown to work and the number of interventions that are being well used in these institutions. The newly established implementation science points to the need for clearer and more careful appreciation of the complex set of variables that affect the use of these practices (Forman et al., 2013Forman, S. G., Shapiro, E. S., Codding, R. S., Gonzales, J. E., Reddy, L. A., Rosenfield, S. A., … Stoiber, K. C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28(2), 77-100. doi:10.1037/spq0000019
https://doi.org/10.1037/spq0000019...
).

A program called #Tamojunto was implemented in the Brazilian national context, constituting an adaptation of the European Unplugged program (Faggiano et al., 2008Faggiano, F., Galanti, M. R., Bohrn, K., Burkhart, G., Vigna-Taglianti, F., Cuomo, L., … The EU-Dap Study Group. (2008). The effectiveness of a school-based substance abuse prevention program: EU-Dap cluster randomised controlled trial. American Journal of Preventive Medicine , 47(5), 537-543. doi: 10.1016/j.ypmed.2008.06.018
https://doi.org/10.1016/j.ypmed.2008.06....
), aimed at preventing the use of alcohol and other drugs among adolescents. It is an intervention consisting of 12 classes with an expected duration of 50 minutes each which are held during the school period and aim to develop the following skills: life skills, psychoeducation about alcohol and drugs and elucidation about the role of normative beliefs (Ministry of Health - Ministério da Saúde, 2018Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Ações Programáticas Estratégicas. (2018). Programa #Tamojunto: Prevenção na escola: Guia do professor [#Tamojunto Program: Prevention at school: Teacher’s guide]. Brasília, DF: Autor. Retrieved from https://bvsms.saude.gov.br/bvs/publicacoes/programa_tamojunto_prevencao_escola_guia_professor.pdf
https://bvsms.saude.gov.br/bvs/publicaco...
). Data from the Brazilian National School Health Survey (Pesquisa Nacional de Saúde do Escolar - PeNSE) indicate that 63.3% of schoolchildren aged between 13 and 17 have already tried alcoholic beverages, 22.6% have consumed cigarettes, and 13% have used illicit drugs (Brazilian Institute of Geography and Statistics - Instituto Brasileiro de Geografia e Estatística [IBGE], 2021Instituto Brasileiro de Geografia e Estatística. (2021). Pesquisa Nacional de Saúde do Escolar: 2019 [National Adolescent School-based Health Survey: 2019]. Rio de Janeiro, RJ: Autor. Retrieved from https://biblioteca.ibge.gov.br/visualizacao/livros/liv101852.pdf
https://biblioteca.ibge.gov.br/visualiza...
).

Medeiros, Pereira, Schneider and Sanchez (2018Medeiros, P. F. P., Pereira, A. P. D., Schneider, D. R., & Sanchez, Z. M. (2018). Percepções da comunidade escolar sobre a implantação do programa unplugged em escolas [School community perceptions on implementation of the unplugged program in schools]. Psicologia Escolar e Educacional, 22(1), 173-184. doi:10.1590/2175-35392018014256
https://doi.org/10.1590/2175-35392018014...
) conducted a process evaluation regarding the program’s fidelity, acceptability and feasibility to better understand the aspects involved in the program’s implementation course in the Brazilian context. The 12 classes of the program were applied in 94% of the classrooms. However, the number of activities per class had to be reduced and the standard teaching schedule had to be restructured to ensure that regular academic content could still be taught. Additionally, teachers reported having to exclude some activities to provide proper program class content in their 50-minute curriculum classes. Most teachers and students had positive perceptions about the program and reported perceived changes in the classroom environment.

Based on the need for reformulations in #Tamojunto, which subsequently became known as #Tamojunto2.0, a new randomized controlled trial (RCT) was carried out and it was considered relevant to also conduct a process evaluation study with more elements than in that by Medeiros et al. (2016), including observations of applying the 12 classes of the program by teachers of 8th grade classes of Elementary School II. In this context, two research questions were raised, namely: (1) How was #Tamojunto2.0 implemented in participating schools? (2) Is the implementation of #Tamojunto2.0 feasible in public schools? This last question breaks down into two: (2.1) What barriers to the implementation of #Tamojunto2.0 were perceived by school teams? (2.2) What aspects were perceived by school teams as facilitators for implementing #Tamojunto2.0?

Thus, the objective of this study was to evaluate the implementation process of the #Tamojunto2.0 school program disseminated by the Ministry of Health for the prevention of the use of alcohol and other drugs among adolescents. Specifically, the study evaluated: quality and content of teacher training, program implementation fidelity, and program implementation feasibility by public schools.

Method

This is a mixed methods study with the data collected during implementation of #Tamojunto2.0 (Unplugged). The data was collected from about 1/3 of the sample of schools in the experimental group of the randomized controlled trial (RCT) which investigated the program’s effectiveness, in which the present study is nested. The RCT sample consisted of 5,371 eighth grade students (mean age = 13.20 ±0.84) from 73 public schools (37 in the intervention group and 36 in the control group) from three Brazilian cities: São Paulo (SP), Eusébio (CE) and Fortaleza (CE). The results of this study are described in Sanchez et al. (2021Sanchez, Z. M., Valente, J. Y., Galvão, P. P., Gubert, F. A., Melo, M. H. S, Caetano, S. C., ... Cogo-Moreira, H. (2021). A cluster randomized controlled trial evaluating the effectiveness of the school-based drug prevention program #Tamojunto2.0. Addiction (Abingdon, England), 116(6), 1580-1592. doi:10.1111/add.15358
https://doi.org/10.1111/add.15358...
).

Participants

The sample consisted of 13 teachers and eight managers (directors and pedagogical coordinators) of 8th grade elementary school classes in 10 public schools located in two Brazilian capitals, and a professional who provided training for applying the program. The selection of schools (five from the city of São Paulo and five from the city of Fortaleza) was performed by a random draw based on the final list of schools that had accepted to participate in the RCT. Two classes from each school were selected by convenience in order to optimize the hours of the observers in the schools. Thus, eight teachers and three managers participated in São Paulo, and five teachers and five managers participated in Fortaleza.

Instruments

Observation registration form - training workshop. This was prepared by the authors and completed by the research team during non-participant observation of teacher training based on the indications of Danna and Matos (1982Danna, M. F., & Matos, M. A. (1982). Ensinando observação: Uma introdução [Teaching observation: An introduction]. São Paulo, SP: Edicon.). The information recorded included: total duration of training; number of participants; program classes that were presented in the training; schedule; organization of training; theoretical matrix of the program; practical information about the program application; working atmosphere and general impressions of the observer about the workshop, with emphasis on the participation of teachers and the didactics of the trainer.

Training workshop evaluation form. This form was prepared by the authors and completed by the teacher training course after the event ended. It is composed of open and closed questions aimed at investigating the contents presented, mastery of the theoretical basis of #Tamojunto2.0 and appreciation of the trainer in relation to their own performance.

Semi-structured interview script (feasibility). The script was prepared by the authors, and is composed of questions aimed at investigating dates foreseen by schools for the program application, anticipated difficulties, availability of time and engagement of teachers and recommendations of the educational institution regarding the presence of observers in the classroom.

Observation registration form - monitoring the application of #Tamojunto2.0 class by class. This form was prepared by the authors and completed by the research team. The first part of the instrument includes fields for recording the number of students present, total duration of the class, as well as items containing the title of each planned activity and a space to mark yes, no, or not applicable with regard to its respective performance or modifications occurred, and the execution time. In the second part, there are the items of student interest, level of interactivity between students, level of teacher-student interactivity, teacher mastery, and preparation level of the room and material, which involve questions to be answered by the observer with yes (0), not (1), or not applicable (2), and an assessment in general terms on a scale ranging from none (0), low (1), medium (2) and high (3).

Semi-structured interview script (final). This script was prepared by the authors, and consists of questions aimed at investigating the perceptions of teachers and managers in relation to the training received, types of support available for the program application, material used, as well as the main facilitators and barriers related to applying the program in the school context.

Procedures

Data collection. The data collection procedures adopted in the study followed the implementation schedule of the Ministry of Health. The collection started with non-participant observation of teacher training that took place on two consecutive days, totaling 16 hours, and the trainer then filled out the evaluation form of the training workshop at the end of this period.

Feasibility interviews with teachers and school administrators were carried out and audio-recorded before starting the program application in schools. Then, observations were made following the schedule proposed by each school for each of the 12 classes provided for by the program. After completing the application of #Tamojunto2.0 in the classroom, the final interviews were performed and audio-recorded. Data collection was carried out by a team whose members were postgraduate professors or researchers at Master’s and Doctoral levels linked to public universities located in the cities of São Paulo and Fortaleza.

Data analysis. The Microsoft Excel program was used to quantitatively analyze the data. They were analyzed through descriptive statistics considering frequencies and prevalence. A thematic analysis (Braun & Clarke, 2006Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. doi:10.1191/1478088706qp063oa
https://doi.org/10.1191/1478088706qp063o...
) of the interview transcripts of the audio-recorded content was performed regarding the qualitative data. The reports were read repeatedly, coded systematically and grouped into potential themes, which were then reviewed, refined and gathered into categories established by the researchers, generating descriptions and titles for each one.

Ethical Considerations

This study was approved by the Research Ethics Committee of the Secretaria Municipal da Saúde de São Paulo (Opinion No. 2,989,248) - (CAAE No. 91838618.1.3001.0086) in accordance with Resolution No. 466, of December 12, 2012, of the National Health Council.

Results

Quality and content presented in teacher training

Teacher training included the following topics: prevention paradigms, elements related to the effectiveness of preventive programs, Unplugged results in the world, presentation of the theoretical model and methodology used in the program, main characteristics, adaptation, results of the 2014 Brazilian version and aspects which were changed in version 2.0.

The material provided for applying the #Tamojunto2.0 version was presented, along with the program elements - with practical activities and group dynamics - support platforms for implementation, presentation of classes and deepening of its contents. At this point, there was a strong emphasis on the negative effects of alcohol use by teenagers, as well as theoretical issues about drugs presented in the program. There was an important change in this training in relation to the 2014 #Tamojunto - expository content on drugs and their effects was added, something much requested by participants in previous training sessions.

The data referring to both the cities of São Paulo and Fortaleza highlighted that the trainer informed the participants about the theoretical basis of the program, demonstrated good command of its content, explained practical aspects related to the application, as well as enlighteningly answered the doubts of the participating teachers present. However, losses were recorded with regard to the training organization and knowledge demonstration of the trainer in relation to the school universe in order to place itself as a reference for teachers.

The observation records also highlighted the trainer’s good ability to manage the group, as well as the promotion of teachers’ participation, but also the felt need to explore the content related to the parents’ workshops and to divide the training into at least two days. In addition, there was concern regarding teachers about inserting the program in the school curriculum and insecurities in discussing the theme related to drugs.

Program application fidelity analysis

A total of 12 of the 20 classes analyzed in the present study received all the classes provided for in the program, eight of which were in Fortaleza and four in São Paulo. Table 1 shows that a smaller number of the classes in which the classes were held performed all the activities foreseen in each class, and an even smaller number performed all the activities foreseen as described in the manual. It is further possible to observe in Table 1 that there was a significant difference in the data found between the cities, indicating differences in the fidelity of the program implementation in each of them.

Table 1
Number of classes, per city, that held each class, performed all the activities planned in each class and performed each class of the #Tamojunto2.0 program without any change

In Table 2, it is possible to verify how the average, minimum and maximum duration time of each class fluctuated throughout the program’s classes, as well as how there was a great difference in the duration of each class in the schools. Two classes in each city chose to make two classes in a row from the school available to carry out each class of the program, totaling 100 minutes of availability for each one.

Table 2
Average, minimum and maximum duration of each class of the #Tamojunto2.0 program per city

The average attendance between the two cities in the 12 classes of the program for the number of students reached by the program was 72% (with a 63% attendance average in São Paulo and 79% attendance average in Fortaleza). The frequency of student attendance varied greatly between classes in the program, with a minimum registration of 30% of students present in Class 6 in São Paulo, and a maximum registration of 100% of students present in classes 1 and 7 in Fortaleza. Climatic factors, proximity of holidays and school holidays, political moment, among others, led to an oscillation in the attendance number.

All components regarding what was evaluated by the observers for interest, interactivity between students and teacher-students, teacher mastery and classroom preparation in each class of the program were well evaluated throughout the program in both cities, indicating that the teachers applied the program with pedagogical quality.

The data presented here indicate that there were differences in the program application in the two participating cities, mainly in the number of classes in the program. Although there was a difference between the predicted content and what was taught (fidelity), quality was observed in the application considering the questions evaluated here.

Feasibility of implementing the program by schools. Table 3 and Table 4 present examples of reports from teachers and managers of participating schools about facilitating aspects and obstacles to implement #Tamojunto2.0, arranged in categories elaborated from the thematic analysis.

Table 3
Facilitating aspects for implementing the #Tamojunto2.0 program from the perspective of participating schools

Table 4
Obstacles to implementing the #Tamojunto2.0 program from the perspective of participating schools

Common facilitating aspects of implementation for participating cities

School management support. The reports showed that the support offered by schools for teachers to apply the program ranged from full support, support with losses and even lack of support for implementing the program. The support demanded involved above all guaranteeing spaces for activities to take place, making the teachers’ attributions more flexible and collaborating in managing students’ behavior. It is possible to highlight that the program application was completed in all the schools in which the support was positively perceived by the teachers.

Technical support. The teachers’ reports most importantly expressed that the initial training offered by the Ministry of Health proved to be very enlightening, and offered support for the application; however, it would be desirable for this to have a longer duration in order to cover the contents of all the program’s classes. Some participants even suggested the need for continuous technical support and training meetings throughout the intervention implementation. The data showed that part of the teachers considered the training received as sufficient for developing the program, especially those who had previous experience in working with projects, while others did not. Such aspects may be related to differences in repertoire and even to the quality of the teaching received in the course of teacher training and its disparities in the Brazilian context in which resources and difficulties are presented, which must be taken into account when planning training for preventive interventions in order to facilitate the appropriation of knowledge by the participants.

Positive teacher-student interaction. The reports showed that aspects such as the existence of a positive relationship between teachers and students, empathy and teacher knowledge about the social reality of students contributed to developing the program. It is worth noting that the schools that mentioned selecting teachers to participate in #Tamojunto2.0 based on such criteria had the application completed.

Teacher motivation. The reports highlighted that involvement in the program, the desire for it to achieve success and appreciation for the participation in projects on the part of the teachers were fundamental elements for the program to have its application finalized in schools.

Common obstacles to implementation for participating cities

Program complexity. The teachers’ reports highlighted difficulties related to the high level of preparation required for applying the classes, executing activities or constant boosts in the program, understanding the material and the language used. Regarding these last factors, the speeches indicated that the teacher considered it necessary at times to replace the vocabulary present in the program instructions with expressions used more frequently in the students’ routines in order to facilitate their understanding.

Competing demands between school and program. The reports above all included competition between the time allocated to the program’s classes and to the regular subjects of the school curriculum, overlapping demands that the school needs to deal with, such as different projects and other events in the school calendar. More specifically, the participating teachers in educational institutions in São Paulo also referred to the competition between the program application and specific assessments and goals to be met. Reports made it clear that, despite the damage to the curriculum content generated by the time used for applying #Tamojunto2.0, the teachers considered that there were gains provided by the program (teaching of socio-emotional skills, assertiveness, prevention of drug use, reduction of behavior problems); however, they also pointed out that this was not recognized by education management bodies, which were only interested in numbers related to performance in subjects of the regular curriculum.

Cultural differences. The most frequent reports mainly referred to problems in student engagement in the strategies proposed by the program, including exposure difficulties, loss of self-esteem related to social vulnerability conditions, as well as mismatches between activities and interaction modes recommended by the intervention material and the school context of the students, who are not used to making decisions, participating, or playing a leading role in the classroom. The data also highlighted a loss in the receptivity of the theme related to drugs by families and students.

Application time in the classroom. The reports of all the participating teachers emphasized the limited time of 50 minutes for the development of each class of the program as insufficient, which made certain components unfeasible, especially the boosters, throughout the applications. Teachers and managers emphasized that the stipulated time is incompatible with the dynamics of classrooms in the Brazilian context.

Discussion

The present study aimed to evaluate the implementation process of the #Tamojunto2.0 school program disseminated by the Ministry of Health for preventing the use of alcohol and other drugs among adolescents. The results of this study are very much in line with what the literature in the area has been pointing out as factors which significantly affect the implementation of preventive interventions in schools, indicating important elements for the decision-making of the public manager (Horner et al., 2017Horner, R. H., Sugai, G., & Fixsen, D. L. (2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20(1), 25-35. doi:10.1007/s10567-017-0224-7
https://doi.org/10.1007/s10567-017-0224-...
; Leeman et al., 2017Leeman, J., Birken, S. A., Powell, B. J., Rohweder, C., & Shea, C. M. (2017). Beyond implementation strategies: Classifying the full range of strategies used in implementation science and practice. Implementation Science, 12, 125. doi:10.1186/s13012-017-0657-x
https://doi.org/10.1186/s13012-017-0657-...
; Mercer et al., 2017Mercer, S. H., McIntosh, K., & Hoselton, R. (2017). Comparability of fidelity measures for assessing tier 1 school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions, 19(4), 195-204. doi:10.1177/1098300717693384
https://doi.org/10.1177/1098300717693384...
; Throuvala et al., 2019Throuvala, M. A., Griffiths, M. D., Rennoldson, M., & Kuss, D. J.(2019). School-based prevention for adolescent internet addiction: Prevention is the key: A systematic literature review. Current Neuropharmacology, 17(6), 507-525.doi:10.2174/1570159X16666180813153806
https://doi.org/10.2174/1570159X16666180...
).

There is a need to invest in the training quality for implementing #Tamojunto2.0, as teachers tend to apply it better when they have received qualified technical support in both their initial training and throughout the application (Bartelink et al., 2019Bartelink, N. H. M., van Assema, P., Jansen, M. W. J., Savelberg, H. H. C. M., Moore, G. F., Hawkins, J., & Kremers, S. P. J. (2019). Process evaluation of the healthy primary school of the future: The key learning points. BMC Public Health, 19(1), 698. doi:10.1186/s12889-019-6947-2
https://doi.org/10.1186/s12889-019-6947-...
; Mihalic et al., 2008Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Implementing the LifeSkills Training drug prevention program: Factors related to implementation fidelity. Implementation Science , 3, 5. doi:10.1186/1748-5908-3-5
https://doi.org/10.1186/1748-5908-3-5...
). It was evident that teachers feel insecure to address the issue of drugs with their students, indicating the need for continued training of these professionals beyond the #Tamojunto2.0 program in order to also collaborate for greater student participation in the classroom and for improving communication with families. The technical training of the trainer and their experience with the daily life of the school not only enables dialogue with the teaching staff, but also contributes to the future implementation and continuity of the program in the school (Morrison et al., 2021Morrison, J. Q., Newman, D. S., & Erickson, A. G. (2021). Process evaluation of literacy practices within a multi-tiered system of supports framework. Journal of Applied School Psychology, 37(2), 140-164. doi:10.1080/15377903.2020.1804030
https://doi.org/10.1080/15377903.2020.18...
; Nelson & Madiba, 2020Nelson, C., & Madiba, S. (2020). Barriers to the implementation of the ward-based outreach team program in Mpumalanga Province: Results from process evaluation. Journal of Primary Care and Community Health, 11(1), 1-13. doi:10.1177/2150132720975552
https://doi.org/10.1177/2150132720975552...
).

Regarding the implementation fidelity of #Tamojunto2.0, the content applied was below the original proposal in relation to both the program completeness and the changes made, reiterating what the national and international literature has pointed out (Medeiros et al., 2016; Throuvala et al., 2019Throuvala, M. A., Griffiths, M. D., Rennoldson, M., & Kuss, D. J.(2019). School-based prevention for adolescent internet addiction: Prevention is the key: A systematic literature review. Current Neuropharmacology, 17(6), 507-525.doi:10.2174/1570159X16666180813153806
https://doi.org/10.2174/1570159X16666180...
). Insofar as the achievement of the intended results is linked to the previously tested intervention, its use without fidelity to the essential components of the program can bring the program into disrepute, as they will not produce the expected effects and thus schools will stop adopting them and other schools will very likely not select it as a reliable action (Horner et al., 2017Horner, R. H., Sugai, G., & Fixsen, D. L. (2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20(1), 25-35. doi:10.1007/s10567-017-0224-7
https://doi.org/10.1007/s10567-017-0224-...
). Implementing the program faithfully into the routine of schools is a matter of high relevance for implementation science, as it is not enough for the intervention to be effective, it must also be viable in its place of action (Forman et al., 2013Forman, S. G., Shapiro, E. S., Codding, R. S., Gonzales, J. E., Reddy, L. A., Rosenfield, S. A., … Stoiber, K. C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28(2), 77-100. doi:10.1037/spq0000019
https://doi.org/10.1037/spq0000019...
).

The results of the present study also highlight the scope of the intervention, considering the presence of students during the program’s application with an average of 63% and 79% in the participating cities. This finding is in line with those found in previous research regarding the high absenteeism of Brazilian public education students (Organization for Economic Cooperation and Development [Organização para a Cooperação e Desenvolvimento Econômico - OECD], 2016Organização para a Cooperação e Desenvolvimento Econômico. (2016). Brasil no PISA 2015: Análises e reflexões sobre o desempenho dos estudantes brasileiros [Brazil in PISA 2015: Analyzes and reflections on Brazilian students’ performance]. São Paulo, SP: Fundação Santillana. Retrieved from https://download.inep.gov.br/acoes_internacionais/pisa/resultados/2015/pisa2015_completo_final_baixa.pdf
https://download.inep.gov.br/acoes_inter...
), a factor which keeps adolescents away from school life and important learning that the institution can support. In this sense, the excess of absences is a factor which limits the effectiveness of the program (Cook, Dodge, Gifford, & Schulting, 2017Cook, P. J., Dodge, K. A., Gifford, E. J., & Schulting, A. B. (2017). A new program to prevent primary school absenteeism: Results of a pilot study in five schools. Children and Youth Services Review, 82(1), 262-270. doi:10.1016/j.childyouth.2017.09.017
https://doi.org/10.1016/j.childyouth.201...
), but it also goes beyond #Tamojunto2.0, inviting public policy managers to get involved in this issue.

It is clear in this study that the complexity of the program, the program’s requirement for a specific time within the class schedule (already busy with other activities), the excess demands on the school, the institutional and family culture issues are challenges and obstacles which are already not only known in the literature for implementation, but also continuity (Mihalic et al., 2008Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Implementing the LifeSkills Training drug prevention program: Factors related to implementation fidelity. Implementation Science , 3, 5. doi:10.1186/1748-5908-3-5
https://doi.org/10.1186/1748-5908-3-5...
; Nelson & Madiba, 2020Nelson, C., & Madiba, S. (2020). Barriers to the implementation of the ward-based outreach team program in Mpumalanga Province: Results from process evaluation. Journal of Primary Care and Community Health, 11(1), 1-13. doi:10.1177/2150132720975552
https://doi.org/10.1177/2150132720975552...
). The results suggest that support from school management is essential, since in addition to implementing the program, teachers end up having to meet the same demands of the position when joining the program. Thus, new assignments are added without reducing the previous ones. In this sense, school management can help to distribute the workload based on the school’s priorities and projects. The findings indicate that although the program has been updated, implementation challenges pointed out by Medeiros et al. (2018Medeiros, P. F. P., Pereira, A. P. D., Schneider, D. R., & Sanchez, Z. M. (2018). Percepções da comunidade escolar sobre a implantação do programa unplugged em escolas [School community perceptions on implementation of the unplugged program in schools]. Psicologia Escolar e Educacional, 22(1), 173-184. doi:10.1590/2175-35392018014256
https://doi.org/10.1590/2175-35392018014...
) in a study on the first version of #Tamojunto are still there.

In another perspective, the implementation of #Tamojunto2.0 signals facilitators in the same direction as previous research, especially continued technical support, management support, teacher motivation (Forman et al., 2013Forman, S. G., Shapiro, E. S., Codding, R. S., Gonzales, J. E., Reddy, L. A., Rosenfield, S. A., … Stoiber, K. C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28(2), 77-100. doi:10.1037/spq0000019
https://doi.org/10.1037/spq0000019...
). These studies have examined the role of implementers and stakeholders, aspects of school organization and the environment outside the school to successfully implement the intervention. Thus, articulations between different spheres of public administration and sectors such as health, for example, would be necessary to deal with the obstacles and work towards the facilitating aspects. The states, municipalities and the federal government are the political units that establish guidelines and procedures, allocate resources and provide technical assistance that enable projects in the school (Horner et al., 2017Horner, R. H., Sugai, G., & Fixsen, D. L. (2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20(1), 25-35. doi:10.1007/s10567-017-0224-7
https://doi.org/10.1007/s10567-017-0224-...
; Mihalic et al., 2008Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Implementing the LifeSkills Training drug prevention program: Factors related to implementation fidelity. Implementation Science , 3, 5. doi:10.1186/1748-5908-3-5
https://doi.org/10.1186/1748-5908-3-5...
).

It is noteworthy that the results obtained do not disqualify the program, but expose the complexity of building the partnership between the three management levels and between health and education. If the program is adapted to the reality of schools, it has the potential for dissemination, but depends on clear support from school management.

In summary, according to the results of this study and research which has been conducted on the implementation and sustainability of preventive programs (Horner et al., 2017Horner, R. H., Sugai, G., & Fixsen, D. L. (2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20(1), 25-35. doi:10.1007/s10567-017-0224-7
https://doi.org/10.1007/s10567-017-0224-...
; Kok et al., 2016Kok, G., Gottlieb, N. H., Peters, G. J. Y., Mullen, P. D., Parcel, G. S., Ruiter, R. A. C., … Bartholomew, L. K. (2016). A taxonomy of behaviour change methods: An Intervention Mapping approach. Health Psychology Review, 10(3), 297-312. doi:10.1080/17437199.2015.1077155
https://doi.org/10.1080/17437199.2015.10...
), the large-scale implementation of #Tamojunto 2.0 may occur to the extent that managers from different public spheres are able to provide/establish: (1) administrative support (policy to support measures that develop the repertoire of life skills, social skills of students; specific funding for the program); (2) local technical capacity for training and monitoring (local implementers with knowledge of the program content and experience with the school context and organization, as well as formalized data collection and use for public management decision-making); (3) development of evaluation systems (common system to evaluate if the program is being used and if it is being beneficial to students and teachers, thereby enabling the use of collected data for decision-making by local teams).

Additionally, it is worth mentioning some points considered as limitations of the study. A first point is that although the schools were selected by a random draw, the classes to be observed were selected by convenience, requiring greater flexibility and availability from the school. In addition, even though the research and implementation teams were different, one of the members participated in both teams. This was due to difficulties in composing the research team to serve schools in one of the cities. Another point refers to the impossibility of knowing obstacles and implementation possibilities from the point of view of public management due to the time and resources available.

Based on the results presented, some suggestions for the implementation and future research of the #Tamojunto2.0 Program are: strengthening the relationship between Health and Education so that health support for the program effectively occurs; invest in the continuing education of teachers on the subject of alcohol and other drugs so that they feel more prepared for the application; adapt the school’s routine to the program’s needs, considering that the school’s dynamics cannot incorporate, without prejudice to the curriculum, the program activities that require a specific space within the class schedule; introduce devices that increase students’ attendance in school, taking into account the high rates of absenteeism, and improve implementation monitoring.

Given the above, the present study contributes to the body of knowledge on the implementation of school drug use prevention programs insofar as it gives visibility to aspects that are not very visible in research and essential aspects related to the training offered to professionals in the context of implementation, and to the application conducted by teachers in the classroom. These aspects can support the decisions of public managers to improve the large-scale implementation of #Tamojunto2.0 in Brazil.

References

  • Bartelink, N. H. M., van Assema, P., Jansen, M. W. J., Savelberg, H. H. C. M., Moore, G. F., Hawkins, J., & Kremers, S. P. J. (2019). Process evaluation of the healthy primary school of the future: The key learning points. BMC Public Health, 19(1), 698. doi:10.1186/s12889-019-6947-2
    » https://doi.org/10.1186/s12889-019-6947-2
  • Bowen, D. J., Kreuter, M., Spring, B., Cofta-Woerpel, L., Linnan, L., Weiner, D., … Fernandez, M. (2009). How we design feasibility studies. American Journal of Preventive Medicine, 36(5), 452-457. doi:10.1016/j.amepre.2009.02.002
    » https://doi.org/10.1016/j.amepre.2009.02.002
  • Bradshaw, C. P., & Pas, E. T. (2011). A statewide scale up of positive behavioral interventions and supports: A description of the development of systems of support and analysis of adoption and implementation. School Psychology Review, 40(4), 530-548. doi:10.1080/02796015.2011.12087528
    » https://doi.org/10.1080/02796015.2011.12087528
  • Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. doi:10.1191/1478088706qp063oa
    » https://doi.org/10.1191/1478088706qp063oa
  • Cook, P. J., Dodge, K. A., Gifford, E. J., & Schulting, A. B. (2017). A new program to prevent primary school absenteeism: Results of a pilot study in five schools. Children and Youth Services Review, 82(1), 262-270. doi:10.1016/j.childyouth.2017.09.017
    » https://doi.org/10.1016/j.childyouth.2017.09.017
  • Danna, M. F., & Matos, M. A. (1982). Ensinando observação: Uma introdução [Teaching observation: An introduction]. São Paulo, SP: Edicon.
  • Dusenbury, L., & Hansen, W. B. (2004). Pursuing the course from research to practice. Prevention Science, 5(1), 55-59.doi:10.1023/B:PREV.0000013982.20860.19
    » https://doi.org/10.1023/B:PREV.0000013982.20860.19
  • Faggiano, F., Galanti, M. R., Bohrn, K., Burkhart, G., Vigna-Taglianti, F., Cuomo, L., … The EU-Dap Study Group. (2008). The effectiveness of a school-based substance abuse prevention program: EU-Dap cluster randomised controlled trial. American Journal of Preventive Medicine , 47(5), 537-543. doi: 10.1016/j.ypmed.2008.06.018
    » https://doi.org/10.1016/j.ypmed.2008.06.018
  • Forman, S. G., Shapiro, E. S., Codding, R. S., Gonzales, J. E., Reddy, L. A., Rosenfield, S. A., … Stoiber, K. C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28(2), 77-100. doi:10.1037/spq0000019
    » https://doi.org/10.1037/spq0000019
  • Horner, R. H., Sugai, G., & Fixsen, D. L. (2017). Implementing effective educational practices at scales of social importance. Clinical Child and Family Psychology Review, 20(1), 25-35. doi:10.1007/s10567-017-0224-7
    » https://doi.org/10.1007/s10567-017-0224-7
  • Instituto Brasileiro de Geografia e Estatística. (2021). Pesquisa Nacional de Saúde do Escolar: 2019 [National Adolescent School-based Health Survey: 2019]. Rio de Janeiro, RJ: Autor. Retrieved from https://biblioteca.ibge.gov.br/visualizacao/livros/liv101852.pdf
    » https://biblioteca.ibge.gov.br/visualizacao/livros/liv101852.pdf
  • Kok, G., Gottlieb, N. H., Peters, G. J. Y., Mullen, P. D., Parcel, G. S., Ruiter, R. A. C., … Bartholomew, L. K. (2016). A taxonomy of behaviour change methods: An Intervention Mapping approach. Health Psychology Review, 10(3), 297-312. doi:10.1080/17437199.2015.1077155
    » https://doi.org/10.1080/17437199.2015.1077155
  • Leeman, J., Birken, S. A., Powell, B. J., Rohweder, C., & Shea, C. M. (2017). Beyond implementation strategies: Classifying the full range of strategies used in implementation science and practice. Implementation Science, 12, 125. doi:10.1186/s13012-017-0657-x
    » https://doi.org/10.1186/s13012-017-0657-x
  • Medeiros, P. F. P., Pereira, A. P. D., Schneider, D. R., & Sanchez, Z. M. (2018). Percepções da comunidade escolar sobre a implantação do programa unplugged em escolas [School community perceptions on implementation of the unplugged program in schools]. Psicologia Escolar e Educacional, 22(1), 173-184. doi:10.1590/2175-35392018014256
    » https://doi.org/10.1590/2175-35392018014256
  • Mercer, S. H., McIntosh, K., & Hoselton, R. (2017). Comparability of fidelity measures for assessing tier 1 school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions, 19(4), 195-204. doi:10.1177/1098300717693384
    » https://doi.org/10.1177/1098300717693384
  • Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Implementing the LifeSkills Training drug prevention program: Factors related to implementation fidelity. Implementation Science , 3, 5. doi:10.1186/1748-5908-3-5
    » https://doi.org/10.1186/1748-5908-3-5
  • Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Ações Programáticas Estratégicas. (2018). Programa #Tamojunto: Prevenção na escola: Guia do professor [#Tamojunto Program: Prevention at school: Teacher’s guide]. Brasília, DF: Autor. Retrieved from https://bvsms.saude.gov.br/bvs/publicacoes/programa_tamojunto_prevencao_escola_guia_professor.pdf
    » https://bvsms.saude.gov.br/bvs/publicacoes/programa_tamojunto_prevencao_escola_guia_professor.pdf
  • Morrison, J. Q., Newman, D. S., & Erickson, A. G. (2021). Process evaluation of literacy practices within a multi-tiered system of supports framework. Journal of Applied School Psychology, 37(2), 140-164. doi:10.1080/15377903.2020.1804030
    » https://doi.org/10.1080/15377903.2020.1804030
  • Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019). Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implementation Science , 14, 1. doi:10.1186/s13012-018-0842-6
    » https://doi.org/10.1186/s13012-018-0842-6
  • Nelson, C., & Madiba, S. (2020). Barriers to the implementation of the ward-based outreach team program in Mpumalanga Province: Results from process evaluation. Journal of Primary Care and Community Health, 11(1), 1-13. doi:10.1177/2150132720975552
    » https://doi.org/10.1177/2150132720975552
  • Organização para a Cooperação e Desenvolvimento Econômico. (2016). Brasil no PISA 2015: Análises e reflexões sobre o desempenho dos estudantes brasileiros [Brazil in PISA 2015: Analyzes and reflections on Brazilian students’ performance]. São Paulo, SP: Fundação Santillana. Retrieved from https://download.inep.gov.br/acoes_internacionais/pisa/resultados/2015/pisa2015_completo_final_baixa.pdf
    » https://download.inep.gov.br/acoes_internacionais/pisa/resultados/2015/pisa2015_completo_final_baixa.pdf
  • Sanchez, Z. M., Valente, J. Y., Galvão, P. P., Gubert, F. A., Melo, M. H. S, Caetano, S. C., ... Cogo-Moreira, H. (2021). A cluster randomized controlled trial evaluating the effectiveness of the school-based drug prevention program #Tamojunto2.0. Addiction (Abingdon, England), 116(6), 1580-1592. doi:10.1111/add.15358
    » https://doi.org/10.1111/add.15358
  • Throuvala, M. A., Griffiths, M. D., Rennoldson, M., & Kuss, D. J.(2019). School-based prevention for adolescent internet addiction: Prevention is the key: A systematic literature review. Current Neuropharmacology, 17(6), 507-525.doi:10.2174/1570159X16666180813153806
    » https://doi.org/10.2174/1570159X16666180813153806
  • Support:

    This research received financial support from the Ministério da Saúde (TED 89/2014).

Edited by

Associate editor:

Eduardo Name Risk

Publication Dates

  • Publication in this collection
    29 July 2022
  • Date of issue
    2022

History

  • Received
    20 Dec 2021
  • Reviewed
    12 Apr 2022
  • Accepted
    02 May 2022
Universidade de São Paulo, Faculdade de Filosofia Ciências e Letras de Ribeirão Preto, Programa de Pós-Graduação em Psicologia Av.Bandeirantes 3900 - Monte Alegre, 14040-901 Ribeirão Preto - São Paulo - Brasil, Tel.: (55 16) 3315-3829 - Ribeirão Preto - SP - Brazil
E-mail: paideia@usp.br