Theory of change for implementing matrix support in mental health

Abstract

Objective

To describe the process of development of the theory of change for the implementation of Matrix Support in a medium-sized municipality in São Paulo and to reflect on the contributions of this approach to the planning and evaluation of this intervention, considering the potentialities and challenges from its execution remotely.

Methods

Qualitative study performed through participatory workshops and construction of narratives with workers and managers of the Psychosocial Care Network following the recommendations of a guide for the development of the theory of change in complex interventions.

Results

The implementation of Matrix Support configures one of the three distal results that culminate in the possibility of establishing the logic of shared care related to the activation of intersectoral network and the implementation of a Continuing Education Committee. External components that should be guaranteed by the organizational context for the effective achievement of some intermediate and long-term results were also recognized.

Conclusion

The development of the theory of change was effective in delimiting objectives and implementation components from the perspective of the actors involved, establishing indicators for its monitoring and evaluation based on context conditions, and enabling the direction of workers’ efforts towards its consolidation.

Implementation Science; Mental health; Evidence base practice

Resumo

Objetivo

Descrever o processo de elaboração da theory of change para implementação do Apoio Matricial em um município paulista de médio porte e refletir sobre contribuições dessa abordagem para o planejamento e avaliação dessa intervenção considerando as potencialidades e desafios a partir de sua execução no formato remoto.

Métodos

Estudo qualitativo realizado por meio de oficinas participativas e construção de narrativas com trabalhadores e gestores da Rede de Atenção Psicossocial seguindo as recomendações de um guia para o desenvolvimento de theory of change em intervenções complexas.

Resultados

A implementação do Apoio Matricial configura um dos três resultados distais que culminam na possibilidade de estabelecer a lógica do cuidado compartilhado relacionando-se à ativação da rede intersetorial e a implementação de um Comitê de Educação Permanente. Ainda foram reconhecidos componentes externos que deveriam ser garantidos pelo contexto organizacional para o efetivo alcance de alguns resultados intermediários e de longo prazo.

Conclusão

A elaboração da theory of change foi efetiva em delimitar objetivos e componentes da implementação a partir da perspectiva dos atores envolvidos, estabelecendo indicadores para seu monitoramento e avaliação a partir das condições do contexto e possibilitando o direcionamento de esforços dos trabalhadores para sua consolidação.

Ciência de implementação; Prática baseada em evidências; Saúde mental

Resumen

Objetivo

Describir el proceso de elaboración de la theory of change para implementar el apoyo matricial en un municipio paulista de medio porte y reflexionar sobre las contribuciones de este enfoque para la planificación y evaluación de esta intervención, considerando las posibilidades y desafíos a partir de su ejecución en formato remoto.

Métodos

Estudio cualitativo realizado por medio de talleres participativos y construcción de narrativas con trabajadores y gerentes de la Red de Atención Psicosocial, siguiendo las recomendaciones de una guía para la elaboración de la theory of change en intervenciones complejas.

Resultados

La implementación del apoyo matricial configura uno de los tres resultados distales que culminan en la posibilidad de establecer la lógica del cuidado compartido y se relaciona con la activación de la red intersectorial y la implementación de un Comité de Educación Permanente. Además, se reconocieron componentes externos que deberían estar garantizados por el contexto organizacional para llegar a algunos resultados intermedios y de largo plazo.

Conclusión

La elaboración de la theory of change fue efectiva para definir objetivos y componentes de la implementación a partir de la perspectiva de los actores involucrados, lo que permitió establecer indicadores para su monitoreo y evaluación a partir de las condiciones del contexto y orientar los esfuerzos de los trabajadores para su consolidación.

Ciencia de la implementación; Salud mental; Práctica basada en la evidencia

Introduction

Shared and collaborative care proposals similar to Matrix Support (MS) have been recognized as good practices in mental healthcare around the world. ( ¹ , ² ) A substantial framework of evidence for collaborative care has emerged since the 1990s. More than 90 randomized clinical trials and several studies with meta-analyses have demonstrated that the collaborative care model is more effective than the usual treatment for patients with depression, anxiety and other behavioral health conditions. ( ³ ) In parallel, in Brazil, the MS is characterized, among other dimensions, as a methodology that seeks to overcome the fragmentation of mental healthcare through the co-responsibility of the interdisciplinary team that comprises the network with the constant communication and joint deliberation. ( ⁴ )

After more than ten years of its incorporation through Support Centers for Family Health, many challenges continue to cross the effective implementation of MS arrangements in municipalities across the country. Among these challenges, the workers’ subjectivities, lack of clear outlines for the development of work processes such as case sharing, deficiencies in referral and counter-referral systems and low intersectoral coordination with little use of community resources are the main challenges cited in the literature. ( ⁴ ) Because it implies the need for reorganization at the various levels and services that make up the Psychosocial Care Network (Portuguese acronym: RAPS), the implementation of MS constitutes a complex health intervention, characterized by encompassing components capable of acting individually and/or jointly: resources, working conditions, organizational climate, workers and users. Coordinating and sustaining this mobilization is a challenge that extends from the design of the intervention to its evaluation phase. ( ⁵ - ⁷ )

As fundamental step for the success of these processes is understanding the relationship of the intervention with the various components of the system in order to produce its effect, a systemic approach that includes the perspective of the various actors involved in its execution is needed. ( ⁸ , ⁹ )

In this sense, the participatory development of the Theory of Change (ToC) in complex health interventions has been cited in the literature as an effective way to plan and evaluate these interventions. ( ¹⁰ - ¹² ) Its use for this purpose has already been documented in at least 49 interventions conducted in the field of Public Health, ( ¹³ ) an area in which its use could provide information about the implementation process, development of indicators, design of the evaluation process and data analysis.

This approach seeks to represent how, why, and to what extent the results can be expected as products of the intervention. It corresponds to a formal and explicit articulation of the assumptions supporting the logic and conception of an intervention. ( ¹⁰ - ¹² )

The purpose of this representation is to name the intermediate results and their coordination with causal chains. These, in turn, direct the relationship between intermediate results and show how they lead to distal results. Among intermediate results, the following stand out: interventions (strategies that aim to integrate the intervention into the system); theoretical justification (evidence that suggests a context-specific approach); external components (external conditions for achievement of the intermediate result); and indicators linked to the expected results (metrics that will allow identifying changes and their proportions). ( ⁵ ) The international literature points to several studies using the ToC in the area of mental health, including studies in countries of contexts similar to Brazil. ( ¹⁴ - ¹⁶ )

Given the above, the aim of the present study is to describe the ToC development process for the implementation of MS in a medium-sized municipality in São Paulo and to reflect on the contributions of this approach to the planning and evaluation of this intervention, considering the potentialities and challenges from of its execution remotely.

Methods

This is a descriptive study on a ToC development process composed of qualitative methodologies such as participatory workshops and construction of narratives ( ¹⁸ , ¹⁹ ) during the months of September and October 2020 with RAPS workers and managers in a medium-sized municipality in São Paulo. The municipality has an estimated population of 122,581 inhabitants and 19 Primary Health Care (PHC) services. In addition to hospital and emergency services, it has three specialized mental health services responsible for serving approximately 1,958 users. Regarding the demand for mental health in the PHC of the municipality, a descriptive study on the matrix research referring to the situational diagnosis of the mental health network was conducted previously, and a number of 3,022 users receiving mental health consultations in 13 Family Health Strategy (FHS) services was estimated. ( ¹⁷ )

Seeking to guide the development of the ToC from the perspective of actors involved in the intervention, we adopted the Research Management Committee (RMC) as a starting point for the selection of participants. ( ²⁰ ) This is a deliberative instance regarding the conduction of the implementation process. It includes participation of the following: seven researchers linked to an educational institution with the role of performing the synthesis and translation of evidence and supporting the implementation; five managers and local workers allocated in mental health services of the municipality; and two members of the municipal management directly involved in the coordination of mental health services, which, together, comprise the implementation of the MS system. In addition to members of the RMC, three workers from PHC services in the municipality were included in the preparation of the ToC, thus totaling 17 participants. The quality of engagement in a pilot training activity included as a component of the intervention was considered in the choice of participants. Note that all professionals who participated in this study were working directly in shared care actions, that is, in the MS of the municipality field of this study. The characterization of participants in chart 1 can help to better understand the sample.

Chart 1
Description of characteristics of study participants

Sex	Age	Training	Training time	Origin function/service	Time in service
F	57	Nursing	34 years	PHC worker	15 years
M	28	Medicine	2 years	PHC worker	2 years
F	39	Social Service	4 years	PHC worker	2 years
M	35	Nursing	5 years	Coordinator of Specialized Care/SMS	3 years
F	62	Nursing	39 years	Coordinator of Specialized Care/SMS	5 years
F	38	Occupational Therapy	16 years	Worker at the Psychosocial Care Center II	2 years
M	34	Psychology	11 years	Coordinator of the Psychosocial Care Center II	9 years
M	28	Psychology	3 years	Worker at the Psychosocial Care Center II	2 years
F	37	Psychology	12 years	Worker at the AD Psychosocial Care Center	2 years
F	30	Psychology	6 years	Coordinator of the AD Psychosocial Care Center	1 year
M	28	Nursing	7 years	Researcher at the Educational Institution	4 years
F	37	Occupational Therapy	17 years	Researcher at the Educational Institution	6 years
M	37	Psychology	14 years	Researcher at the Educational Institution	13 years
F	26	Psychology	3 years	Researcher at the Educational Institution	2 years
F	28	Occupational Therapy	6 years	Researcher at the Educational Institution	5 years
M	27	Psychology	4 years	Researcher at the Educational Institution	2 years
M	49	Psychology	24 years	Researcher at the Educational Institution	17 years

• SMS: Municipal Secretary of Health

Access to the perspective of actors involved in the intervention concomitantly with the development of the ToC took place through meetings. Initially, such meetings were not foreseen in a specific number and they occurred according to the general consensus of the group, totaling three meetings and one validation. During the four meetings, all participants mentioned in Table 1 had full and regular attendance.

In this process, the practical guide proposed by Silva et al. ( ⁵ ) “Using Theory of Change in the development, implementation and evaluation of complex health interventions: A practical guide” was followed. A familiar facilitator from the theoretical-practical point of view regarding the development of the ToC was appointed. The facilitator was a nurse researcher in public health and his role was to mediate the dialogue between the group using trigger questions with the group in order to raise aspects related to the intended change and the necessary elements for its achievement.

Given the health context imposed by the COVID-19 pandemic, the meetings took place through the Google-Meet ^© platform with a private link controlled by the workshop facilitator. Participants were encouraged to use audiovisual resources as the preferred means of contact, saving the use of chat for times of connection difficulties. The meetings lasted an average of two hours and the following script was used: (1) opening; (2) clarification on the dynamics of the dialogue; (3) establishing the setting; (4) dialogue; (5) summary of previous moments and (6) closure.

In order to coordinate the development of the ToC on a platform that all participants could visualize, a virtual whiteboard was used through the Miro ^© tool, which offers this feature free of charge, allowing collaborative editing, and the real-time preview through screen sharing. Two researchers were responsible for recording the ideas raised by the group with the use of virtual sticky notes in different colors, which is a first exercise in coding the units of record.

In the dialogue conducted in the form of brainstorming, the facilitator used some guiding questions defined in a script adapted from the proposal of the practical guide: 1) What is the impact or change that we want to achieve in the RAPS of ITATIBA?; 2) What are the intermediate outcomes (what needs to happen midway) to produce this impact or change?; 3) What interventions are needed for these things to happen? and 4) What resources are needed to deliver these interventions? Are these resources available?

Participants answered each question individually and were also encouraged to interact with each other in order to explore and clarify individual and shared perspectives. The purpose of the questions was to bring out perceptions about the following aspects: (1) the expected impact as a final outcome of the intervention; (2) the intermediate outcomes needed to make this happen; (3) interventions needed to ensure adoption, integration, and sustainability of the intervention; (4) theoretical and/or empirical justifications for performing the proposed actions; (5) the external components that affect the intervention and finally (6) the indicators linked to each of the expected outcomes.

The recording of workshops was performed by means of the construction of narratives. Two researchers were responsible for recognizing the core arguments present in the dialogues, thereby producing dense material faithful to the events of meetings, although not necessarily to their temporal sequence nor to the lexical forms of the group. ( ¹⁸ ) The narratives were constructed in a synchronous and shared way with use of the Google-Docs ^© tool, and validated with the group through reading, discussion and rectification of the material at the end of each meeting.

Figure 1 demonstrates how the produced narratives were used as analysis material for the systematization of the ToC design. In synchronous meetings, priority was given to surveying the content and perceptions about how the intervention should interact with the context to the detriment of clear definitions of nomenclatures or a well-designed graphic representation.

Figure 1
Summary overview of the ToC development process

The study was submitted to the Research Ethics Committee of the Faculdade de Ciências Médicas at UNICAMP and approved under opinion number 3.065.312 (Certificate of Ethical Appreciation Presentation: 00827918.8.0000.5404). All subjects agreed to participate by signing the Informed Consent form that ensures anonymity.

Results

The development of the ToC resulted in the clear definition that the implementation of the MS comprises one of the three distal results culminating in the possibility of implementing the logic of shared care in the RAPS of the municipality. In order to achieve this objective, the implementation of MS would also be related to activation of the intersectoral network and the implementation of a Continuing Education Committee. The achievement of distal results was conditioned to the establishment of seven intermediate results, five of which were direct results of implementation efforts and the other two produced collaterally as a result of its influence in the context in question. It is also noteworthy that four of the results raised, two intermediate and two distal, were scored as having a feedback relationship. The complete graphic representation of the ToC developed from this study is seen in figure 2 .

Figure 2
Complete graphic representation of the development of the ToC from the study

The creation of the RMC itself and the evaluation of the network functioning were identified as the initial resources of the implementation process; the latter was performed in the pre-implementation phase with a view to establishing a situational diagnosis of the RAPS. In addition to scoring theoretical justifications for these and other aspects ( Chart 2 ), participants identified a series of interventions (implementation strategies) aimed at enabling the achievement of intermediate and distal results.

Chart 2
Theoretical justifications adopted by the group during the development of the ToC

Highlighted element	Theoretical justification	Selected reference
Research Management Committee	Based on the need to establish a participatory process.	Furtado; Onocko-Campos ( 18 )
Implementation of a Continuing Education Committee	Based on the need to promote the sustainability of the implementation.	Treichel et al. ( 20 )
Evaluation of network functioning	Based on the need for a joint diagnosis and assessment of the implementation scenario.	Jamtvedt et al. ( 21 )
Aware and trained workers for shared work	Based on the need to disseminate knowledge about the proposed implementation.	Brasil ( 22 )
Activation of the intersectoral network	Based on the need for intersectoral action to achieve successful implementation.	Gonçalves et al. ( 23 )

The ToC development process also resulted in the agreement of eight indicators, two qualitative and six quantitative. From a qualitative point of view, the following were condensed into a single indicator: degree of acceptability, adoption, adequacy and feasibility, essential implementation outcomes for the achievement of the proposed interventions. The incorporation of the Continuing Education Committee to the organizational chart of the Municipal Health Secretariat was also understood as a qualitative indicator, since, according to participants’ understanding, this would constitute evidence of its institutionalization. From a quantitative point of view, the following were listed: indicators of the implementation process itself and its direct repercussions, indicators of the quality and completeness of care provided in the services, and indicators related to changes in the organization and flow of the network, understood as the main markers of effectiveness of the MS and shared care in the network. The interpretation, data source used and measurement method for each of the agreed indicators is shown in chart 3 .

Chart 3
Interpretation, data source used and measurement method of the agreed indicators through the development of the ToC

Indicator	Interpretation	Data source	Measurement method
Degree of acceptability, adoption, adequacy and feasibility of implementation.	It gives indications of readiness for change and greater predisposition to engage with the proposed interventions, increasing the chances of a successful implementation	Report from workers and managers.	In-depth interviews; Focus groups; Narratives.
Proportion of users who received minimal physical health care.	It gives evidence of the quality and comprehensiveness of the care offered in services, thereby seeking to reduce the disparities in health experienced by people with mental disorders.	Users’ medical records.	Number of users who had their anthropometric data, vital signs, and capillary blood glucose measured in the previous 3 months / by total active users with mental health complaints who consulted in the prior 3 months.
Proportion of workers and managers trained for networked work.	It gives indications of the scope of the training planned as an intervention and of the network’s appropriation of the knowledge necessary for the success of the implementation.	Minutes and records of the training process.	Number of trained workers and managers/ by total number of workers and managers registered in specialized and PHC services (except for administrative assistants).
Proportion of cases assisted in specialized services that are known by PHC.	It gives indications of the quality of the integration between the specialized network and the PHC, in addition to serving as a proxy for assessing the continuity of care between the different levels of care.	Users’ medical records.	Number of users of specialized services who have their mental health demand registered in their PHC medical records / by the number of active users of specialized services.
Proportion of users admitted to specialized services through spontaneous demand.	It gives indications about the capacity of absorption and adequate referral of the care network, reserving this type of access to cases of users in crisis.	Users’ medical records.	Number of users embraced by spontaneous demand in the previous year / by the total number of users embraced in the previous year.
Proportion of users admitted to specialized services through referral from PHC.	It gives indications about the capacity of identification and referral by PHC and of its constitution as a gateway to the RAPS.	Users’ medical records.	Number of embraced users from PHC referral in the previous year / by the total number of users embraced in the previous year.
Reorientation of referral flow between CAPS-AD and Therapeutic Communities.	It gives indications in relation to the role of RAPS in relation to the supplementary system.	Users’ medical records.	Number of embraced users after referral from Therapeutic Communities in the previous year / by the total number of embraced users in the previous year.
Incorporation of the Continuing Education Committee into the organizational chart of the Municipal department of Health.	It gives evidence of the institutionalization of this arrangement, thereby enabling the continuing training of professionals and the sustainability of the implementation.	Report from workers and managers.	In-depth interviews; Focus groups; Narratives.

Finally, in addition to the control of the implementation project, four external components that should be made available for the effective achievement of some intermediate and long-term results were recognized: acquisition of inputs and materials for the qualification of users’ clinical care; implementation of an electronic medical record system for qualification of the referral and counter-referral process; guarantee of physical structure and necessary resources for matrix meetings; and guarantee of physical and budgetary structure for the operation of the Continuing Education Committee originated from the qualification of workers and managers with the RMC.

Discussion

The development of the ToC is documented in several ways in the literature, with experiences based on literature reviews ( ²⁴ ) and evaluation of administrative records. ( ²⁵ ) However, as observed by Breuer et al., ( ¹³ ) we evaluated that conducting this process with the contribution of different interested parts through collaborative workshops made it possible to expand and deepen the participatory process. This enabled the reach of a realistic consensus among those involved about how and what results should be achieved from the implementation, as well as what resources will be needed and what restrictions are imposed on the process.

It is also noteworthy that working with stakeholders allowed for a shift in implementation planning and its evaluation from the theoretical field to the practical field, enabling an outline based on the perspective of those who experience the daily reality of services and have concrete conditions to act on them. ( ²⁶ ) However, the lack of participation of users and their families is signaled as a weakness of the process, especially when considering that a way to enhance the development of the ToC is to be as inclusive as possible, recognizing the importance of including a range of perspectives in the understanding of the theoretical foundations of the implemented intervention. ( ²⁷ )

In this sense, it must be considered that the participation of different actors in the development of the ToC is conditioned to the context, ( ²⁶ ) which in our experience was marked by the need for social distancing imposed by the Covid-19 pandemic. Therefore, we chose to conduct the workshops virtually, which on the one hand made it impossible to effectively include these stakeholders, but at the same time made it possible to continue the implementation even in such a challenging context.

Note that the virtual performance of activities demanded the search for previous experiences to identify tools and methodologies that could be used in order to keep the format of workshops as participatory as possible. Based on recent experiences reported in the literature, ( ²⁸ , ²⁹ ) we identified the use of synchronous communication platforms and virtual whiteboards as possible strategies.

While the use of synchronous communication platforms made it possible to meet participants, the use of the virtual whiteboard provided opportunities for a joint visualization of the graphic representation of the ToC, as well as its collective construction. Although the resource allowed participants’ involvement, aspects such as the varied quality of participants’ connection and the lack of security to use the platforms posed a challenge to perform this task. This demonstrates the importance of the method used with two researchers (as previously described), in which they were responsible for translating participants’ oral or written notes to the whiteboard, in addition to the use of narratives. ( ¹⁹ )

The choice to work with narratives was based on successful previous studies with their use in the conduction of participatory evaluation processes ( ²³ ) and agreement on indicators. ( ³⁰ ) In these studies, the use of narratives stood out for allowing the recording of aspects of the relationship between subjects before the events, experiences and meanings produced by the encounter, thereby producing a collective memory narrated in the first person plural. ( ¹⁹ , ³⁰ ) In our experience, the narratives were used by researchers as a basis to structure the graphical representation of the ToC between meetings, subsequently validating it with participants.

One of the most important results obtained from the development of the ToC in our study was the delimitation of indicators that would be used to evaluate the implementation process. Through this exercise, the indicators related to short, medium and long term results could be defined, thus favoring the design of an evaluation process guided by the real conditions of the context, and the integration of evaluations of implementation and effectiveness under the same theoretical structure. ( ³¹ - ³³ )

As in a previous study, ( ¹² ) participants were initially reluctant to define indicators for evaluating the implementation, especially quantitative indicators. In our assessment, this reality was associated with a fear that an eventual failure to reach these indicators would imply a negative evaluation of workers’ performance. This feeling was dissipated as the facilitator proposed linking the results and indicators to the external components that need to be worked on so that the goals could be achieved, thereby dissociating from the idea that the successful implementation depends solely on the performance of participants.

However, the presentation of this process is innovative insofar as it describes a valuable participatory methodology, exploring peculiarities of the work processes and the agreement on effectiveness indicators from the perspective of actors crossed by the practice of mental healthcare (in this case, managers and workers of the network). This supports the development and evaluation of interventions based on the real needs and factors of their respective contexts. In view of this, the present study contributes to filling the gap between the practice and the theory of evidence-based practices in mental healthcare, considered one of the great challenges of Implementation Research and the world. ( ³⁴ )

Conclusion

The study concludes that the objectives were achieved: on the description of the ToC for the implementation of the MS, the details and graphical representations are accurately described in the text, allowing transparency in the process and visualization of the applicability of the theory in an implementation study. Regarding the reflections on the contributions of this approach, the ToC development process was effective in delimiting the objectives and components of the implementation from different perspectives, as well as the relationship established between them and the results expected as the unfolding of each planned action. Likewise, it was a powerful process for establishing indicators for the monitoring and evaluation of the intervention, as, when these are agreed upon based on real conditions of the context, the actors will be able to direct efforts towards care actions that favor the success of the intervention. Therefore, the development of the ToC is recognized as a potent activity to guide the paths of this process and its evaluation.

Acknowledgements

To the Fundação de Amparo à Pesquisa do Estado de S. Paulo (FAPESP), the Universidade Estadual de Campinas (UNICAMP), the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CNPQ) and the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) for funding.

Referências

• ¹
  Bower P , Gilbody S , Richards D , Fletcher J , Sutton A . Collaborative care for depression in primary care . Making sense of a complex intervention: systematic review and meta-regression. Br J Psychiatry . 2006 ; 189 ( 6 ): 484 – 93 . Review.
• ²
  Smith SM , Cousins G , Clyne B , Allwright S , O’Dowd T . Shared care across the interface between primary and specialty care in management of long term conditions . Cochrane Database Syst Rev . 2017 ; 2 ( 2 ): CD004910 .
• ³
  Advanced Integrated Mental Health Solutions (AIMS) . Collaborative-care . Washington : University of Washington ; 2023 [ cited 2023 Mar 21 ]. Available from: https://aims.uw.edu/collaborative-care/evidence-base-cocm
  » https://aims.uw.edu/collaborative-care/evidence-base-cocm
• ⁴
  Treichel CA , Onocko-Campos RT , Campos GW . A consolidação e efetividade do apoio matricial em saúde mental no Brasil - gargalos e desafios . Interface Comun Saúde Educ . 2019 ; 23 : 1 - 18 .
• ⁵
  Silva MD , Lee L , Ryan G . Using theory of change in the development, implementation and evaluation of complex health interventions. A practical guide . Genebra : Mental Health Innovations Network ; 2015 [ cited 2021 Oct 4 ]. Available from: https://www.mhinnovation.net/sites/default/files/downloads/resource/MHIN%20ToC%20guidelines_May_2015.pdf
  » https://www.mhinnovation.net/sites/default/files/downloads/resource/MHIN%20ToC%20guidelines_May_2015.pdf
• ⁶
  Craig P , Petticrew M . Developing and evaluating complex interventions: reflections on the 2008 MRC guidance . Int J Nurs Stud . 2013 ; 50 ( 5 ): 585 – 7 .
• ⁷
  Craig P , Dieppe P , Macintyre S , Michie S , Nazareth I , Petticrew M ; Medical Research Council Guidance . Developing and evaluating complex interventions: the new Medical Research Council guidance . BMJ . 2008 ; 337 : a1655 .
• ⁸
  Leischow SJ , Best A , Trochim WM , Clark PI , Gallagher RS , Marcus SE , et al . Systems thinking to improve the public’s health . Am J Prev Med . 2008 ; 35 ( 2 Suppl ): S196 – 203 .
• ⁹
  De Silva MJ , Breuer E , Lee L , Asher L , Chowdhary N , Lund C , et al . Theory of Change: a theory-driven approach to enhance the Medical Research Council’s framework for complex interventions . Trials . 2014 ; 15 ( 1 ): 267 .
• ¹⁰
  Paina L , Wilkinson A , Tetui M , Ekirapa-Kiracho E , Barman D , Ahmed T , et al . Using Theories of Change to inform implementation of health systems research and innovation: experiences of Future Health Systems consortium partners in Bangladesh, India and Uganda . Health Res Policy Syst . 2017 ; 15 ( S2 Suppl 2 ): 109 .
• ¹¹
  Ghate D . Developing theories of change for social programmes: co-producing evidence-supported quality improvement . Palgrave Commun . 2018 ; 4 ( 1 ): 90 .
• ¹²
  Maini R , Mounier-Jack S , Borghi J . How to and how not to develop a theory of change to evaluate a complex intervention: reflections on an experience in the Democratic Republic of Congo . BMJ Glob Health . 2018 ; 3 ( 1 ): e000617 .
• ¹³
  Breuer E , Lee L , De Silva M , Lund C . Using theory of change to design and evaluate public health interventions: a systematic review . Implement Sci . 2016 ; 11 ( 1 ): 63 .
• ¹⁴
  Breuer E , De Silva MJ , Fekadu A , Luitel NP , Murhar V , Nakku J , et al . Using workshops to develop theories of change in five low and middle income countries: lessons from the programme for improving mental health care (PRIME) . Int J Ment Health Syst . 2014 ; 8 ( 1 ): 15 .
• ¹⁵
  Esponda GM , Ryan GK , Estrin GL , Usmani S , Lee L , Murphy J , et al . Lessons from a theory of change-driven evaluation of a global mental health funding portfolio . Int J Ment Health Syst . 2021 ; 15 ( 1 ): 18 .
• ¹⁶
  Breuer E , De Silva MJ , Shidaye R , Petersen I , Nakku J , Jordans MJ , et al . Planning and evaluating mental health services in low- and middle-income countries using theory of change . Br J Psychiatry . 2016 ; 208 ( Suppl 56 ): s55 – 62 .
• ¹⁷
  Treichel CA , Campos RT . Campos RT . Avaliação da atuação da rede comunitária de saúde mental em um município paulista de médio porte . Saúde Debate . 2022 ; 46 ( 132 ): 121 – 34 .
• ¹⁸
  Furtado JP , Campos RO . Participação, produção de conhecimento e pesquisa avaliativa: a participação de diferentes atores em um estudo de saúde mental . Cad Saude Publica . 2008 ; 24 ( 11 ): 2671 – 80 .
• ¹⁹
  Onocko Campos RT , Furtado JP . Narratives: use in qualitative health-related research . Rev Saude Publica . 2008 ; 42 ( 6 ): 1090 – 6 .
• ²⁰
  Treichel CA , Silva MC , Presotto RF , Onocko-Campos RT . Research Management Committee as strategic device for a mental health implementation research . Saúde Debate . 2019 ; 43 : 35 – 47 .
• ²¹
  Jamtvedt G , Flottorp S , Ivers N . Audit and feedback as a quality strategy . In: Busse R , Klazinga N , Panteli D , Quentin W , editors . Improving healthcare quality in Europe: characteristics, effectiveness and implementation of different strategies . Copenhagen, Denmark : European Observatory on Health Systems and Policies ; 2019 [ cited 2022 Oct 12 ]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK549284/
  » https://www.ncbi.nlm.nih.gov/books/NBK549284/
• ²²
  Brasil . Ministério da Saúde . Política Nacional de Educação Permanente em Saúde: o que se tem produzido para o seu fortalecimento? Brasília (DF) : Ministério da Saúde ; 2018 [ citado 2022 Out 12 ]. Disponível em: https://bvsms.saude.gov.br/bvs/publicacoes/politica_nacional_educacao_permanente_saude_fortalecimento.pdf
  » https://bvsms.saude.gov.br/bvs/publicacoes/politica_nacional_educacao_permanente_saude_fortalecimento.pdf
• ²³
  Brasil . Ministério da Saúde . Guia prático de matriciamento em saúde mental . Brasília (DF) : Ministério da Saúde ; 2011 [ citado 2022 Out 12 ]. Disponível em: https://bvsms.saude.gov.br/bvs/publicacoes/guia_pratico_matriciamento_saudemental.pdf
  » https://bvsms.saude.gov.br/bvs/publicacoes/guia_pratico_matriciamento_saudemental.pdf
• ²⁴
  Walker JS , Matarese M . Using a theory of change to drive human resource development for wraparound . J Child Fam Stud . 2011 ; 20 ( 6 ): 791 – 803 .
• ²⁵
  Schierhout G , Hains J , Si D , Kennedy C , Cox R , Kwedza R , et al . Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change . Implement Sci . 2013 ; 8 ( 1 ): 119 .
• ²⁶
  Furtado JP , Onocko-Campos RT , Moreira MI , Trapé TL . Desenvolvimento participativo de indicadores para avaliação de saúde mental . Cad Saude Publica . 2013 ; 29 ( 1 ): 102 – 10 .
• ²⁷
  Mukumbang FC , van Belle S , Marchal B , van Wyk B . Towards developing an initial programme theory: programme designers and managers assumptions on the antiretroviral treatment adherence club programme in primary health care facilities in the metropolitan area of western cape province, South Africa . PLoS One . 2016 ; 11 ( 8 ): e0161790 .
• ²⁸
  Corrêa LM , Lima RC . O transtorno bipolar na rede: a construção do diagnóstico em um grupo on-line . Physis (Rio J.) . 2018 ; 28 ( 4 ): e280406 .
• ²⁹
  Nobrega S , El Ghaziri M , Giacobbe L , Rice S , Punnett L , Edwards K . Feasibility of Virtual Focus Groups in Program Impact Evaluation . Int J Qual Methods . 2021 ; 20 : 10.1177/16094069211019896 .
  » https://doi.org/10.1177/16094069211019896
• ³⁰
  Bustamante V , Onocko-Campos R , Silva AA , Treichel CA . Assessment indicators of psychosocial children and teenager care centers: results of an intervention research . Interface Comun Saúde Educ . 2020 ; 24 : e190276 .
• ³¹
  Breuer E , De Silva MJ , Fekadu A , Luitel NP , Murhar V , Nakku J , et al . Using workshops to develop theories of change in five low and middle income countries: lessons from the programme for improving mental health care (PRIME) . Int J Ment Health Syst . 2014 ; 8 ( 1 ): 15 .
• ³²
  Weitzman BC , Mijanovich T , Silver D , Brecher C . Finding the impact in a messy intervention: using an integrated design to evaluate a comprehensive citywide health initiative . Am J Eval . 2009 ; 30 ( 4 ): 495 – 514 .
• ³³
  Clarke DJ , Godfrey M , Hawkins R , Sadler E , Harding G , Forster A , et al . Implementing a training intervention to support caregivers after stroke: a process evaluation examining the initiation and embedding of programme change . Implement Sci . 2013 ; 8 ( 1 ): 96 .
• ³⁴
  Proctor EK , Landsverk J , Aarons G , Chambers D , Glisson C , Mittman B . Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges . Adm Policy Ment Health . 2009 ; 36 ( 1 ): 24 – 34 .

Edited by

Associate Editor (Peer review process): Thiago da Silva Domingos (https://orcid.org/0000-0002-1421-7468) Escola Paulista de Enfermagem, Universidade Federal de São Paulo, São Paulo, SP, Brazil

Publication Dates

• Publication in this collection
  21 Aug 2023
• Date of issue
  2023

History

• Received
  21 Oct 2022
• Accepted
  16 May 2023