Evaluation matrix for health promotion programs in socially vulnerable territories

Abstract

Health promotion has a set of strategies for advancing health and reducing inequalities. However, evaluating the effectiveness of health promotion programs has been a challenge. This paper shows the development and application of the Evaluation Matrix, constructed with qualitative-quantitative and multidimensional indicators supported by public policies targeting socially vulnerable territories. This is a cross-sectional study with the implementation of a health promotion program in order to develop an Evaluation Matrix to be applied in two distinct socio-environmentally vulnerable areas. The Evaluation Matrix proved to be easily applicable and enabled the detection of strengths and weaknesses of health promotion programs applied in different territories. The participation of managers, teams, population and multiple sectors of society was decisive for the success of the program. Furthermore, community health workers stood out as essential stakeholders due to their linkages with the population. Contributions include a tool and methodology for evaluating health promotion programs to be applied in different territories and modified according to the territory.

Key words
Primary Health Care; Inequities; Territory

Resumo

A promoção da saúde possui um conjunto de estratégias para trazer saúde e reduzir desigualdades. No entanto, tem sido um desafio avaliar a efetividade de programas de promoção de saúde. Este artigo apresenta o desenvolvimento e a aplicação de Matriz de Avaliação, construída por meio de indicadores quali-quantitativos e multidimensionais, que estão apoiados nas políticas públicas voltadas para territórios com vulnerabilidade social. Estudo transversal, com realização de programa de promoção da saúde e a finalidade de desenvolver uma Matriz de Avaliação, com possibilidade de aplicação em duas áreas de distinta vulnerabilidade socioambiental. A Matriz de Avaliação demonstrou ser de fácil aplicação e permitiu detectar os pontos fortes e fracos dos programas de promoção da saúde aplicados em diferentes territórios. Verificou-se que a adesão dos gestores, das equipes, da população e dos diversos setores da sociedade foram decisivos para o sucesso do programa. Adicionalmente, os agentes de saúde se destacaram como atores essenciais devido ao seu vínculo com a população. As contribuições incluem instrumento e metodologia para avaliar os programas de promoção da saúde, que podem ser aplicados em realidades distintas e modificados de acordo com o território.

Palavras-chave
Atenção básica; Iniquidades; Território

Introduction

In 1946, Sigerist referred to the term “health promotion” in defining essential medical tasks such as health promotion, disease prevention, patient recovery and rehabilitation¹.

In the 1970s, with the revival of nineteenth-century social medical thinking, the term health promotion was once again used mainly in Canada and in Western European countries, from the discussion about effective high-level curative health care technology, increasing health medicalization and the need to reduce the cost of the current biomedical model².

In the 1980s, the International Conferences of Ottawa (1986), Adelaide (1988) and Sundsvall (1991) initiated the global health promotion movement and brought to the discussion the concepts of health, risk, social vulnerability, territory, intersectoriality, participation and surveillance, as well as the current conceptual and political bases for health promotion³.

In Brazil, this discussion was inspired by the progressive model, with a scientific stance and critical analysis of the relationship between health and society. Thus, there was an extensive production of papers aimed at understanding this relationship and the origin of the different epidemiological profiles found in our society, characterized by inequality. Therefore, health promotion began being perceived through its determinants and the understanding of the health-disease process was expanded³.

In the last two decades, some Latin American and Caribbean countries have implemented reforms in their health systems that have fostered inclusion, citizen participation and equitable access to health care⁴. Despite these initiatives, huge inequalities in the coverage of health interventions persist in most countries of the region.

In Brazil, the economic advances observed during the military regime have disproportionately benefited the privileged segments of society and the movements for democracy provided a broad discussion on the needs of the population, thus catalyzing the Health Reform⁵. The construction of the Unified Health System (SUS), based on the principles of universalization, equity and integrality was a commitment of the State to comply with its duty to provide health promotion policies. It is in this context that Primary Health Care of SUS, primarily through the Family Health Strategy (FHS), acquired the important role of responsible for providing humanized access, coordinated care, comprehensive services and equity in its actions⁶. Health promotion programs have been implemented by SUS with the aim of reducing inequalities in different territories⁷, but there is no evidence of an evaluation tool that originates in territorialized social realities^8,9 that considers the participation of societal sectors and stakeholders⁷.

The need to follow-up on the quality of health systems through monitoring and systematic evaluations has been emphasized by the World Health Organization (WHO)¹⁰, but they generally use disease outcomes such as mortality and morbidity. Previous studies have shown that health promotion has the potential to act effectively in the health-disease process in a way consistent with the reality of each territory^2,3,7; but evaluating health promotion programs is a challenge, since health promotion has a differentiated vision based on territoriality and is unrelated to the exclusively biomedical vision of disease. In collaboration with WHO, since 2002, the International Union for Health Promotion and Education (IUHPE) has started to support initiatives to assess and show the effectiveness of health promotion to health system managers¹⁰. In 2004, the Pan American Health Organization (PAHO) engaged in this debate, which emphasized the importance of establishing health promotion policies and/or programs built on participatory methodology and agreement of guiding values and principles for the evaluation of health promotion¹⁰.

At the national level, SUS’ performance is evaluated with PROADESS¹¹ and the quality of Primary Health Care services is evaluated with the PMAQ¹². However, SUS does not have a tool that evaluates its proposal for the qualification of health promotion actions.

Academia is also addressing this challenge. Restrepo¹³ considers that the evaluation of health promotion should be part of planning and developed through social participation and sustainable actions. Pedrosa⁷ stresses this characteristic of health promotion evaluation when affirming that it has to be participatory, where stakeholders negotiate, agree and decide collectively in order to achieve the desired changes. According to the author⁷, evaluation thought of in this form acts between the established and the transformative, and is transdisciplinary and multicultural, highlighting the need to be institutionalized in order to be effective, since issues, criteria and evaluation parameters are built from the articulation of the evaluation and its object.

These trends are also found in the quality assessment model, the “Systemic Evaluation Model” proposed by Donabedian^14,15, which is composed of the analysis of the realms: structure, process and results. For each realm several indicators are selected and used jointly with the intention of enabling stakeholders involved in producing health to appropriate the methods and tools both to make a diagnosis about the organization and operation of services and practices and to build intervention projects for the identified challenges.

The challenge of evaluating health promotion programs is even greater when considering the complex health-disease process in areas of socio-environmental vulnerability, where poverty-related diseases coexist with chronic non-communicable diseases, external causes of injury, inadequate sanitation conditions and difficult access to prophylactic measures, including treatment and educational measures^16-20. In this context, intestinal parasitic infections (IPI) perpetuate the disease-poverty-disease cycle¹⁸ by impairing cognitive function and school performance and, consequently, employability conditions^6,21. The impact of IPIs has been largely ignored in Brazil and in other developing countries, neglecting these diseases even more^18,22,23.

This study builds on the hypothesis that qualitative-quantitative and multidimensional health indicators contribute to the evaluation of the effectiveness of health promotion programs based on public policies, the promotion of perceived health from its determinants and the expanded conception of the health-disease process³.

Thus, this study aimed to develop an Evaluation Matrix based on the implementation of a health promotion program, with the theme of confronting intestinal parasites (IPs) in vulnerable territories, using the theoretical bases of health promotion, the quality assessment model^14,15, health planning^8,13 and evaluation⁷ for its construction.

Methodology

This cross-sectional qualitative-quantitative study was carried out through the implementation of a health promotion program in the period of 2013-2015, using participant observation, interviews and census in the 559 households enrolled in the Family Health Strategy (FHS) of the municipality of Laje do Muriaé, RJ, aiming at the construction of the Evaluation Matrix and subsequent testing of its applicability in territories with different realities. Program implementation counted on the partnership of the Municipal Health Secretariat (SMS) and adherence of FHS and Endemic teams of the Municipal Culture and Education Secretariats. Secondary school students from the municipality, members of Scientific Pre-Initiation Program for Young Science Talents, from the Foundation for Research Support of the State of Rio de Janeiro (FAPERJ) also participated in the initiative.

Study areas

The health promotion program was developed in the municipality of Laje do Muriaé, which is located in the northeast of Rio de Janeiro, Brazil (21°12’24”S, 42°7’57”O). It has the second highest Social Vulnerability Index (IVSop = 0.82, 1 being the most vulnerable)²⁴, the highest rate of declining population in the State (-0.53% per year)²⁵, the second lowest GDP of the State²⁵ and a large force enrolled in the Bolsa Família (Family Grant) Program (4.208 inhabitants, 60.0%)²⁵.

Laje Muriaé has a total area of 250 km² and a population of 7,487 inhabitants²⁵; 3,126 (42%) have some occupation, of which only 940 (30%) have a formal employment relationship. The municipality has poor sanitation conditions, piped sewage, but which is exhausted in natura in the River Muriaé that traverses all its urban area.

The Matrix was later applied to the Manguinhos Complex of Favelas located in the northern part of the city of Rio de Janeiro (22°52’47.04”S, 43°14’57.18”W), with approximately 40,000 people²⁶. The Complex is composed mostly of salaried, underemployed and/or unemployed workers, poor sanitation and water supply conditions and defective public services²⁷. Since Manguinhos is not a municipality and does not fit into district boundaries, the Manguinhos Complex of Favelas has limited official population data. However, it is a region with issues related to poverty and increasing violence²⁸. Communities also coexist with air, water and soil contamination resulting from polluting industries, namely, the Manguinhos refinery.

Both locations have determinant environmental conditions in the development of health-disease processes.

Stages of the Health Promotion Program

The program carried out in Laje do Muriaé was structured in three stages (Figure 1):

1. Situational diagnosis (pre-test), aiming to identify the frequency and profile of intestinal parasitoses; knowledge, attitudes and practices about IPs and socio-environmental conditions²⁹.

2. Educational intervention: health education actions with the population based on the identified problem situation, using integrative practices that consider local culture and knowledge and continuing education actions.

3. Epidemiological, socio-environmental and educational re-evaluation (post-test)²⁹ and the evaluation of the quality of drinking water and peridomiciliar soil³⁰; with distribution of educational material on the proper care of drinking water, water reservoirs and filters.

Figure 1
Flowchart Model of the Health Promotion Program developed in the municipality of Laje do Muriaé, Rio de Janeiro, 2013-2015.

At all stages, observations made with the population and the FHS team were recorded²⁸, aiming to identify qualitative-quantitative indicators in a multidimensional participatory perspective for the construction of an Evaluation Matrix.

All stages were also carried out in partnership with community health workers. Participants were enrolled through Informed Consent Form (ICF) signed by the legal representative of the family. The partnership with the SMS started after program objectives were agreed, when the Deed of Undertaking and Terms of Agreement were signed with the managers and the FHS team, respectively.

Building the Evaluation Matrix

Monitoring of the health promotion program adapted from Moraes Neto et al.²⁹ allowed for the categorization of the program's stages in the following realms: structure, processes and results^14,15, and the identification of the need to map the work processes of each realm^14,15. The analysis of this mapping allowed us to outline the flowchart of the stages of the aforementioned program³¹ (Figure 1) and to identify the efficiency and effectiveness³² of each realm (strengths and weaknesses)^31,32, thus making the necessary adjustments. These results allowed to evaluate the articulation between structure, processes and results³² and to identify the two additional realms not proposed by the systemic model of Donabedian^14,15:

1. Context, since the program started with negotiation and agreement with the FHS team and stakeholders;

2. Continuity, due to the discontinuity of the program and its non-inclusion in the Municipal Health Plan of Laje do Muriaé.

It also allowed for the realization that the program considered nine principles of the PNPS⁶ and, therefore, that the construction of research descriptors should be bound to these principles.

Evaluation tool (questionnaire)

The evaluation tool (questionnaire) was prepared through the following:

1. Identification of 23 evaluation descriptors linked to the principles of the policy (Chart 1);

2. Allocation of questions for each descriptor;

3. Establishing the typology of these questions:

   • –

     Closed with scale: Yes, No, I don't know;

   • –

     Semi-open that allow for the addition of comments and;

   • –

     Developed from some important issues in order to obtain better information quality³².

Chart 1
Principles of the National Health Promotion Policy (2015) and descriptors that meet these principles, 2013-2015.

PNPS principles	Descriptors
Sustainability	D.1. Requires permanent and continuous actions and interventions, taking into account the political, economic, social, cultural and environmental realms.
Intrasectoriality	D.2. Adherence of local health managers in the planning, development, control and evaluation process.
Intrasectoriality	D.3. Participation of other sectors linked to health promotion, aiming at the construction of cooperative and resolutive networks.
Intersectoriality and social participation	D.4. Participation of different stakeholders in the planning, development, control and evaluation process.
Intersectoriality	D.5. Partnerships with other municipal, state and federal institutions.
Sustainability	D.6. Adequate space to carry out the different actions of the Health Promotion Program.
	D.7. Allocation of specific financial resources to carry out the PPS.
	D.8. Training and adherence of the team to carry out the Health Promotion Program.
Territoriality	D.9. Recognizing the knowledge, attitudes and practices of the population.
Territoriality	D.10. Recognizing socio-environmental determinants.
Territoriality	D.11. Recognizing the epidemiological profile of the population.
Equity	D.12. Recognition of vulnerable situations (complexity and uniqueness) of individuals and groups.
Empowerment	D.13. Intervention encourages subjects to seize control of decision-making and lifestyle choices (self-care).
Territoriality	D.14. Valuing popular and traditional knowledge and integrative and complementary practices.
Integrality	D.15. Considering the complexities, singularities and specificities of individuals and groups in the territory.
Social participation	D.16. Equitable participation of the population.
Intersectoriality	D.17. Participation of different professionals.
Territoriality	D.18. Monitoring and comparison of socio-environmental determinants of health.
Territoriality	D.19. Monitoring and comparison of knowledge, attitudes and practices of the population.
Territoriality	D.20. Monitoring and comparison of the epidemiological profile of the population.
Autonomy	D.21. Individuals may choose priorities consciously, considering the group.
Sustainability	D.22. Recognition of results by local team, managers and population.
Sustainability	D.23. Requires permanent and continuous actions and interventions, taking into account the political, economic, social, cultural and environmental realms.

Validation of applicability

The validation process of the evaluation tool (questionnaire) used micro and macro analyses based on its applicability. The tool was evaluated through the adapted Delphi Technique, with the collaboration of specialists in education, health promotion and health services using face-to-face meetings³³ (Figure 2).

Figure 2
Process of construction and validation of health promotion programs evaluation tools.

Submission to the Delphi Technique generated reformulations of the evaluation questionnaire³³, with each specialist contributing to improve the tool's applicability:

1. The Education specialist carried out an indepth analysis of the tool by broadening its qualitative character;

2. The health promotion specialist shortened the questionnaire, implementing improvements in the statement and reducing the application time to approximately 15 minutes;

3. The health service specialist stimulated the scope of the issues in order to adapt the instrument according to the needs of different territories with varying epidemiological profiles.

The differences in focus of analyses were not stimulated and emerged spontaneously from the evaluators. After obtaining the consensus of experts, the questionnaire that composed the Evaluation Matrix was established.

Then, questions received scores:

1. Closed questions with scale: yes = 2, no = 1, I don't know = 0;

2. Developing questions, one (1) point for each selected option;

3. Semi-open questions: one (1) point for each element added;

4. Time of application of questionnaires following the educational action:

Score Time 1 point Less than 6 months 2 points 6 months 3 points 6 months to 1 year 4 points 1 to 2 years 5 points Over 2 years16

Score	Time
1 point	Less than 6 months
2 points	6 months
3 points	6 months to 1 year
4 points	1 to 2 years
5 points	Over 2 years16

Intervals and the classification of scores obtained with the application of the Evaluation Matrix (Chart 2) were established following the assignment of scores.

Chart 2
Health promotion program evaluation matrix.

HEALTH PROMOTION PROGRAM EVALUATION MATRIX
Realms	Questions	Qualitative Indicator	Quantitative Indicator	Pontos
Context	1. Is the Health Promotion Program included in the report of the Municipal Health Conference and the Municipal Health Plan?	( ) Yes ( ) No ( ) Don't know	-
	2. Did local managers participate in the Health Promotion Program?	( ) Yes ( ) No ( ) Don't know	-
	2.1. When did this participation occur?	-	Select all that apply.
			( ) Planning
			( ) Execution
			( ) Monitoring
			( ) Assessment
	3. Has the Health Promotion Program counted on the participation of other sectors linked to health promotion (education, social welfare, environment, others)?	( ) Yes ( ) No ( ) Don't know	-
	3.1. Which were these sectors?	-	Select all that apply.
			( ) Education
			( ) Social welfare
			( ) Environment
			( ) Other:
	3.2. When did these sectors participate?		Select all that apply.
			( ) Planning
			( ) Execution
			( ) Monitoring
			( ) Assessment
	4. Has the Health Promotion Program counted on the participation of different stakeholders?	( ) Yes ( ) No ( ) Don't know	-
	4.1. Which were these stakeholders?	-	Select all that apply.
			( ) Individuals
			( ) Press
			( ) Institutions
			( ) Organized groups
			( ) Councils
			( ) Entities
			( ) Trade Unions
			( ) Churches
			( ) Political parties
			( ) Other:
	4.2. When did they participate?	-	Select all that apply.
			( ) Planning
			( ) Execution
			( ) Monitoring
			( ) Assessment
	5. Has the Health Promotion Program counted on the partnerships of municipal, state and federal institutions?	( ) Yes ( ) No ( ) Don't know	-
	5.1. Which were these partnerships?	-	Select all that apply.
			( ) Municipal ( ) Federal ( ) State ( ) Other:
	5.2. When did they participate?		Select all that apply.
			( ) Planning
			( ) Execution
			( ) Monitoring
			( ) Assessment
Structure	6. Was the space available to carry out the Health Promotion Program adequate?	( ) Yes ( ) No ( ) Don't know	-
	7. Has the Health Promotion Program obtained the necessary financial resources for its own implementation?	( ) Yes ( ) No ( ) Don't know	-
	8. Has the team adhered to perform the Health Promotion Program?	( ) Yes ( ) No ( ) Don't know	-
	8.1. Was the team trained to carry out the Health Promotion Program?	( ) Yes ( ) No ( ) Don't know	-
Process	9. Was a survey conducted on the knowledge, attitudes and practices of the population?	( ) Yes ( ) No ( ) Don't know	-
Result	9.1. Did the ESF team identify the knowledge, attitudes and practices of the population?	( ) Yes ( ) No ( ) Don't know	-
Process	10. Was a socio-environmental survey conducted on the population?	( ) Yes ( ) No ( ) Don't know	-
Result	10.1 Did the ESF team identify the socioenvironmental profile of the population?	( ) Yes ( ) No ( ) Don't know	-
Process	11. Was an epidemiological survey conducted on the population?	( ) Yes ( ) No ( ) Don't know	-
Result	11.1. Did the ESF team identify the epidemiological profile of the population?	( ) Yes ( ) No ( ) Don't know	-
Process	12. When identifying situations of vulnerability (complex and/or unique cases) of individuals or groups, was the team capable of producing responses?	( ) Yes ( ) No ( ) Don't know	-
	13. Was the “Educational Action” performed?	( ) Yes ( ) No ( ) Don't know	-
	13.1. Has the methodology used led to the questioning of the topics covered?	( ) Yes ( ) No ( ) Don't know	-
	13.2. Did it favor the collective construction of new strategies?	(( ) Yes ( ) No ( ) Don't know	-
	13.3. Did it change the lifestyles / habits (self-care)?	( ) Yes ( ) No ( ) Don't know	-
Process	14. What were the activities performed in the “Educational Action”?	-	Select all that apply.
			( ) Workshops
			( ) Creation of sentences (slogan)
			( ) Educational games
			( ) Conversation rounds
			( ) Guides or manuals
			( ) Role play
			( ) Posters
			( ) Lectures
			( ) Other:
	15. Was “Education Action” planned from the identified problem situation?	( ) Yes ( ) No ( ) Don't know	-
	15.1. Was educational material distributed?	( ) Yes ( ) No ( ) Don't know	-
	15.2. Was the language used in the educational material adequate to the socio-cultural reality and knowledge of the population?	(( ) Yes ( ) No ( ) Don't know	-
	16. Has the population adhered to the “Educational Action” performed?	( ) Yes ( ) No ( ) Don't know	-
	16.1. What means of dissemination were used to invite the population to the “Educational Action”?	-	Select all that apply.
			( ) Home visit
			( ) Car audio messaging
			( ) Flyers
			( ) Social networks
			( ) Electronic mail
			( ) Other:
	17. Have professionals from other areas participated in the initiative?	( ) Yes ( ) No ( ) Don't know	-
Process	18. Was a socio-environmental survey conducted after the “Educational Action”?	( ) Yes ( ) No ( ) Don't know	-
	18.1. How long after the “Educational Action”?	-	( ) Below 6 months
			( ) 6 months
			( ) Between 6 months and 1 year
			( ) Between 1 and 2 years
			( ) Over 2 years.
Result	18.2. Has the ESF team identified improvements in the socio-environmental conditions?	( ) Yes ( ) No ( ) Don't know	-
Process	19. Was a survey on the knowledge, attitudes and practices of the population conducted after the “Educational Action”?	( ) Yes ( ) No ( ) Don't know	-
	19.1. How long after the “Educational Action”?	-	( ) Below 6 months
			( ) 6 months
			( ) Between 6 months and 1 year
			( ) Between 1 and 2 years
			( ) Over 2 years.
Result	19.2. Has the ESF team identified the appropriation of knowledge, attitudes and practices by the population?	( ) Yes ( ) No ( ) Don't know	-
Process	20. Was a survey on the epidemiological profile of the population conducted after the “Educational Action”?	( ) Yes ( ) No ( ) Don't know	-
	20.1. How long after the “Educational Action”?	-	( ) Below 6 months
			( ) 6 months
			( ) Between 6 months and 1 year
			( ) Between 1 and 2 years
			( ) Over 2 years.
Result	20.2. Has the ESF team identified improvements in epidemiological profile?	( ) Yes ( ) No ( ) Don't know	-
Process	21. Did the population and the stakeholders participate in defining the priority themes of “Educational Action”?	( ) Yes ( ) No ( ) Don't know	-
	22. Were the results discussed by local staff and managers?	( ) Yes ( ) No ( ) Don't know	-
	22.1. Were the results disseminated to the population?	( ) Yes ( ) No ( ) Don't know	-
	22.2. What were the dissemination strategies?		Select all that apply.
			( ) Periodic meeting of the ESF team
			( ) Meeting of the Municipal Health Council
			( ) Web page
			( ) Social network
			( ) Form for suggestions, criticisms and comments
			( ) Other:
Continuity	23. Was there an indication of development and / or further analysis of the themes based on the evaluation of the results of the Health Promotion Program?	( ) Yes ( ) No ( ) Don't know	-
	23.1. Was there an indication of permanence, aiming at the institutionalization of the Health Promotion Program?	( ) Yes ( ) No ( ) Don't know	-
	23.2. Has the Health Promotion Program been maintained in the report of the Municipal Health Conference and the Municipal Health Plan?	( ) Yes ( ) No ( ) Don't know
TOTAL
Matrix Score
Intervals 104 to 130 77 to 103 50 to 76 23 to 49 0 to 22			Classification Excelent Very good Good Bad Very bad

Theoretical and potential alignment of the Evaluation Matrix Intervention

The Evaluation Matrix's potential for intervention and alignment with the theoretical foundations was discussed and evaluated in three workshops, which included the participation of public health researchers, FHS health professionals and municipal health managers.

Workshop 1: Theoretical alignment

* The Relevance of Evaluation and Monitoring of Health Promotion Programs, within the framework of the IOC/FIOCRUZ Study Center: with the presentation of three lectures and discussion by the plenary.

• –

  Health promotion and creativity.

• –

  The evaluation and monitoring of health program and the management of cities.

• –

  National Program for Improving Access and Quality of Primary Health Care (PMAQ).

Workshops 2 and 3: Potential for intervention

* Health promotion programs: relevance of evaluation and monitoring for public management.

* Health promotion programs: relevance of evaluation and monitoring for the population.

Workshops 2 and 3 were recorded and suggestions were registered by two rapporteurs and read at the end of the plenary. They were then incorporated into the Evaluation Matrix (Chart 2) following the participants’ approval.

The Evaluation Matrix was then applied to the coordinator and to the health workers who participated in the programs carried out in Laje do Muriaé and the Manguinhos Complex of Favelas, the latter by free choice of the FHS team of the Victor Valla Family Clinic, with authorization from the Education and Research Coordination of the Germano Sinval Faria School Health Center, National School of Public Health, Fiocruz, RJ. The results of these applications were submitted to a descriptive analysis (distribution of variables’ frequencies).

Ethical Considerations

The Fiocruz Ethics Committee on Human Research: CEP/IOC/Fiocruz 2013 approved this study in the Plataforma Brasil Platform.

Results

Observations recorded during the implementation of the health promotion program in Laje do Muriaé identified community health workers as the FHS health professionals involved in all of the stages of the program and with the greatest proximity to the population. The other members of the health team adhered to the program; however, they remained within the health facility, providing support for health promotion actions directed by community health workers. In 2015, endemic disease agents joined the FHS³⁴ team and, therefore, joined the program in stage two, participating in health education and continuing education actions, and stage three (in the posttest), in the collection of drinking water and peridomicile soil for physical-chemical and cholimetric analyses.

Coordinators and managers of the municipality of Laje do Muriaé supported the actions by increasing the governability of health workers to implement the program and solve problem situations, such as positive results for Taenia sp, which required specific actions by the FHS teams. The population was actively involved throughout the program, participating in the planned activities and seeking continuity with the health workers. The inclusion of secondary school students of the Pre-Scientific Initiation Program favored the population's adherence, since they were FHS users.

The fact that the educational activities depended on the participation of the population, managers and stakeholders and reflected the specific characteristics of the study area, facilitated the construction of knowledge by all stakeholders and favored the joint identification of mechanisms for the promotion, prevention and control of the positive association between A. lumbricoides and Taenia sp (2/11 (18.2%) vs. 5/764 (0.7%), p = 0.003 (Fisher's exact test).

The analysis of residents’ knowledge, attitudes and practices (KAP) in the post-test, when compared to pre-test showed that 88.3% considered IPIs as “disease / contamination caused by worms”, and no more: “It's natural, everyone has it, we're already born with it”. The result of the post-test epidemiological profile was better than the pre-test, since no association between parasites (A. lumbricoides and Taenia sp) was observed.

However, although 99.3% of the households received treated water, participant observation revealed that the population had a habit of drinking water from a well or a mine, and therefore, the post-test drinking water and peridomicile soil sample analysis was performed, showing that 84.2% of the water collected in households, water mines and wells were unfit for human consumption due to evidence of fecal coliforms E. coli and/or Salmonella sp. The prevalence of IPs in the soil was 82.7%.

The Evaluation Matrix proved to be easy to apply and facilitated the detection of strengths and weaknesses of health promotion programs carried out in territories with different characteristics: Laje do Muriaé and the Manguinhos Complex of Favelas.

The main strengths and weaknesses identified in the evaluation carried out by the Laje do Muriaé team were:

The Evaluation Matrix's application time with the Laje do Muriaé team averaged 15 minutes. The mean score was 102 (78.5%), corresponding to the classification “Very good”.

The main strengths and weaknesses identified in the evaluation carried out by the Manguinhos Complex of Favelas team were:

The Evaluation Matrix's application time with the Manguinhos Complex of Favelas team averaged 13 minutes. The mean score was 75 (57.7%), corresponding to the classification “Good”.

Discussion

The elaboration of the Evaluation Matrix built on the hypotheses that qualitative-quantitative and multidimensional health indicators contribute to the evaluation of the effectiveness of health promotion programs and that these programs are effective in the promotion, prevention and control of poverty-related diseases. It also considered that the construction of health indicators should be based on social reality⁸, participation and negotiation among stakeholders as suggested by Pedrosa⁷ and Restrepo¹³.

Pinheiro & Silva-Junior³³ affirmed the need for a collaborative relationship between managers and evaluators for the institutionalization of health promotion programs. The results of the application of the Evaluation Matrix in the two locations also showed the importance of the adherence of health managers and stakeholders to the negotiations, but also the need for their inclusion in the Municipal Health Plan in order to ensure their sustainability and consequent institutionalization.

Pedrosa⁷ argues that the lack of institutionalization occurs because municipal managers fail to consider health promotion as a public policy, since they believe that treatment is what matters, understanding health promotion as a chronologically anticipated prevention³⁵. Felisberto³⁶ says that institutionalizing evaluations should be understood as contributing to the quality of Primary Health Care. It helps to trigger a process for building knowledge and better practices, promoting the construction of structured and systematic processes consistent with the principles of SUS, in its various realms of management, care and intervention in the territory.

The proposal of a Health Promotion Evaluation Matrix has been discussed by WHO and authors such as Pedrosa⁷, who, in his proposed health promotion evaluation, identified the construction of three major matrices: health promotion linked to the conception of epidemiological and social risk, health promotion articulated to broad proposals for sustainable development, and a third, intermediate matrix focused on actions of intersectoriality, active participation of the population with a priority for local development⁷. The Evaluation Matrix shown in this paper synthesizes the three strands of the proposed Matrices.

It also is in line with the discussion established by Carvalho et al.³⁵ based on the challenges associated with the conceptual field of Health Promotion and the requirements of evidence of effectiveness and efficiency faced by managers, evaluators and agents in the development of intersectoral health actions. These authors understand that the need for evidence in public policies can lead to an increased gap between complex health promotion interventions and their conceptions, and that the determination of changes in this area requires collaborative processes that consider the stakeholders involved in the implementation and evaluation of interventions.

Thus, the evaluation cannot be a purely technical procedure, it must have a methodological design which aims for social stakeholders to participate with decision-making power and for the evaluation results to be incorporated into management^37,38, allowing managers to get acquainted with different problem situations and contribute to the well-being and improvement of the quality of life of the population. The second stage of the program (educational intervention) held in Laje do Muriaé, with workshops and round tables with the population, stakeholders, health managers and the FHS team brought to the forefront the need to outline the methodological design of the program in order to consolidate the results and favor the identification of qualitative-quantitative and multidimensional indicators for the Evaluation Matrix^37,39.

Chart 3
The main strengths and weaknesses identified in the Laje do Muriaé team.

Realm	Strengths
Context	Adherence of managers, staff, population, the various sectors involved in health promotion and stakeholders.
	Partnership with FIOCRUZ.
Structure	Physical space and adequate financial resources, training of health and endemic workers.
Processes	Performing pre-test, educational and post-test actions with participation of the population, stakeholders and other sectors linked to health promotion.
	Dissemination of results.
Results	Treatment and follow-up of the parasitized individuals.
Realm	Weaknesses
Continuity	Lack of sustainability.

Chart 4
The main strengths and weaknesses identified in Manguinhos Complex of Favelas.

Realms	Strengths
Structure	Team adherence.
Continuity	Signs of in-depth analysis of themes development.
	Program remained in the report of the Municipal Health Conference and the Municipal Health Plan.
Realms	Weaknesses
Context	Lack of engagement of stakeholders and other sectors linked to health promotion.
Processes	“Not knowing” whether a survey on KAP; socio-environmental conditions and epidemiological profile after the educational actions was performed.
	“Not knowing” whether results were disseminated to the population.

We could also observe that the three stages of the program carried out in Laje do Muriaé, in addition to contributing to the epidemiological diagnosis in the six micro areas covered by the local Primary Health Care services, the educational intervention, treatment and follow-up, further strengthened interactions between the population and the FHS team, as well as those with the local managers. In considering the results of the socio-environmental, educational and epidemiological surveys in the shared construction of educational actions, the program linked the specific characteristics of the territory to health education and to the preparation of the teams to engage in health education activities during home visits and throughout the entire process of care²³.

Besen et al.⁴⁰ say that the FHS does not yet include health education as a major focus, since managers are unaware of the health promotion rationale and most have vertical educational practices, in a short-sighted and curativist relationship with the population, still inserted in the biomedical model.

Rootman et al.⁴¹ stressed that there is a need to use multiple strategies to promote health and it would be necessary to be supported by principles of empowerment, integrality, participation, intersectoriality, equity and sustainability, principles found in the 2006 National Health Promotion Policy, which were extended with the introduction of intrasectoriality, territoriality and autonomy principles when reformulated in 2015⁶. This need to have the policy's principles as a strategic basis in the determination of evaluation indicators was identified at the end of the implementation of the program carried out in a participatory manner with all the segments involved in local Primary Health Care and facilitated the establishment of the Evaluation Matrix based on 23 descriptors elaborated from the nine principles of said Policy⁶ and linked to the realms of the expanded systemic model of Donabedian^14,15, aiming to enable the use of multiple strategies required to meet the characteristics of each territory, integrated by different vulnerable groups.

Thus, the Evaluation Matrix was established, corroborating the hypothesis that qualitative-quantitative and multidimensional health indicators contribute to the evaluation of the effectiveness of health promotion programs and that the participatory and integrated evaluation process has the potential to boost the management of quality and to promote the internalization of good public management practices and the continuous improvement of the teams’ work processes⁸, aiming at confronting situations of exclusion and inequities, based on the establishment of the determinants of the disease-health process.

Conclusion

The rapid urbanization of cities evidences important situations of exclusion and great inequities. The development of health promotion programs and evaluation of its effectiveness are essential strategies for the promotion of equity and the right to the city.

Thus, contributions of this work include a health promotion program model and an evaluation methodology, namely, the Evaluation Matrix, built from a case study in a municipality with high socio-environmental vulnerability and which has been shown to be applicable in both territories with different characteristics by favoring:

• –

  Self-assessment by teams and identification of weaknesses in their work processes;

• –

  The identification of individual and collective health problems and the consequent need to act with integrity and equity;

The Evaluation Matrix mainly showed the potential to boost quality management and favor the internalization of good public management practices and the continuous improvement of work processes, aiming to cope with situations of exclusion and inequities from the identification of the determinants of the health-disease process.

Acknowledgments

This study was funded by the Foundation for Research Support of the State of Rio de Janeiro (FAPERJ), Coordination for the Improvement of Higher Education Personnel (CAPES), and by the Laboratory of Innovations in Therapies, Education and Bioproducts - LITEB/IOC/Fiocruz. We would like to thank Anna Cristina Calçada Carvalho, Diana Pinheiro Marinho, Elisete Casotti, Felipe de Carvalho Vommaro Marincola, Júlio Cesar Pegado Bordignon, Lívia Marinho da Silva, Maria de Fátima Leal Alencar, Patrícia Ocampo, Sandra Escovedo Selles and Tania Cremonini de Araújo Jorge.

Referências

• ¹
  Sigerist HE. The University at the Crossroads. Addresses and Essays. Nova Iorque: Henry Schuman;1946.
• ²
  Buss PM, Carvalho AI. Desenvolvimento da Promoção da Saúde no Brasil nos últimos vinte anos (1988-2008). Cien Saude Colet 2009; 14(6):2305-2316.
• ³
  Czeresnia D, Freitas, CM, organizadores. Promoção da Saúde: conceitos, reflexões, tendências. 2ª ed. rev. e amp. Rio de Janeiro: Fiocruz; 2009.
• ⁴
  United Nations Children's Fund Health (Unicef). Equity Report: Analysis of reproductive, maternal, newborn, child and adolescent health inequities in Latin America and the Caribbean to inform policymaking Panama City: Unicef; 2016.
• ⁵
  Paim J, Travassos C, Almeida C, Bahia L, Macinko J. O Sistema de Saúde Brasileiro: histórias, avanços e desafios. Lancet 2011; 377 (9779):1778-1797.
• ⁶
  Brasil. Ministério da Saúde (MS). Secretaria de Vigilância em Saúde. Secretaria de Atenção à Saúde. Política Nacional de Promoção da Saúde (PNPS):revisão da Portaria MS/GM n° 687, de 30 de março de 2006. Brasília: MS; 2015.
• ⁷
  Pedrosa PIS. Perspectivas na avaliação em Promoção da Saúde: uma abordagem institucional. Cien Saude Colet 2004; 9(3):617-626.
• ⁸
  Minayo MCS. Saúde e Ambiente Sustentável: estreitando nós. Rio de Janeiro: Fiocruz; 2011.
• ⁹
  Kusma SZ, Moysés ST, Moysés SJ. Promoção da saúde: perspectivas avaliativas para a saúde bucal na atenção primária em saúde. Cad Saude Publica 2012; 28(Supl.):S9-S19.
• ¹⁰
  World Health Organ (WHO). The World Health Report 2008: Primary Health Care. Geneva: WHO. [acessado 2015 Maio 8]. Disponível em: http://www.who.int/whr/2008/en/
  » http://www.who.int/whr/2008/en/
• ¹¹
  Brasil. Ministério da Saúde (MS). Departamento de Monitoramento e Avaliação do SUS. O Programa de Avaliação para a Qualificação do SUS (PROADESS). Proposta de Avaliação de Desempenho do Sistema de Saúde: indicadores para monitoramento. [relatório final]. Rio de Janeiro: LISIC/ICT/Fiocruz; 2011.
• ¹²
  Brasil. Ministério da Saúde (MS). Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ) Brasília: MS; 2011.
• ¹³
  Restrepo H. Generalidades sobre evaluación de experiências y proyectos de promoción de la salud. In: Restrepo H, Málaga H, organizadores. Promoción de la salud: como construir vida saludable Bogotá: Editorial Médica Panamericana; 2001. p. 212-217.
• ¹⁴
  Donabedian A. The quality of care: How can it be assessed? Arch Pathol Lab Med 1997; 121(11): 1145-1150.
• ¹⁵
  Donabedian A. La calidad de la atención médica: definición y métodos de evaluación. México: La Prensa Mexicana; 1984.
• ¹⁶
  Prata PR. The Epidemiologic Transition in Brazil. Cad Saude Publica 1992; 8(2):168-175.
• ¹⁷
  Omram AR. The epidemiologic transition: a theory of the epidemiology of population change. Bulletin of the World Health Organization 2001; 79(2):161-170.
• ¹⁸
  Araújo-Jorge T. Embasamento técnico e sugestões para ações de controle das Doenças da Pobreza no Programa de Erradicação da Pobreza Extrema no Brasil Rio de Janeiro: Instituto Oswaldo Cruz 2011. [Nota Técnica N. 1/2011/IOC-FIOCRUZ/ Diretoria]. [acessado 2017 Ago 30]. Disponível em: http://www.fiocruz.br/ioc/media/NotaTecnica_1_2011_IOC.pdf
  » http://www.fiocruz.br/ioc/media/NotaTecnica_1_2011_IOC.pdf
• ¹⁹
  Malta DC, SilvaJúnior BJ. O Plano de Ações Estratégicas para o Enfrentamento das Doenças Crônicas Não Transmissíveis no Brasil e a definição das metas globais para o enfrentamento dessas doenças até 2025: uma revisão. Epidemiol. Serv. Saúde 2013; 22(1):151-164.
• ²⁰
  United Nations Children's Fund (Unicef). Health Equity Report. Analysis of reproductive, maternal, newborn, child and adolescent health inequities in Latin America and the Caribbean to inform policymaking. Summary. Report. New York: Unicef; 2016.
• ²¹
  Mello Jorge MHP, Koizumi MS, Tono VL. External causes: what they are, how they affect the health sector, how they can be measured, and a few subsidies to prevent them. Revista Saúde UNG 2008; 1(1):37-47.
• ²²
  Cordón JA. Dificuldades, Contradições e Avanços, na Inserção da Odontologia no SUS. Divulg. saúde debate 1996; 13:44.
• ²³
  Smeke ELM, Oliveira NLS. Educação em saúde e concepções de sujeito. In: Vasconcelos EM. A saúde nas palavras e nos gestos: reflexões da rede educação popular e saúde. São Paulo: Hucitec; 2001. p. 115-136.
• ²⁴
  Barata MML, Confalonieri UEC, Lima ACLL, Marinho DP, Luigi G, De Simone GC, Ferreira IB, Pinto IV. Vulnerability map of the state's population of Rio de Janeiro to the impacts of climate Buss [S.l.] :[s.n.]; 2011_.
• ²⁵
  Brasil. Instituto Brasileiro de Geografia e Estatística (IBGE). Censo 2010, 2011, 2015. [acessado 2017 Ago 30]. Disponível em: www.ibge.gov.br
  » www.ibge.gov.br
• ²⁶
  Ignacio FC, Silva MEC, Handam NB, Alencar MFL, Sotero-Martins A, Barata MML, Moraes Neto AHA. Socioenvironmental conditions and intestinal parasitic infections in Brazilian urban slums: a cross-sectional study. Rev Inst Med Trop São Paulo 2017; 59:e56.
• ²⁷
  Matos M, Bruno P. Programa de Controle da Dengue em Manguinhos (PCDM): reflexões sobre uma experiência coletiva no campo da saúde. In: Matos M, Bruno P. Território, participação popular e saúde: Manguinhos em debate Rio de Janeiro: Fiocruz; 2010.
• ²⁸
  Lima CM, Bueno LB, organizadores. Território, participação popular e saúde: Manguinhos em debate Rio de Janeiro: Fiocruz; 2010. p. 51-60.
• ²⁹
  Moraes Neto AHA, Pereira APMF, Alencar MFL, Souza-Júnior PRB, Dias RC, Fonseca JG, Santos CP, Almeida JCA. Prevalence of intestinal parasites versus knowledge, attitudes, and practices of inhabitants of low-income communities of Campos dos Goytacazes, RJ. Parasitol Res 2010; 107(2):295-307.
• ³⁰
  Sotero-Martins A, Duarte NA, Carvajal E, Sarquis MIM, Fernandes OCC. Microbiological and Quality Control of Recreation Areas. Rev. Eletrónica Gestão & Saúde 2013; 4(1):1075-1092.
• ³¹
  Bethlem AS. Estratégia Empresarial: conceitos, processo e Administração Estratégica. 4ª ed. São Paulo: Atlas; 2002.
• ³²
  Lima GBA, Carvalho NC, Herkenhoff DA. Avaliação de desempenho baseada na ISO 9004:2000: Estudo de caso em uma empresa de manutenção. INGEPRO - Inovação, Gestão e Produção 2010; 2(6)97-107.
• ³³
  Pinheiro R, Silva Júnior AG. Práticas avaliativas e as mediações com a integralidade na saúde: uma proposta para estudos de processos avaliativos na atenção básica. In: Pinheiro R, Silva Júnior AG, Mattos RA, organizadores. Atenção básica e integralidade: contribuições para estudos de práticas avaliativas em saúde. Rio de Janeiro: CEPESC; 2008. p. 17-42.
• ³⁴
  Brasil. Presidência da República. Casa Civil. Decreto n° 8474, de 22 de junho de 2015. Dispõe sobre as atividades de Agente Comunitário de Saúde e de Agente de Combate às Endemias. Diário Oficial da União 2015; 24 jun.
• ³⁵
  Carvalho AI, Bodstein RC, Hartz Z, Matida AH. Concepções e abordagens na avaliação em promoção da saúde. Cien Saude Colet 2004; 9(3):521-529.
• ³⁶
  Felisberto E. Monitoramento e avaliação na atenção básica: novos horizontes. Rev. Brasileira de Saúde Materno Infantil 2004; 4(3):317-321.
• ³⁷
  Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z. Avaliação: conceitos e métodos Rio de Janeiro: Editora Fiocruz; 2011.
• ³⁸
  Akerman M, Mendes R, Bógus CM. É possível avaliar um imperativo ético? Cien Saude Colet 2004; 9(3):605-615.
• ³⁹
  Thiollent M. Pesquisa-ação nas organizações. 2ª ed. São Paulo: Atlas; 2009.
• ⁴⁰
  Besen CB, Souza Netto M, Ros MA, Silva FW, Pires MF. The Family Health Strategy as an object of Health Education. Saúde Soc 2007; 16(1):57-68.
• ⁴¹
  Rootman I, Goodstadt M, Hyndman B, McQueen DV, Potvin L, Springett J, Ziglio E. Evaluation in Health Promotion: Principles and Perspectives Copenhagen: World Health Organization; 2001.

Publication Dates

• Publication in this collection
  Dec 2017

History

• Received
  30 Aug 2017
• Reviewed
  04 Sept 2017
• Accepted
  03 Oct 2017