Concepts and approaches in the evaluation of health promotion

Concepções e abordagens na avaliação em promoção da saúde

DEBATE DEBATE

Concepts and approaches in the evaluation of health promotion

Concepções e abordagens na avaliação em promoção da saúde

Antônio Ivo Carvalho^I; Regina Cele Bodstein^II; Zulmira Hartz ^III; Álvaro Hideyoshi Matida ^IV

^IEscola Nacional de Saúde Pública, Fiocruz. Av. Leopoldo Bulhões 1.480, sala 319, 3º andar, Manguinhos, 21041-310, Rio de Janeiro [e-mail]

^IIDepartamento de Ciências Sociais, Ensp/Fiocruz

^IIIGroupe de Recherche Interdisciplinaire en Santé (GRIS) e Departamento de Medicina Preventiva e Social da Universidade de Montreal

^IVAssociação Brasileira de Pós-Graduação em Saúde Coletiva e Assessoria de DST/Aids da Secretaria de Estado de Saúde do Rio de Janeiro

ABSTRACT

The demands and tensions surrounding evidence-based policy (EBP) as part of results-based management have frequently indicated a gap between these concepts and the complex nature of health promotion interventions. This article discusses the challenges associated with the conceptual field of Health Promotion and the requirements for "proof" of effectiveness and efficiency faced by managers, evaluators, and local agents in the development of inter-sector health programs. The authors identify the limitations of epidemiological trials for the evaluation of social policies and use arguments related to "theories of change" in order to discuss the relationship of the "constructs" in the social policy intervention model and provide the basis for the "analysis of the contribution" of its effects. Systematic reviews of the "realist synthesis" type are discussed, due to their capacity for highlighting the theoretical framework of a specific program and explaining the underlying action mechanisms common to different programs and/or contexts. The authors argue that the expression and maintenance of expected social changes require the construction of collaborative processes, considering the set of (bottom-up) stakeholders involved in all stages of the process of developing and evaluating interventions.

Key words: Program evaluation, Health promotion, Evidenced-based policies

RESUMO

Demandas e tensões em torno do uso de evidências em políticas públicas (evidence-based policy – EBP), como parte de uma gestão orientada por resultados, indicam muitas vezes um distanciamento entre as concepções e a natureza complexa das intervenções de promoção da saúde. Este artigo parte dos desafios associados ao campo conceitual da Promoção da Saúde e das exigências de "provas" de efetividade e de eficiência com que têm se confrontado gestores, avaliadores e agentes locais, no desenvolvimento de ações intersetoriais em saúde. São apontadas as limitações dos ensaios epidemiológicos na avaliação de políticas sociais e utiliza-se de argumentos relacionados às "teorias de mudança" com vistas a discutir a relação dos constructs no modelo das intervenções das políticas sociais e fundamentar a "análise da contribuição" de seus efeitos. Revisões sistemáticas do tipo realist synthesis são enfocadas devido sua capacidade de realçar marcos teóricos de um programa específico, explicando os mecanismos subjacentes de ação comuns aos diferentes programas e/ou contextos. Argumenta-se que a explicitação das mudanças sociais esperadas requer processos colaborativos, considerando o conjunto dos atores envolvidos (bottom-up) no processo de desenvolvimento e na avaliação das intervenções.

Palavras-chave: Avaliação de programas, Promoção da saúde, Políticas baseadas em evidência

Introduction

The emphasis on the production and use of evidence-based policy (EBP) as part of results-based management often indicates a gap between these concepts and the complex nature of health promotion interventions. In addition to the context of heavy ambiguity and intense uncertainties involving multiple stakeholders (intrinsic characteristics of such initiatives), there are different expectations by managers, professionals, and community groups towards the evaluation methods and practices. In the last decade, the demands and tensions surrounding evidence of efficacy and effectiveness, especially stemming from funding agencies, have generated a variety of approach models that are frequently reduced to the "rationalism" of experimental and relatively controlled designs that overlap with the dialogical approaches that are more "realist" and adequate for this type of evaluation research.

This paper deals with the challenges associated with a coherent construction of the conceptual field of Health Promotion and the requirement of "proof" of effectiveness and efficiency faced by managers, evaluators, and local agents in inter-sector health actions and programs. The main issue relates to the relationship between the interventions and the observed effects, applying the rules of scientific evidence of the International Union for Health Promotion and Education (IUHPE, 2003). The issue is to inquire as to the effectiveness of a given intervention, i.e., the process by which the initiative was capable of achieving its objectives or producing effects in the short, medium, and long term (Rootman, et al., 2001).

Tondeur (1996) reviewing 200 studies on health education or promotion initiatives catalogued by the UIPES (Union Internationale de Promotion et d'Education pour la Santé), observed that such initiatives contributed to: progress in knowledge; awareness of the risk focus; mobilization of decision-makers for the definition or adjustment of the legislative framework; and improvement in the state of health of certain populations, to the point of decreasing certain diseases, in addition to compressing health expenses. However, the author highlights the lack of studies demonstrating the long-term impact of educational actions devoted to promoting healthy behaviors, although stating that community-based interventions display the potential for significant and lasting social changes.

These and other findings from more recent reviews of the field (IUHPE, 2000; 2003) allow one to conclude that it is relevant to continue investing in multi-factor and inter-sector prevention and health promotion strategies and evaluative research, which can provide such initiatives with backing as a political priority. This prioritization is reinforced by the particularly striking lack of evaluation studies from the perspective of community initiatives, in contrast with a greater accumulation of experience in the literature consulted (especially in South America) on health-promoting actions focused on development and individual choices favoring better quality of life.

As an initial reflection to launch the debate, we chose to prioritize the dialogue between the conceptual field of Health Promotion and inter-sector actions focused on community development, and approaches for the evaluation of social programs, available in the literature, which we find more adequate for these interventions. As a reference we take the same approach as Thurston & Potvin (2003): Our interest here is particularly focused on evaluating social change (...) social change programs are those that aim to change the social practices of social agents, including their relationships...These programs are often rooted in principles of social justice and equity.

From this perspective, it is important to highlight that the strong idea in this debate is not limited to the concept of evidence-based practices in the field of Health Promotion (Nutbeam, 1996). It is also relevant to identify their use and limitations in the evaluation of multi-focal social development interventions, where participation and adherence by various partners and communities is a crucial variable for the success of the interventions.

The following are several underlying theoretical premises for the elaboration of this work:

• The use of "theories of change" with a view towards an in-depth discussion of the relationship between the "constructs" in the social policy intervention model and the basis for the "analysis of the contribution" of their effects (Holmes et al., 1993; Mayne, 2001; Pawson, 2003; Thurston & Potvin, 2003);

• The limits of epidemiological trials for the evaluation of social policies provide the point of departure for the incorporation of innovative approaches based on the inherent challenges and tensions in the process of producing evidence of effectiveness in community health promotion programs (Potvin, 1994; Hartz, 2000; Hughes & Traynor, 2000; McQueen, 2001; Potvin et al., 2001; Potvin & Richard, 2001; Sanderson, 2002);

• The expression and maintenance of the expected social changes requires, by principle and due to their complexity, the construction of collaborative processes involving the stakeholders as a whole (from the "bottom up") in all stages of the development and evaluation of the interventions (Sullivan et al., 2002);

• The responsibility and inherent challenges involved in the analyses of the effects of public policies spawned by heterogeneous interests cannot be taken on by isolated social groups (Hughes & Traynor, 2000); and

• Systematic reviews of the "realist synthesis" type highlight the theoretical framework of a specific program and also bring to the surface the underlying action mechanisms common to different programs and/or contexts, thereby favoring learning from the positive and negative effects of the various social theories (Pawson, 2002a).

Without intending to exhaust the literature review or the resulting arguments, our purpose is to focus on some preliminary lessons related to the advantages and limits of some methodological alternatives, as well as their possibilities for application in community Health Promotion actions. Our expectation is to foster the multidisciplinary and international debate, underline the benefits of these advances, and expand the visibility and validity of their contributions to this field of knowledge.

The conceptual field of Health Promotion and evaluation of inter-sector programs

The myth that community development initiatives are only partially susceptible to evaluation needs to be challenged, since there rests on the community development practitioners and managers as much obligation to consider effectiveness as with any other social intervention (Hughes & Traynor, 2000).

As a field under construction and in constant development, Health Promotion (HP) has consolidated as the point of convergence for a set of reflections and practices committed to surmounting the biomedical model. Beginning with the positive and expanded concept of health, and taking the social process of its production as the focus, Health Promotion has proven capable of convening interdisciplinary forms of knowledge and mobilizing inter-sector practices with an aim towards expanding the intervention model beyond both the biomedical approach and the growing medicalization of social problems. Health Promotion is thus based on, and chooses as it objective, the defense of life and human development.

From the Health Promotion point of view, the socioeconomic determinants of the health-disease process constitute an indispensable analytical reference, not only for theoretical reflection, but for designing public policies, programs, and interventions focused on improving quality of life, as well as evaluation approaches and methodologies. Health Promotion thus transcends the health sector and increasingly impacts public policies in general, engaging in dialogue with the diversity of fields, organizations, and subjects involved in the multiple dimensions of social life.

In Brazil, Health Promotion has developed within the transformation of health which in recent years gave rise to the Unified National Health System (SUS). Proposed by the Health Reform Movement, in 15 years the SUS has made its mark as one of the most successful Brazilian sectoral policies. Its democratic and participatory institutional design, its universalist and socially redistributive nature, and its technical capability to supply comprehensive health care constitute a milestone for social and political inclusion, conquered on the basis of principles that are analogous to those of Health Promotion.

However, the persistence of a socioeconomic context that generates sustained social inequality (unemployment, fiscal injustice, and social security inefficiency) penalizes the lower classes, maintaining broad population contingents under heavy social and health risk, a situation aggravated by centuries of regional inequalities.

Health Promotion thus appears as an opportunity not only to formulate an agenda for enhancing the National Health System, but also to point to a "new" agenda, re-qualifying health policy for the new millennium, recovering the banner of health reform in its dimension of "social change and the struggle against social inequalities," necessary for the construction of health and life with dignity.

This perspective requires overcoming the notion of the health sector as a voracious consumer of resources, competing with "productive" areas or even with "social assistance" areas, meanwhile valuing it as an essential component of human development, as part of an integrated and sustainable development model, viewed as both the goal and motor force of economic development. The issue is thus to view the health sector and the population's health as a fundamental economic investment for human and social development.

By critically challenging the biomedical paradigm and its insufficiency in the face of the multiplicity and complexity of factors intervening in the contemporary dimension of the health-disease process, Health Promotion calls into its field all the social and technical complexity required to face the challenge of producing health and quality of life. It is thus shaped as a social movement, defending innovative health agendas and strategies in all their dimensions. Health Promotion now represents an expansion and conceptual and operational re-qualification of the health issue in its growing complexity, wagering on new intervention policies and practices in the health-disease process and improved quality of life in general.

Beginning with the positive and expanded concept of health, which is written into the 1988 Brazilian Constitution, and surmounting the simplified definition of health as absence of disease, Health Promotion is based on the following concepts and values:

• It views health as quality of life, resulting from the satisfaction of vital individual and collective needs;

• It contends that vital needs are not defined normatively, but are socially established through pacts between subjects in economic, social, and cultural contexts that are historically situated and dated, involving universal access to a set of services and goods;

• It recalls that social needs are registered and legitimized in social contracts that define rights and duties – like the right to health and education, among others;

• It views health as a fundamental human right (the right to life), and thus as a universal ethical imperative and not merely as a socially determined right;

• It defines health as a means to (and capacity for) life, transcending normative parameters and reductionist approaches.

Thus focused, Health Promotion places at the center of its scope (and as a fundamental dimension of health and quality of life) the quest for "autonomy" by individuals and groups (the capacity to live life) and the pursuit of social "equity" (equitable distribution of this capacity among individuals and groups). A critical and expanded approach to Health Promotion points to the need for discussion and improvement of public policies, redistribution of powers, and the establishment of a new mix of rights and responsibilities in various spheres of social life, affecting professional relations / citizens, collectives / individuals, governments / society, economy / social, sectoral / extra-sectoral spheres, among others.

The pursuit of equity and the struggle against social inequalities becomes a high-priority issue. As a constitutional goal that is unattainable through the means available in the narrower sphere of health services, equity depends on the mobilization of political will and resources that extrapolate sector boundaries. To assume social inequalities as the main obstacle to health and quality of life for all (and the latter as both an ethical imperative and a requirement for a stable economy) highlights the need for a broad coalition in favor of more equitable, effective, and efficient public policies which should involve government and society and can and should be proposed by the health sector, considering its scope and legitimacy vis-à-vis social demands and the population's daily life.

Health's task thus falls not only to health professionals as subjects but to the citizenry as a whole, re-qualifying the issue of "social participation," not exclusively in relation to the health sector as a field of conceptualization and practice in health, but to the various other fields of professional knowledge and practice, with the emphasis on interdisciplinarity and thus on inter-sector actions. "Participation" and "inter-sector action" are thus two key categories for HP, besides serving as operational principles in the new strategies for building health and quality of life.

In reality, based on these definitions, which qualify Health Promotion as a paradigmatic set of concepts and practices oriented towards the social construction of autonomy and equity, adopted as fundamental parameters for health and quality of life, Health Promotion can be understood as a strategy for change in at least three levels, namely:

• Health care, requiring demedicalization and reorientation of health care services so as to be able to harbor and support individuals and groups in constructing autonomy;

• Local management and development, in which inter-sector actions for health and quality of life are implemented as the result of (and motivation for) empowerment of populations and subjects; and

• The national development model and policies, where the subordination of the economic to the social is pursued, and where the determinants of equity are actually defined.

The richness of the international debate in the field of Health Promotion derives mainly from the fact that it inspires policies, actions, programs, and initiatives through the design of multi-focal interventions aimed at broad and sustainable social changes, where the emphasis lies on the roles of both individual subjects and social organizations as well as that of institutional development (Abrasco, 2002). Thus, a highly significant contribution to the debate derives form the understanding of Health Promotion as a field of applied knowledge, built on the basis of the design, implementation, and evaluation of policies and programs, in which evaluation is crucial to the dialogue between theory and practice and thus to the consolidation of the field (Rootman et al., 2001). In fact, the concept of Health Promotion programs redefines and expands the traditional approach to health problems in two ways, by transcending:

a) the risk approach which takes the individual as the unit; and

b) health problems defined on the basis of a specific biomedical dimension, focusing rather on broad and sustainable social changes within socially defined contexts.

Health Promotion is thus developed through a set of initiatives and programs aimed not only at the problems of disease prevention and identification of individuals and groups subject to risk factors (MacKinlay, 1996; Kawachi, 2002; Lochner et al., 1999), but especially through inter-sector and multidisciplinary community approaches focused on the social determinants of health for the consolidation of so-called "social capital" and the empowerment of social subjects and stakeholders.

From this perspective, the inherent and constitutive dynamic of Health Promotion programs derives from the fact that they involve inter-sector and participatory goals aimed at social development and empowerment, from the perspective of reduction of inequalities and expansion of social inclusion. They encompass a broad range of strategies for changes in schools, health services, communities, and the environment, among other areas. Such programs are complex by nature and presuppose mobilization, participation, and knowledge of "prevailing cultural and social standards."

Aimed at sustainable social changes that depend on a convergent multi-sector agenda, health-promoting actions prioritize strategies and actions that incorporate socially shared values and principles. Therefore, the participation and reflexive and organizational capacity of the communities and stakeholders are crucial dimensions in Health Promotion programs. In short, the basic characteristics of such programs are: a broad scope; participatory strategies; long-term results; and flexible planning (Potvin et al., 2001).

The complex nature of Health Promotion interventions requires equally complex evaluation approaches, since the programs are permeated by multiple external influences and uncontrolled variables (Potvin, 1996). Thus, methods designed for linear medical interventions do not apply to these programs and strategies (McQueen & Anderson, 2001). Various authors further argue that the design of a simple causality chain (and thus that of the intervention's linear effects) should not be taken into consideration in the evaluation of community programs, given that establishing causal relations for social phenomena is a particularly difficult task (Mohr, 1999; McQueen & Anderson, 2001).

Evaluative approaches in inter-sector policies and initiatives focused on local development require innovative and complex approaches, since identifying theories and mechanisms by which actions and programs lead to changes within a given social context involves the utilization not only of varied methods, but especially of qualitative methodologies that are coherent and consistent with the respective problems, with a recurrent understanding of meanings, perceptions, and cultural aspects. At any rate, to make progress with this debate, what matters is to emphasize that the same principles and research criteria should provide the basis for both the quantitative and qualitative methods.

In short, despite its importance, the Health Promotion field has been challenged to show theories, results, and effects both for empowerment and the right to health, as well as the improvement in quality of life and the environment. Thus, the relationship between multi-focal initiatives and programs and the identification of evidence of results and changes appears to be the guiding thread in evaluation methodologies currently under discussion in the Health Promotion field (Rootman et al., 2001), revealing a certain preoccupation with the variability of methodological procedures adopted in the field.

Theory and practice in the evaluation of Health Promotion programs

Programs are theories... evaluation is theory-testing... One of the involuntary virtues of the theory-driven approach to evaluation is that it forces to contemplate programs in their true and awesome complexity (Pawson, 2003).

If community development needs to be evaluated like any other social intervention, some impediments include problems of causality and attribution, in which the context is simultaneously a dependent and independent variable, and the need to focus on methodological processes and approaches for its measurement. Among the alternatives for dealing with this problem without loss of methodological rigor, the theoretical orientation of evaluation is an indispensable procedure, according to various authors (Holmes et al., 1993; Hughes & Traynor, 2000; Sanderson, 2002).

According to Pawson (2003), two strategies shape these theory-driven evaluation (TDE) approaches: the theories of social changes and realist research, which we will deal with as complementary. The core idea in the former case is that programs are created through iterative sequences of theories and corresponding objectives whose evaluation requires their verification at each stage. So-called "realist" evaluation fundamentally targets the mechanisms sustaining programs with greater complexity in their respective contexts. The basic ingredients in this complexity are distributed throughout a chain operated by makers of policies/programs, professionals, and participants who mobilize various theoretical frameworks beginning, for example, with a theory of social exclusion that justifies the intervention, moving on to theories of social mobilization and organizational theories of the operational agents, and reaching theories of behavior change in the target population.

If a program's theory, aimed a given social change, is to allow an understanding of the dimensions and variety of processes that can lead to the success or failure of the intervention (according to a log frame), a first stage of negotiation is indispensable for its construction (BID, 1997; Contandriopoulos et al., 2000). This stage is indispensable in the evaluability studies that should precede any evaluation, thereby minimizing future conflicts. After all, if one of the pillars of health promotion is community empowerment, then

...it might sometimes be about decreasing the power or control of one group over another... it explicitly requires an accord among key players about what social change means to them (Thurston & Potvin, 2003).

The option for a TDE-type evaluation model, which takes into consideration not only causality but also conditionality (contextualization or responsiveness) in the analysis of interventions, likewise means the utilization of designs that incorporate quantitative/ qualitative techniques (Datta, 1997b), where it is more appropriate to speak of an "analysis of contribution" than "attribution", as proposed by (Mayne, 2001), with the following characteristics:

• Developing a more detailed chain of results, showing how an intermediate result leads to another;

• Locating variations in the program and its impacts, where differences in the program can be identified (in time, between locations, and between different target groups), gathering data referring to corresponding differences;

• Gathering information that adds credibility to the intervention's contribution, considering the measurements of results or effects;

• Including both continuous measures (performance monitoring/ measurement) and occasional measures, responding to evaluation questions;

• Recognizing that by definition the observed products or effects are influenced not only by the programs' action but also by external factors (like other programs and political decisions) as well as socioeconomic factors, considering the influence they may have; and

• Interviewing "enlightened" persons about their opinions concerning the extent to which the program is making a difference, whether overall or at some intermediate point in the chain of results; performing case studies, performing one or several specific cases within the program that can provide confirmative evidence.

Some experiences with the evaluation of community programs, incorporating a mix of previously described approaches in different degrees and models, have been conducted with promising results for organizational learning, particularly in the British experiences with social health and environmental programs (Hughes & Traynor, 2000; Shah & Gregson, 2001; Sullivan et al., 2002; Mickwitz, 2003). Given the space constraints here, we will merely comment briefly on the evaluations conducted at the local and national levels concerning the actions related to the so-called "Health Action Zone" (HAZ), a good illustration of the latter's use aimed at encouraging a reading of the original texts, due to the careful planning and "realist" evaluation (Shah & Gregson, 2001; Sullivan et al., 2002).

The local HAZ experience in Luton (LHAZ), where the program was implemented in 1998 and aimed to reduce social inequalities with actions to improve the population's health and quality of life, had four main components or thrusts in its work: Health and Social Care (access and service modernization); Capacity for Health (community development and capacity-building); Healthy Environments (housing, environment, and transportation); and Structural Inequalities (employment and anti-poverty actions).

Since the original design, the responsibility of the operational organizations was clear not only in achieving the contractually defined results, but also in communicating how and why outcomes emerge and the way that they do, thus characterizing them as Learning Organizations. Evaluation was expected to play a fundamental role by supporting the stakeholders in their activities, allowing an analysis of the processes and results achieved. The project's construction began with a participatory planning workshop where two major objectives were perceived:

1) To elicit theories of change (log frame): articulate a shared vision; look at what needs to be done at the national, regional, and local levels; explore what could help and hinder the process; set realistic, achievable and measurable outcomes; and

2) To learn about research methodologies and appropriate data collection tools.

From the point of view of controlling the implementation, monitoring was done every two months and the log frame reviewed every four months. The most interesting preliminary results relate to the importance ascribed by managers to continually evaluate the project's implementation and the understanding of the theoretical basis for the log frames ("The system is only as good as it's implementation;" "Thinking back to projects on food and healthy living... we didn't really know what we were trying to achieve... so theories change by asking us what is it that we were trying to achieve at the beginning and helped to qualify the process "), as well as the need for resources and champion evaluators to allow for an intervention/evaluation with these characteristics. These conclusions by Shah & Gregson (2001) concerning the national evaluation of the HAZ agree with the advantages and limits of this approach, as similarly highlighted by (Sullivan et al., 2002), and should stimulate our own debate in Brazil.

Final remarks

...because programs are theories incarnate, they can be shaped by the vision of people beyond those with direct responsibility for their conduct (including the theories of those long dead!... the task of transferring knowledge in evaluation belongs to theory (Pawson, 2003).

Assuming that a sector's challenges involve the redefinition of policies and practices, Health Promotion expands the focus of health based on the understanding of the complexity of the social changes under way, reiterating the importance of acting not only on the issue of demedicalization and reorientation of health services and practices, but especially in the sphere of local development and empowerment, in the defense of public policies and a more effective and equitable national development. Health Promotion reiterates the importance of partnerships between teaching and research institutions, public administration, and social and community organizations, aimed at the effectiveness and sustainability of the experiences and proposals.

In its most critical watershed, Health Promotion updates and expands the debate on the social, cultural, political, and economic determinants of the health-disease process, reaffirming health as an ethical imperative and citizen's right. Health Promotion currently provides the inspiration for a vigorous debate on healthy public policies vis-à-vis various initiatives to deal with social inequalities.

The current global debate in the field of Health Promotion points to the understanding that the pursuit of equity in all its dimensions cannot succeed without efficient and effective public policies. For society at large, the issues of monitoring and evaluation and the production and use of evidence are therefore highly relevant. Evaluation is thus a strategic issue for the production of knowledge, confirming or refuting theoretical postulates and producing proof of effectiveness, fundamental for decision-makers, managers, professionals, local agents, and society at large. As discussed here, evaluative approaches have the merit of reviewing theories, allowing for a dialogue between various experiences and methodologies. From the evaluative perspective, Health Promotion assumes the existing tensions between theoretical and methodological issues and the diversity of methodological approaches, in addition to pointing to the complexity involved in the relationship between production of evidence and its utilization in generating more healthy public policies.

Constituting a fundamental component for consolidating the field of Health Promotion, evaluation allows one to understand that the complex and innovative nature of experiences under way derives from interdisciplinarity and interlocution among the various managers in public administration, giving rise to initiatives and programs with an emphasis on inter-sector actions and the formulation of integrated, participatory local development agendas. The expansion and legitimization of the debate on health and quality of life requires participation and adherence by communities and stakeholders, crucial factors for the success of the experiences and for evaluative approaches and methodologies.

Defending the position that the most radical and innovative perspective in evaluation is that of the "realist" approach (which we consider indispensable for studies on the implementation or effectiveness of social programs), we contend that the principal unit of analysis in evaluation, or assessment of the potential for generalization of lessons learned on social changes, is related much more closely to the theories than the programs themselves. It would thus be more productive to prioritize the study of a set of initiatives that share the same theories, rather to deal with single initiatives vis-à-vis each problem. This directionality in the foci of observation should be prioritized both in the sphere of original studies and the review/synthesis of multiple studies (Pawson, 2002a; 2002b).

In the latter case, a systematic review of the literature should also be oriented towards a critique of similar mechanisms that theoretically generate social changes, as proposed by different inter-sector actions, labeled or caricaturized as "carrots, sticks, and sermons" (Pawson, 2003). The specific inclusion of the analysis linked to the use of programmatic incentives or the "carrot theory," as exemplified by (Pawson, 2002a), surmounting the limits of meta-analysis and narrative review, could favor the improvement of studies such as those that have been conducted with the support of UIHPE/PAHO (Salazar et al., 2003).

To launch the debate concerning the feasibility of managers and evaluators dealing in practice with the complexity of Health Promotion programs in "realist" evaluation studies, several points can sum up our preliminary discussion:

• Systematic indicators in the field of Health Promotion should reflect the guidelines in the conceptual framework. The development of favorable personal attitudes and the reorientation of health actions require "healthy" public policies with community empowerment;

• Epidemiology needs to become involved in a sociological/participatory paradigm, featuring patient interdisciplinary work in the pursuit of "local understanding rather than universal truths" (Schawab & Syme, 1998);

• Scientific evidence is merely one among other criteria for setting priorities, which (as in other social interventions) also respond to market (client-financier) and political/ ideological motives.

Colaboradores

R Bodstein trabalhou na concepção teórica e na discussão conceitual; Z Hartz, na concepção teórica e na orientação da pesquisa bibibliográfica, em especial no que tange ao campo da avaliação em promoção da saúde; A Ivo, na sistematização da discussão metodológica e conceitual e das conclusões; e AH Matida contribuiu na pesquisa bibliográfica e na discussão conceitual.

• Abrasco 2002. Relatório da I Oficina sobre Perspectivas e Desafios da Avaliação em Promoção da Saúde. GT Promoção da Saúde e Desenvolvimento Local. V Congresso Brasileiro de Epidemiologia, Curitiba.
• BID [Inter-American Development Bank] 1997. Evaluación: Una herramienta de gestión para mejorar el desempeño de los proyectos. Oficina de Evaluación (EVO) Banco Interamericano de Desarrollo.
• Contandriopoulos AP, Champagne F, Denis JL & Avargues MC 2000. L' évaluation dans le domaine de la santé: concepts et méthodes. Révue d'Epidemiologie et Santé Publique 48:517-539.
• Datta LE 1997. Multimethod evaluations. Using case studies together. Evaluation 344-349.
• Hartz Z 2000. Pesquisa avaliativa em promoção da saúde, pp. 131-137. In PM Buss (org.). Promoción de la salud pública: una contribuición para el debate entre las escuelas de salud pública de América Latina y el Caribe (Mimeo).
• Holmes D, Teresi J & Ory M 1993. Serendipity and pseudoscience. Evaluation & the Health Professions 16: 363-378.
• Hughes M & Traynor T 2000. Reconciling process and outcome in evaluating community initiatives. Evaluation 6(1):37-49.
• IUHPE 2000. The Evidence of Health Promotion Effectiveness. ECSC-EC-EAEC
• IUHPE 2003. The Evidence of Health Promotion Effectiveness. ECSC-EC-EAEC, Paris.
• Kawachi I 2002. Social epidemiology (Editorial). Social Science & Medicine 54:1.739-1.741.
• Kawachi I & Berkman LF 2000. Social cohesion, social capital, and Health, pp. 174-190. In LF Berkman & I Kawachi (eds.). Social epidemiology Oxford University Press, New York.
• Lochner K, Kawachi I & Kennedy B 1999. Social Capital. A Guide to its Measurement. Health and Place 5:259-270.
• Mayne J 2001. Addressing attribution through contribution analysis: using performance measures sensibly. Canadian Journal of Program Evaluation 16(1):124.
• Mc Kinlay JB 1996. More appropriate evaluation methods for community level health interventions (introduction to the special issue). Evaluation Review 20(3):237-243.
• McQueen DV & Anderson LM 2001. What counts as evidence: issues and debates. In IG Rootman et al. Evaluation in health promotion: principles and. World Health Organization. Copenhagen.
• Mickwitz P 2003. A framework for evaluating environmental policy instruments: contexts and key concepts. Evaluation 9(4):415-436.
• Mohr LB 1999. The Qualitative Method of Impact Analysis. American Journal of Evaluation 20(1):69-84.
• Nutbeam D 1996. Achieving "Best Practice" in Health Promotion: Improving the Fit Between Research and Practice. Health Education Research 11(3):317-326.
• Pawson R 2002a. Evidence-based policy: in search of a method. Evaluation 8(2):155-156.
• Pawson R 2002b. Evidence-based policy: the promise of realist synthesis. Evaluation 8(3):340-358.
• Pawson R 2003. Nothing as practical as a good theory. Evaluation 9(4):471-490.
• Potvin L, Haddad S & Frohlich L 2001. Beyond process and outcomes evaluation: a comprehensive approach for evaluating health promotion programs, pp. 45-62. In IG Rootman et al. (eds.). Evaluation in health promotion. Principles and perspectives. World Health Organization, Copenhagen.
• Potvin L & Richard L 2001. The evaluation of community health promotion programmes. Evaluation in Health Promotion. Principles and Perspectives. E. Ziglio. Copenhagen, World Health Organization: 213-240.
• Potvin L & Lessard R 1994. Le paradoxe de l'evaluation des programmes communautaires multiples de promotion e la santé. Ruptures: 45-57.
• Rootman IG et al. 2001. Evaluation in Health Promotion: Principles and Perspectives. Copenhagen, World Health Organization.
• Salazar L, Vélez J & Ortiz Y 2003. En busca de evidencias de efectividad en promoción de la salud en América Latina.
• Sanderson I 2002. Finding out works: evaluating community-based action for promoting positive outcomes for individuals, families, and neighbourhoods. The UK Evaluation Society: 4-5.
• Schwab M & Syme L 1998. On paradigms of community participation and the future of public health. AJPH 87(12):2.049-2.051.
• Shah A & Gregson D 2001. Evaluation of Luton Health Action Zone. Annual Conference of UKES.
• Sullivan H, M Barnes & Matka E 2002. Building collaborative capacity through theories of change: early lessons from the evaluation action zones in England. Evaluation 8(2):205-226.
• Thurston W & Potvin L 2003. Evaluability assessment: a toll for incorporating evaluation in social programs. Evaluation 9(4):453-470.
• Tondeur L 1996 - L'ëducation pour la santé, un efficacité sous conditions. Actualité et Dossier en Santé Publique, 16:XXII-XXVI.

Publication Dates

• Publication in this collection
  05 June 2007
• Date of issue
  Sept 2004