Evaluation of initiatives and intersectorial programs in health: challenges and learning

Abstracts

O objetivo deste artigo é contribuir para a avaliação das iniciativas voltadas para a construção de arranjos intersetoriais no âmbito das políticas públicas de saúde e de proteção social. São tomados como focos analíticos a iniciativa de Desenvolvimento Local Integrado e Sustentável de Manguinhos (DLIS-Manguinhos) - uma proposta de base territorial e comunitária associada à estratégias de colaboração e negociação bottom up- e a experiência de implementação do programa federal de transferência condicionada de renda Bolsa Família (PBF). Nesta perspectiva, aspectos ligados às redes de mobilização social, perfil dos atores envolvidos, tipos de incentivos e níveis de integração institucional são tratados como elementos cruciais para a análise do alcance de programas e iniciativas que articulam propostas intersetoriais. Conclui-se que a interface e o diálogo entre pesquisa, avaliação e acompanhamento de processos decisórios constituem eixos centrais para o maior aprendizado social e institucional na área.

Intersetorialidade; Avaliação; Promoção da Saúde; Iniqüidades sociais

ARTICLE

Evaluation of initiatives and intersectorial programs in health: challenges and learning

Rosana Magalhães; Regina Bodstein

Departamento de Ciências Sociais, Escola Nacional de Saúde Pública Sergio Arouca, Fundação Oswaldo Cruz. Rua Leopoldo Bulhões 1480/910, Manguinhos. 21045-210 Rio de Janeiro RJ. [e-mail]

ABSTRACT

The objective of his article is to contribute to the evaluation of the initiatives aimed at the creation of intersectorial arrangements in the scope of public health policies and social protection. The focus is on the Integrated and Sustainable Local Development of Manguinhos (DLIS-Manguinhos) - a territory and community based proposal associated to bottom up cooperation and negotiation strategies - and the experience of establishing the conditioned cash transference federal program called Bolsa Família (Family Grant Program - PBF). In this perspective, the aspects related to social mobilization networks, profile of the players involved, types of incentive and levels of institutional integration are treated as crucial elements in the analysis of the programs and initiatives that articulate intersectorial proposals. It is concluded that the interface and dialog among research, evaluation and follow up of decision processes constitute the central axes for better social and institutional learning in the area.

Key words: Intersectorial, Evaluation, Health promotion, Social disparity

Introduction

As of the 1990s, the evaluation of the low effectiveness of social policies and the criticisms to the paternalist profile - residual in poverty fighting actions in Latin America - has gained repercussion through important international promotion agencies, among them the World Bank¹. In Brazil, the policies and programs aim at the protection of poor families against changes in their income and consumption by means of cash transferences, and also aim at guaranteeing investments for the improvement of their quality of living. In this course of action, the gains in school attendance and greater access to comprehensive health care, besides the opportunities to generate income and work, become crucial to the creation of benchmarks and recommendations to overcome the cycle of poverty transmission throughout generations.

In the health area, discussion on social determinants and the mechanisms that produce inequalities redirects² the agenda beyond the benchmarks of the sector. In this path, literature on the evaluation of policies and social programs - especially aimed at discussing implementation processes and concerned about generating knowledge applied to public and health policies (evidence based policy) - contributes to the creation of new analytical and methodological approaches³. Amid the top down planning criticism, evaluation emerges as a permanent dialogue in which officials, technicians, users and the policy communities significantly influence dynamics and effects of interventions⁴. The programs are then understood as action systems and as the creation of social and technical networks involving broad alliances and multiple interests⁵. Thus, understanding successful actions incorporates the analysis of the involvement of decision makers, officials, professionals, technicians and population.

In this perspective, the article attempts to contribute to the analysis of the strategies that increase cooperation among different sectors, government levels and players by means of an intersectorial cooperation that includes the mobilization and appreciation of a group of social organizations that are present an active in specific contexts^6,7. Political disputes and electoral interests traverse dialogues and partnerships, harming the development of an integrated agenda to implement these actions. Thus, we seek to recognize the complex and challenging nature of these interventions through an exam, even if partial, of two actual experiences - The Integrated and Sustainable Local Development of Manguinhos (DLIS) and the Bolsa Família program (PBF) - as they pressure traditional coordination mechanisms, recover previous learning experiences and involve a greater cooperation among multiple agents and social groups in different institutional and community spaces.

Social initiatives: flexibility, complexity and new governance networks

In the last decades, different public initiatives were carried out in the scope of social policies and programs aimed at reducing inequalities and improving life quality of vulnerable communities and populations. In general, such programs have, as main direction, the idea of rationalizing and improving already existing social actions and interventions, which were mainly related to health, education, culture, basic sanitation and public safety issues, by means of the coordination of these services, or the addition of new ones, covering existing gaps^8,9.

The evaluation of these initiatives imposes a dynamic understanding of the creation of players and interests, since such a process tends to shape activities and objectives of the programs besides unforeseen effects and results¹⁰. However, one of the difficulties in this intersectorial articulation is due to the fact that some sectors act through formal procedures described in Constitutions and laws and others represent organizations that have flexible power and interest¹¹. What frequently happens is that there is a strenuous negotiation process and the maximization of gains and minimization of losses in a permanent trade off, so as to adjust to interests and engagements that define the course of events and the performance of proposed activities.

Locally defined issues and priorities are negotiated with community leaderships and organizations, assuming these social groups and spaces are representative and legitimate¹². Thus, these are initiatives that shape complex and open organization systems which act on human and non human resources, making actions and services available through mobilization of popular, technical and scientific knowledge - an intricate multisectorial arrangement responsible for the implementation of the intended changes. According to Barnes et al. such initiatives¹³ can be characterized by different levels of vertical and horizontal partnerships and by medium and long term periods, surpassing the traditional periods of government.

Another important dimension which contributes to a great complexity of these experiences, is social capital vis-à-vis community empowerment¹⁴. As Coleman¹⁵ points out, social capital is an aspect of social structure that facilitates the obtention of certain purposes and advantages, which otherwise couldn't be obtained. It is vital for the development of initiatives and emerges as an important aspect of evaluation approaches, since the analysis allow evaluating the degrees of acceptance and engagement of communities and, therefore, possible strategies to strengthen partnerships and positive sociability circuits, even in contexts of deep social inequality and extreme violence¹⁶. For Jackson et al.¹⁷ in the perspective they call community empowerment and community capacity there is a combination of capacities, talents and social- environmental conditions inside and outside the communities, which may potentialize or not the manifestation of existing talents and abilities in the community.

Several times, a social initiative or program with well defined goals fails in recognizing the complexity of shortage and of family's problems in an intricate network of needs. This happens because no sectorial program is capable of separately facing the multiple aspects of social problems. Understanding the differences and points of contact among recent intersectorial experiences in the Brazilian scenario contributes to the systematization of multiple logics that cross social and health policies and, at the same time, consolidate important learning points about opportunities and dilemmas that involve creating alternatives and reaching results.

Community participation, intersectorial relations and local administration: the Manguinhos experience

In Brazil, Integrated and Sustainable Local Development (DLIS) programs were designed in the mid 1990s from a concept of "local development as a social process that gathers economic advancement with redistribution and life quality improvement for the community" carrying along an environmental sustainability perspective. Such a process involved intersectorial policies vis-à-vis social support network organizations in a certain social-territorial area¹⁸.

The DLIS experiences in the period took place in small municipalities in the interior of the country, giving priority to mechanisms that generate income and jobs, through public-private partnerships¹⁹. In the end of the 1990s, such a perspective was still unheard of in great metropolitan regions and, especially in great peripheral and slum areas, where a considerable share of the population virtually excluded from citizenship rights. Based on this model vis-à-vis a broad concept of health practices, in 1999, the Manguinhos DLIS program was created. The beginning took place when a coordination group was created consisting in representatives from academic institutions, state and municipal offices and state and private companies, led by the Public Health National School (ENSP/FIOCRUZ), aiming at an integrated action to improve life quality in the region. The group, which was responsible for the coordination from the beginning, involved ENSP researchers to follow the program and develop an evaluation proposal on health promotion and community development. The participation of the evaluation group in the coordination of the DLIS was essential for the analysis of the decision process, from the action planning and the creation of a local agenda to the understanding of difficulties to implement the program.

The perspective of the evaluation of Manguinhos program attempted to examine in the intervention that was inspired by the DLIS proposal and understood health promotion as a radical and systemic point of view - based on a social determinant approach^{20, 21}. The evaluation approach adopted involved monitoring and systematizing the program's actions so as to cooperate with the creation of the proposal, guarantee the propagation of the methodology to implement DLIS - and, above all, seek what could be characterized as an innovative practice to promote health through the involvement of the ENSP Health Unit. The interface among research, scientific methodology and decision process, and moreover, the frail coexistence of these fields, as Rütten²² shows, were at issue.

The evaluation intended to embrace an especially complex intervention, with a number of players, conflicts and interests at stake, but which reasserted the involvement and active cooperation of several state sectors, town hall and companies in the region, and principally, the mobilization and active participation of inhabitants and social organizations present in the area²³. Facing such a challenge, the understanding of the insufficiency of the evaluation only seeking results and effects and the selection of some key aspects to follow the program's implementation dynamic were essential. The evaluation approach focused on what seemed essential: the dynamic to create representativeness and legitimacy of actions through community mobilization (local forums) mechanisms and strategies, local participative diagnosis and agreement on an agenda of multisectorial development. In this course, the growing rates of violence and its impact not only on life quality, but also on sociability, cooperation and collective action standards were identified as crucial elements. Thus, the context was understood not only as a geographic or institutional space, but also as a social space, understandable by means of a set of rules, norms, values and relations and which define the development of the programs²⁴.

In this case, the evaluation concentrated on a proposal that was introduced in a highly unfavorable context from the point of view of the lack of services and public equipments in general and especially from the perspective of citizenship and trust in the public power. The evaluation approach was benefited by knowledge about the conception of the main characteristics of the program vis-à-vis and the knowledge of the context, which made the decision making process and the debate of proposals and actions with the community in Manguinhos difficult. Thus, the priority were the analyses of the difficulties and challenges of a proposal in a highly conflictive organizational and participative scenario and also unfavorable to cooperative actions. Registering and understanding political conflicts that were becoming aggravated and hindered cooperation and horizontal dialogue became important activities as well as the analysis of the investment of resources for the proposed interventions in housing, urban infrastructure improvements and income generation.

In Manguinhos's case, the difficulties to strengthen the intersectorial relations were concentrated in the intergovernmental scope, as well as in the local context based on oppositions and ruptures inside communities. The evaluation process and the dialogue about the program was carried out based on support and active participation of a group of inhabitants, identified as "cultural mediators", that is, translators that are capable of expressing the logic and demand of inhabitants and the bigger picture in which the local communities are included. The cooperation of these players began in the participative diagnostic phase in that region, when inhabitants could voice their main concerns and demands. The identification and work with the mediators, inhabitants and local leaders belonging to NGO's and the ENSP Health Center made the work feasible in a region of extremely difficult access because of the high violence rates. Thus, the evaluation process could not only recognize the importance of the leaders' work to mobilize a local circuit of sociability and solidarity - essential for life quality and for the decrease of local violence - but also make the "social-cultural capital" present in the community more systematic and visible. In general, the experience revealed the importance of understanding local actors as agents who are able to understand what they do while they are doing it. Understanding such groups and individuals as reflexive agents involved every day in the Manguinhos region helped overcome traditional normative interpretations about local administration and the reach of social programs and interventions. As Giddens points out, this approach means the knowledge the agents have of their own history is a powerful influence to that motivates change²⁵.

Institutional integration and local cooperation to reduce poverty: The Bolsa Família Program case

The Bolsa Família Program was created in October 2003 when federal programs - Food Grant, School Grant, Food Card and Gas Voucher - were unified and it therefore carries changes related to governability and management of resources in the scope of cash transference policies existing in different governmental agencies. Facing criticism due to the superposition and fragmentation of public policies in social and poverty issues, Bolsa Família, whose coverage reached around 11 million families in 2008, joined objectives, goals and conditions required for user of previous programs so as to maximize efforts and avoid parallelism, discontinuities and inefficiency. Beside, the conception of the Bolsa Família Program as the entrance for the set of social policies, articulating actions to follow up on children's and teenagers' school attendance, vaccine schedule and pre-natal care and not only cash transference to the poorest, guaranteed a better political and social support.

As Draibe²⁶ analyzed, after two decades of institutional reconstruction of assistance programs in the country, opposition and incompatibility between universalizing and focused interventions in the area of social protection is reviewed and actions aimed at vulnerable groups are no longer seen as residual or strictly compensatory. The idea that poverty is a complex problem - with a number of aspects that require a combination of monetary and non monetary resources in order to gain cultural capital, health conditions and occupational inclusion through intersectorial actions - is adopted as an important axis of the "program's theory"²⁷.

However, the institutional consequences of such arrangement revealed the difficulties of integration and cooperation among different agencies and government levels. On one hand, the unification of previous programs linked to the Ministry of Education, Health and Social Assistance represented a loss of resources and sectorial governability, reverberating in different adhesion levels. In the health area, such power displacement was combined to the exhaustion of the nutritional risk criteria for the selection and follow up of benefited parties. On the other hand, in the middle of the undefined roles and rules for financial cooperation among states and municipalities that characterize the Brazilian federative system, the intergovernmental articulation in the program's scope remained limited²⁸. Even though a complement to the benefits destined to the families from the state and municipal governments had been defined through the Cooperation Terms in practice, the lack of strong political and financial stimulation to consolidate these partnerships often caused a simple replacement of local initiatives to transfer cash for the federal government.

If there are difficulties in an intergovernmental organization, the intersectorial integration in the local level also tends to remain fragile before mechanisms and incentives mature in this field. In the case of the Bolsa Família, as - besides the direct transference of the monetary benefit to families through Caixa Econômica Federal - there are conditions required in the education area and in basic health care; previous joint work experiences, exchange of information, and encouragement for greater levels of involvement of officials and technicians from health municipal offices, assistance and education are crucial. It's important to remember that the extension of the Unified Registry of Social Policies (CADÚNICO) coverage - created in 2001 and included in the program as the main mapping and selecting tool to choose beneficiaries and guarantee of shared information - faced difficulties due to the limited resources and training. In the local scope, this has been a recurring complaint from the officials revealing a disorder between the speed of registry expansion and the necessary investments for a great managerial capacity, offering equipments and other support structures²⁹.

In some local experiences, information systems with the number of registered families being followed up or disconnected remain superposed, out of date or incomplete³⁰. The program administrators and implementing agents in different municipalities of the country often ignore or have variable information on the motives that lead families to wait long periods to receive benefits after being enrolled or even to have their benefit canceled. This fact confirms the existence of a feeling of loss of capacity to interact, propose changes and participate in a meaningful way to the course program on a local level.

In this aspect, it's important to highlight that the information on families assisted in each region, the disconnection criteria and the regularity of the payment and value of the benefit are important strategic aspects for the program design as a social right and also for the strengthening of intersectorial and community partnership involving officials and other stakeholders. Actually, Bolsa Família, not counting on a monetary line of "official"³¹ poverty, adopted arbitrary per capita income amounts to define who is poor and extremely poor in the perspective of consumption observed as well as deep regional differences in the destitute profile. Evaluation researches on program focus have shown important discrepancies as to the results found^32-34. Thus, reaching the poorest among the poor in the scope of the program is still a challenge. Often, the eligibility criteria, as Lidert et al.³⁵ point out may not be translated in precise definitions of the target population. The registry strategies adopted and the reinterpretation of the selection criteria of the beneficiaries in the street-level bureaucracy may affect focus result different from what was originally formulated in the programs operational design.

In this process, complying with selection criteria of beneficiaries also involves the evaluation of legitimacy and coherence of normative principles used. Even though Bolsa Família reaffirms the idea that transferring income without promoting human capital does not reduce poverty, limiting the follow up of educational gains by only analyzing school frequency ignores performance issues, essential to make actions more effective in this area³⁶. As to health, Estrella and Ribeiro³⁶ suggest that the quality of public health services, regardless of the offer, tends to be neglected in most municipalities. Likewise, the low performance related to the implementation of complementary programs for income generation and autonomization of the families compromises the broader character of objective and goals. The concern with the interdependence of actions in order to reach greater program effectiveness can also be noticed as to the nutritional gains with the increased access to food by means of cash benefits. As a recent research carried out by IBASE²⁹ showed, families tend to increase food consumption, but this cannot always be understood as healthier nutritional practices. The increase in sugar and canned food consumption in several regions of the country, for example, reveals the importance of combining educational actions and the promotion of health for the target population in basic care services and school cafeterias.

To favor intersectorial engagements and dialogue about program goals and results the Decentralized Administration Index (IGD) was created by the federal government in 2006³⁷. Even though such incentive had a late implementation, it certainly indicated the government's intention to correct program weaknesses. As Mazmanian and Sabatier⁴ point out, clear and consistent legal benchmarks can deal with conflicts about goals and objectives and guarantee a greater effectiveness in the implementation of public actions. In this aspect, the IGD, while transferring resources to different municipal offices based on the evaluation of the quality of the registration, the follow up on school attendance frequency and the health agenda can improve the quality of local intersectorial administration and the effectiveness of the program. Even if, according to the MDS³⁸, the transference of resources is centralized by municipal Social Assistance offices in almost 80% of the municipalities covered by the program, the perspective is involving Health and Education areas and favoring the shared decision on priorities and demands.

As a conditioned transference policy that intends to articulate a set of social policies, especially emphasizing health and education policies, Bolsa Família reasserts a horizontal conception of public intervention to fight poverty. Such guidance means the social players involved in the implementation of actions must be capable of creating goals cooperatively, integrating diagnoses, procedures and results. However, as local implementation of the program loses capacity to guarantee a better hierarchical integration and establishing strong connections with the set of social policies of basic sanitation, adult literacy, income and job generation, professional capacitation among others, there can be a greater verticalization tendency of the program. Moreover, the frailty of the participative arenas and the levels of social control surrounding Bolsa Família tends to raise difficulties in the bigger learning with the civic engagement³⁹ dynamics present in the recent course of social policies in the country. The greater participation of beneficiaries in the decisions related to the program implementation may have a meaningful impact on the efficiency and legitimacy of the actions. As Burris et al.⁴⁰ point out "even the imposition of 'good solutions' in a top down manner without real decision-making participation by those most affected, is paternalistic and illegitimate from a democratic perspective".

As a study carried out by the World Bank in 2005⁴¹ showed, among 245 municipalities researched, 26% did not have social control agencies related to the program. At the same time, among the counsels in action, the low attendance at meetings, the lack of information in the local implementation process and the weak participation of the civil society reveal the limits of these evaluation and public follow up spaces.

Challenges of the implementation and evaluation of intersectorial strategies

The main difficulties to implement intersectorial actions and activities take place when most of the sectorial representatives bring to the different negotiation arenas a previously defined agenda. On one hand, the negotiation about issues and priorities tends to be complex and difficult, which reflects in the agreements and implementation of intersectorial actions. On the other hand, intersectorial relations in the level of macro-negotiations and of proposal formulation are usually a success since no sector is unfavorable to initiatives in social development. However, such initiatives are not sustainable, predominating low levels of adhesion to dialogue and agenda agreement as well as political dispute and electoral interests. The maintenance of partnerships becomes a huge challenge to create and integrated agenda, both for short term specific issues and for structural issues.

At the same time, the limits between interventions conceived top down or bottom up gave to be, in come cases redone or have to become more flexible. As Silva⁴² points out, the centralization and decentralization of decisions can emerge as related and combined alternatives to reach greater efficiency in the interventions. In the case of Bolsa Família Program, the broadening of governance strategies with a greater participation of non governmental agents, private initiative and different social groups tends to potentialize efforts and resources aimed at job generation and local incomes, guaranteeing the "exits" in the program.

In the case of DLIS - Manguinhos, the initiative operates from the start based on the belief that policies and actions in poverty and inequality contexts depend on mobilization of all social sectors and broad participation of civil society. Participation and involvement of the population and formation of networks that acquire a great importance not only for the efficiency and success of the intervention, but also to generate broader and more sustainable changes in their social, environmental and political contexts.

Final considerations

Despite the advancements in creating a broader approach on health and wellbeing, the daily challenge is still to deal with structural and complex problems and most of the times they involve different government levels, institutions and public agencies besides several abilities, resources and levels of information. In this scenario, implementation and evaluation of policies and social programs gain greater complexity and cannot be faced as something instrumental or as "applicable sets in any context"⁴³.

Therefore, the design of new evaluation proposals must start from understanding the social programs as reflexive systems considering its complexity vis-à-vis the context it is inserted and its respective broad and intersectorial strategies to face social inequalities. This approach tends to put in doubt the objectives of the interventions, its dependence and close connection to political decisions that influence and redirect strategies vis-à-vis the multiple perspectives and the diversity of effects and results⁴⁴. The understanding on the nature and meaning of programs require an examination of the players' dynamics, interests and activities that shape and render meaning to social programs. This is necessary because the evaluation of actions is in itself a complex intervention since analyzing conflicts and controversies they have the effect of remodeling or redefining the course of programs and strategies.

As practices and political actions which operate based on new ideas, resources, activities and mechanisms, variable throughout time and that need to be questioned when facing the diversity of social spaces in which they are inserted. Frequently, effects and results have little or nothing to do with the officially present objectives and agreed in the beginning of the program's conception. New concentration arenas and meanings are created and recreated based on interactions among players acting directly or indirectly in the interventions.

The intersectorial programs must be understood as organizational systems that mobilize human and non human resources and that make services available through technical-scientific and popular knowledge - being permanently at issue the effectiveness of the activities vis-à-vis their purposes. Organizations have hierarchical positions, roles and rules are their raw material and for this reason, different and conflicting interests. Intersectorial actions involve players and strategies for the maximization of gains and minimization of losses in a trade off process. It is important, therefore, to overcome the controversy on the outcome or summative evaluation character⁴⁵. An original view on goals, reaches and results is crucial, starting from the study of connections among social spaces where the actions are implemented and, also, the aspects that favor or hinder the changes. Besides the articulation and dialogue among qualitative and quantitative approaches, what is at stake is the possibility to understand implementation processes as results in themselves, the objectives as the expression of conflictive relations in real contexts of interactions where changes tend to be negotiated and the period of time of the intervention is more flexible and less determined.

References

• 1. World Bank. Social Protection Strategy Paper Washington, D.C.: World Bank; 1999.
• 2. Buss PM, Pellegrini Filho A. A saúde e seus determinantes sociais. Physis: Revista de Saúde Coletiva 2007; 17(1):78-93.
• 3. Shadish WR, Cook TD, Leviton LC. Foundations of program evaluation: theories of practice Thousands Oaks, California: Sage; 1991.
• 4. Mazmanian DA, Sabatier P. Implementation and Public Policy, with a New Postscript. New York: University Press of America; 1989.
• 5. Potvin L, Haddad S, Frohlich KL. Beyond Process & Outcome Evaluation: a Comprehensive Approach for Evaluating Health Promotion Programmes. In: Rootman I, Goodstadt M, Hyndman B, McQueen DV, Potvin L, Springett J, Ziglio E, editors. Evaluation in Health Promotion: Principles and Perspective Copenhagen: WHO Regional Publications; 2001.
• 6. Barnes M, Matka E, Sullivan H. Evidence, Understanding & Complexity - Evaluation in non-linear Systems. Evaluation 2003;9(3):265-284.
• 7. Bodstein R, Zancan L.Monitoramento e Avaliação do Programa de Desenvolvimento Local Integrado e Sustentável - The Integrated Local Development Project [relatório de pesquisa]. Rio de Janeiro: Abrasco/Cida-Cpha/Finep/Fiocruz; 2003.
• 8. Connell JP, Kubish AC. Applying a Theory of Change Approach to the Evaluation of Comprehensive Community Initiatives: Progress, Prospect & Problems. In: Connell JP, Kubisch AC, Schorr LB, Weiss CH, editors. New Approaches to Community Initiative: Theory, Measurements & Analysis Washington, D.C.: Aspen Institute; 1998.
• 9. Bourdieu P. Razões práticas: sobre a teoria da ação 7Ş ed. São Paulo: Papirus; 2005.
• 10. Bodstein R. The complexity of the discussion on effectiveness and evidence in health promotion practices. Promotion & Education 2007; 1:16-20.
• 11. Burris S, Hancock T, Lin V, Herzog A. Emerging Strategies for Healthy Urban Governance. Journal of Urban Health 2007; 84(1):154-163.
• 12. Weiss C.Evaluation. New Jersey: Prentice Hall; 1998.
• 13. Barnes M, Matka E, Sullivan H. Evidence, Understanding & Complexity - Evaluation in non-linear Systems.Evaluation 2003;9(3):265-284.
• 14. Putnan R.Comunidade e democracia - a experiência da Itália moderna Rio de Janeiro: FGV Editora; 1996.
• 15. Coleman JS. Foundation of Social Theory Cambridge: Harvard University Press; 1990.
• 16. Peres F, Bodstein R, Marcondes W, Ramos C. Lazer, Esporte e Cultura na Agenda Local: a experiência de Promoção da Saúde em Manguinhos. Cien Saude Colet 2005; 10(3):757-769.
• 17. Jackson SF, Cleverly S, Poland B, Burman D, Edwards R, Robertson A. Working with Toronto Neighborhoods toward Developing Indicators of Community Capacity. Health Promotion International 2003; 18(4):339-350.
• 18. Buss PM, Ramos CL. Desenvolvimento Local e Agenda 21. Desafios da Cidadania. Cadernos da Oficina Social 2000; 3:13-65.
• 19. Krutman HM. Fatores críticos no êxito da gestão de projetos de desenvolvimento local integrado e sustentável [tese]. Rio de Janeiro (RJ): COPPE, UFRJ; 2004.
• 20. Carvalho AI, Bodstein R, Hartz Z, Matida AH. Concepções e abordagens na avaliação em promoção da saúde. Cien Saude Colet 2004; 9(3):521-530.
• 21. Ferreira JR, Buss PM. O que o desenvolvimento local tem a ver com a promoção da saúde? In: Zancan L, Bodstein R, Marcondes WB, organizadores. Promoção da saúde como caminho para o desenvolvimento local: a experiência em Manguinhos-RJ. Rio de Janeiro: Abrasco/Fiocruz; 2002. p. 15-37.
• 22. Rütten A. The implementation of health promotion: a new structural perspective. Source 1995; 41(12):1627-1637.
• 23. Bodstein R, Zancan L, Estrada D. Guia de equipamentos e iniciativas sociais em Manguinhos Rio de Janeiro: Fiocruz; 2001.
• 24. Pawson R. Evidence-based policy: the promise of realist synthesis London: Sage; 2002.
• 25. Giddens A. A constituição da sociedade São Paulo: Martins Fontes; 2003.
• 26. Draibe S. Bolsa Escola Y Bolsa Família. Cadernos NEPP 2006; 76:1-14.
• 27. Chen H. Theory-Driven Evaluations. London: Sage; 1990.
• 28. Fonseca A, D´ávila ALV. "Tensões e avanços na descentralização das Políticas Sociais" o caso do Bolsa Família" In: Fleury S, organizador. Democracia, descentralização e desenvolvimento: Brasil e Espanha, Rio de Janeiro: FGV Editora; 2008.
• ²⁹
  IBASE. Repercussões do programa Bolsa Família na segurança alimentar e nutricional das famílias beneficiadas Rio de Janeiro: IBASE; 2008.
• 30. Magalhães R, Burlandy L, Senna M, Schotz V, Scalercio G.O Programa Bolsa Família no estado do RJ: as experiências de São Francisco de Itabapoana e Duque de Caxias. Cien Saude Colet 2007; 12(6):1513-1524.
• 31. Rocha S. A pobreza no Brasil, afinal do que se trata? Rio de Janeiro: FGV Editora; 2003.
• 32. Carvalho M. Direitos universais- gastos focalizados: a focalização como instrumento para universalizar direitos sociais [tese]. Rio de Janeiro (RJ): IUPERJ; 2006.
• 33. Schwartzman S. Programas sociais voltados à educação no Brasil. Sinais Sociais 2006; 1(1):114-144.
• 34. Rocha S. Transferências de renda federais: focalização e impactos sobre pobreza e desigualdade. Rev. Econ Contemporânea 2008;12(1):67-96.
• 35. Lindert K, Skoufias E, Shapiro J. Redistributing Income to the Poor and the Rich: Public Transfers in Latin America and the Caribbean [texto para discussão n. 605]. Washington, D.C.: World Bank; 2006.
• 36. Estrella J, Ribeiro LM. Qualidade da gestão das condicionalidades do programa Bolsa Família: uma discussão sobre o Índice de Gestão Descentralizada. Rev Adm. Pública FGV 2008; 42(3):625-641.
• ³⁷
  Brasil. Ministério do Desenvolvimento Social e Combate à Fome. Portaria nº 148 de 27 de abril de 2006. Estabelece normas, critérios e procedimentos para o apoio à gestão do Programa Bolsa Família e do Cadastro Único de Programas Sociais do Governo Federal no âmbito dos municípios, e cria o Índice de Gestão Descentralizada do Programa. Diário Oficial da União 2006; 28 abr.
• ³⁸
  Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Alimentação e nutrição para as famílias do Programa Bolsa Família: manual para os agentes comunitários de saúde [documento da Internet] [acessado 2008 dez 14] [cerca de 52 p.] Disponível em: http://nutricao.saude. gov.br/documentos/manual_acs.pdf
• 39. Putnan R. Comunidade e democracia- a experiência da Itália moderna Rio de Janeiro: FGV Editora; 1996.
• 40. Burris S, Hancock T, Lin V, Herzog A. Emerging Strategies for Healthy Urban Governance. Journal of Urban Health 2007; 84(1):154-163.
• 41. Lindert K, Linder A, Hobbs J, Brière B. The Nuts and Bolts of Brazil's Bolsa Família Program: Implementing Conditional Cash Transfers in a Descentralized Context [texto para discussão n. 6209]. Washington, D.C.: Social Protection/World Bank; 2007.
• 42. Silva PLB. Modelo de avaliação de programas sociais prioritários [relatório final]. Campinas: NEPP; 1999.
• 43. Potvin L, Richard L. Evaluating Community Health Promotion Programmes. In: Rootman I, Goodstadt M, Hyndman B, McQueen DV, Potvin L, Springett J, Ziglio E, editors. Evaluation in Health Promotion: Principles and Perspective Washington, D.C.: WHO Regional Publications; 2001. European Series 92.
• 44. Weiss C. Nothing as Practical as Good Theory: exploring theory-based evaluation for comprehensive community initiatives for children and family. In: Connell JP, Kubisch AC, Schorr LB, Weiss CH, editors. New Approaches to Evaluating Community Initiatives Washington, D.C.: Aspen Institute; 1995. p. 65-93.
• 45. Patton M. Development Evaluation.Evaluation Practice 1994; 15(3):311-319.

Publication Dates

• Publication in this collection
  15 June 2009
• Date of issue
  June 2009

History

• Accepted
  08 Feb 2009
• Reviewed
  09 Jan 2009
• Received
  31 Oct 2008