The institutionalization of monitoring and evaluation practices: challenges and prospects in the view of the Brazilian National Health System managers

Abstracts

The aim of this study is to review the perception of state leaders on the practices and process for monitoring and evaluation (M & E) within the BNHS management. This is a qualitative study with a focus group, comprising managers from the states of Bahia, Mato Grosso, Rio de Janeiro, Santa Catarina, Sergipe and Tocantins. The guide was developed to analyze how this theme is placed on the agenda of managers and its concreteness in everyday management. Findings: the weakness of institutionalization of M & E as an disjointed process between the three entities; overlapping between different management functions; low-skilled teams; insufficient integration of information systems, and vicious cycles generated from fragmented practices, hindering the institutionalization of processes.

Monitoring; Evaluation of Health; Decision; Institutionalization

O objetivo deste estudo é verificar a percepção dos dirigentes estaduais quanto às práticas e processos de monitoramento e avaliação (M & A) no âmbito da gestão do SUS. Trata-se de um estudo qualitativo, realizado com um grupo focal de gestores estaduais da Bahia, Mato Grosso, Rio de Janeiro, Santa Catarina, Sergipe e Tocantins. O roteiro foi desenvolvido a partir do posicionamento da temática na agenda dos gestores e sua concretude no cotidiano da gestão. Verificaram-se: a fragilidade da institucionalização do M & A, enquanto um processo inarticulado entre os três entes; sombreamentos entre diversas funções de gestão; baixa qualificação das equipes; integração insuficiente dos sistemas de informação, e ciclos viciosos gerados a partir de práticas fragmentadas, dificultando a institucionalização dos processos.

Monitoramento; Avaliação; Decisão; Institucionalização

El objetivo de este estudio fue el de comprobar la percepción de los líderes del estado en las prácticas y proceso de seguimiento y evaluación (M & A) dentro de la gestión del NHS. Se trata de un estudio cualitativo con un grupo de discusión que tuvo lugar junto con los administradores de los estados de Bahía, Mato Grosso, Rio de Janeiro, Santa Catarina, Sergipe y Tocantins. El guión fue desarrollado a partir de la colocación del tema en la agenda de los directivos y su concreción en la gestión cotidiana. Se hizo referencia a la debilidad de la institucionalización de M & A como un proceso desarticulado entre las tres entidades, con matices entre las diferentes funciones de gestión, equipos de baja cualificación, insuficiente integración de los sistemas de información y se observaron los ciclos viciosos generados a partir de prácticas fragmentadas, cosa que dificulta la institucionalización de estos procesos.

Seguimiento; Evaluación; Decisiones; Institucionalización

Introduction

According Felisberto¹ the promotion of "evaluative culture" requires qualification of the technical capacity at various levels of the health system, enabling effective combination of monitoring and evaluation activities as subsidiaries or intrinsic to the planning and management, supporting the formulation of policies and decision-making and training of the subjects involved.

According Brito², the new responsibilities with the health system management require the incorporation of evaluation as a component in the planning process; as an activity capable of supporting decision-making, and an aid in initiatives aimed at changing the care model. It is adoption as a systematic activity within the municipal, state and federal administrations, can be considered one of the change indicators of the care model.

Barros et al.³ stated that evaluate became an act of greater magnitude and political implications and, therefore, we have tried to advertise the idea that the evaluative act comprises a set of ethical and political practices that sustain intentions and carry out actions effect to measure, describe and judge the direction, the merit and relevance of human intervention, in order to produce learning for the subjects involved in it in a particular way; and to society in general.

Tamaki et al.⁴ points out that the concept of monitoring commonly used, refers to the routine monitoring of relevant information. Monitoring is proposed to check for changes, but not their reasons, and includes the definition of temporality for collection and systematization of data. It is a systematic and continuous process of health indicators for monitoring, and implementation of policies, programs and services in order to obtain information in a timely manner to support decision making, identification, solution of routing and reduction problems as well as the correction direction. According to the author, the assessment for managing the dominant goal is its establishment as an element able to effectively participate in decision making processes, that is, produce answers to questions related to the assessed object was more prevalent on the scientific knowledge produced.

The purpose of evaluation should not simply distinguish successful or unsuccessful interventions. According to Rua⁵, appropriating the assessment as a process to support continuous learning for better decisions and ripening management "is much more important.

Furtado⁶, citing Figueiredo and Tanaka⁷ says that, in Brazil, the evaluation arouses growing interest from the 1990s, when increasing the number of initiatives related to social policies increases and in the context of the Constitution of 1998, the civil rights of citizens Brazilians are magnified. The Health System, this bulge is exposed to economic and political importance of health services and programs.

Paim⁸ understand that the process of institutionalization of the management of the National Health System can be characterized as a pendulum movement of decentralization/centralization, governed by the effort to implement the federal pact incorporated in the Constitution of 1988. This process, mainly triggered from 1993, with the movement around the "daring to enforce the law", has been punctuated over the past fifteen years, the design and implementation of policies and strategies that focus on the mission of managing institutions in each sphere of government, establishing the configuration of intergovernmental relations.

It also highlights the Operating Standards (NOB 01/93⁹, NOB 01/96¹⁰) as inducers of redefining roles and responsibilities of the three levels of government (federal, state and municipal) as regards the management, organization and delivery health services, through the transfer of resources (financial, basically, but also physical, human and material) of the federal and state levels, to municipalities. The Operational Norm of Health Care (2001/2002)¹⁰ strengthened the movement of (re) centralization with emphasis on strengthening the role of the State Health Departments, with the central strategy of the instrument design and implementation that intended to encourage the regionalization of assistance by organizing micro-regional health systems.

The author asserts that this process was temporarily interrupted by the change of government in 2003, settling a deliberation about the excessively option "rules" adopted during the 90s, culminating with the approval of the Pacts of Health in 2006¹¹, new policy instrument that aims to establish an ongoing negotiation process between managers, to ensure the implementation of priority policies and actions. Thus, the Ministry of Health tries to replace the strategic adopted earlier, namely, to induce decision-making at the state and municipal levels, from financial incentives, for another centered on the political commitment among managers, to be built in the space of Commissions Intermanagement Tripartite at national level; and Commissions Intermanagement Bipartite in each state, by signing the "Terms of Commitment" politically agreed.

Paim⁸also emphasizes an obvious "amateurism status" characterized by a shortage of qualified staff to carry out multiple and complex driving tasks, planning, scheduling, auditing, control and evaluation, regulation and management of resources and services; and the persistence of political patronage in the indication of the occupants of positions and management functions at all levels of the system.

Carvalho et al.¹² stated that the recent construction of the Health Pact and its improvement with the promulgation of Decree 7508 of 28 June 2011¹³, which regulates aspects of Law 8080/90, is based on respect for the constitutional principles of the SUS, with emphasis on the needs of the population, involving the simultaneous pursuit of the definition of articulated and integrated priorities; seeking to improve access to health actions and services; the strengthening of regional planning with the consequent definition of care networks in health regions; the improvement of governance mechanisms, and the qualification of the tripartite pact processes.

Felisberto¹⁴ highlights that, in Brazil, the health evaluation is presented also through new processes, some corporate practices, having almost always a more prescriptive and bureaucratic character. Not part of the institutional culture, appearing unsystematic way, and not always contributing to the decision-making process, requiring, therefore, technical order and political investment by the public administration sector.

The author points out that even with advances can still be considered initiatives, the actions developed by the Ministry of Health that identifies some of the limits of these initiatives: 1) the absence or lack of explicit policy guidelines and strategic definition to guide the evaluation; 2) fragmentation and resulting diversity of guidelines governing the evaluation process, preventing would assist the coordinated action; 3) emphasis on variables related to process; and 4) reviews of specific results and/or spatially restricted¹⁵.

In this line, Filho¹⁶ signalize that recent document prepared by the Department of Regulatory Assessment and the Ministry of Health Control, highlights the many difficulties in the development of assessment activities in the SUS, citing limitations related to the management culture that does not press for the use of planning, databases (even if available), the construction of indicators and the setting, monitoring and benchmarking goals. Also draws attention to the methodological difficulties of qualitative assessment, and even absence of information systems in more integrated health.

Even with these limitations, fit highlight recent initiatives such as the strong investment in the construction of the Monitoring and Quality of Care Evaluation of Primary (PMAQ); the initiatives of Monitoraids and PlanejaSUS; the creation of the Department of Monitoring and SUS management evaluation within the Secretariat of Strategic and Participative Management, and the very process of implementing the Pact for Health, which brings consistency with the operational range of the system, respect for differences local and regional adding the previously existing agreements; strengthening the organization of health regions; establishing mechanisms for co-management, co-responsibilities and regional planning; strengthening the spaces and mechanisms of social control; qualifying the population's access to comprehensive health care; Redefining the regulatory instruments, programming and evaluation; valuing the macro technical cooperation function between managers and proposing a tripartite funding that encourages equity criteria in transfers from fund to fund.

Contrandriopolus¹⁷, in turn, argues that the evaluation of institutionalizing capacity as a tool to improve the health system is paradoxical, though justified by the assumption that the information produced will help to streamline decision-making processes. It is noted, however, that the different actors who occupy different positions, often fail to reach a consensus on the relevance of the results that the evaluation produces.

Denis¹⁸ to institutionalize means creating a new role, new professional roles, new decision rules, in what may be called: ruling coalition in government and public administrations in health.

So, for this process to be instituted effectively, are needed improvement of the pact mechanisms involving the monitoring and evaluation practices of SUS management, which should be deployed in all federal units, setting up accountability states and municipalities and the federal administration, with a view to strengthening public management capacity of health.

In this perspective, Tanaka¹⁹ reaffirms the conviction that it is essential to evaluate the clear identification of who has the ability to mobilize resources and, consequently, for whom the assessment is intended. Therefore, it is important and crucial to careful observation of everyday social actors in the management of health systems teams, so we can identify the concrete's ability to mobilize resources, the real subjects of the evaluation of the institutionalization process.

But it should be emphasized, according Miranda²⁰, that given the diversity of initiatives and institutional evaluation experiences and monitoring of government management in health, which phenomenon requires a more methodical and thorough investigation in order to identify general trends, systematize more relevant experiences, assess the compliance with legislative provisions, and especially evidence meanings related to government actors involved. This article aims to analyze the perception of state authorities on monitoring and evaluation practices, and to identify the challenges and perspectives regarding its institutionalization under the SUS.

Methods and Material

It is a qualitative study on the monitoring and evaluation processes within the health departments of the states, to which was made a focus group to collect and perception of state managers. The choice of method is based on Gatt²¹, defending a technique used by researchers to a set of selected people, for discussion and comments about a subject or object of research, from their personal experiences, it is a communicative process flexible.

The group was formed from the intentional choice, taking into account the regional distribution, the observation of different population sizes, the actions of leaders, and technical references of SES, construction schedules involving the Department of Monitoring and Evaluation of Management SUS/DEMAGS/SGEP. The participants, managers linked to the sector responsible for the development of monitoring and evaluation processes, were nominated by their SES.

The first eight were selected State Departments, however, only six participated: Bahia, Mato Grosso, Rio de Janeiro, Santa Catarina, Sergipe and Tocantins. The Secretariats of Pará and São Paulo were dropouts.

For the development of the work used a list of questions that allowed the themes inherent to approach (the): practices and processes of M&A and its centrality in the agenda of managers; instruments used in the construction of the practices and processes of M&A management; operational aspects of the practices of M&A management and the challenges and perspectives regarding its institutionalization.

The contents produced by the focus group were transcribed in full and subsequently analyzed using the software Analyse lexicale par Contexte d'un Ensemble of Segments of Text (ALCESTE), in its version 4.10, thus allowing a lexical analysis, considering the words inserted as units in the occurrence contexts in order to know the structure and organization of the speech, providing access to lexical universes according Reinert²².

The program allows textually analyze the semantic productions of the subjects, using the relationship between the words to identify how often appear in the discourse and their associations in classes (calculation χ²), and replenish the shared discourse between subjects.

Shimizu²³, it points out that the program helps to identify the common ground of ideas shared by a social group about a particular social object.

Results

From the analysis conducted by ALCESTE we obtained a dendrogram divided into three themes and subdivided into five classes as follows:

Axis I: From the management of M&A in the management, consisting of the Class A: The M&A practices in the SUS management (49%) and Class Two: Situations critical for M&A of management (16%);

Axis II: The instruments and methods, comprising the Class Four: instruments identified in the construction of the practices and processes of M&A management (9%) and Class Five: inherent operational aspects of the practices of M&A management (8%);

Axis III: Of Responsibilities inter-federative, composed of Class Three: Practical and M&A processes in the federative relations (18%).

Below, Figure 1, with the dendrogram, details axle and class the percentage regarding the repetition of words and phrases that characterize the way managers understand and/or address the content inherent in the processes of institutionalization of practices and monitoring and evaluation processes SUS management.

Figure 1
Dendogram the perception of managers about monitoring and evaluation practices in SUS

The axis I analysis, with classes one and two, it appears that the monitoring component appears more strongly highlighted as practical in the management. Another important aspect concerns the finding of shading areas between the functions of planning, monitoring, evaluation and audit are in the form of the organization of the sectors in the secretariats and/or organization of management practices. It is evident the existence of a sector or sectors that develop these functions, but does not guarantee a good practice or efficiency in the management of the shares. With respect to class two, the findings show that one of the major obstacles to the institutionalization of monitoring and evaluation practices process is linked to the lack of monitoring and evaluation, especially how to institutionalize these practices.

It is found that such a situation leads managers to work constantly in crisis situations. So also highlight the importance of working in the preparation of teams, the appropriate infrastructure for anticipating crisis situations and to monitor adequately the implementation of health policies.

Again managers highlight the fragmentation of the organizational process, which makes the assumption in the management. Also, address the problem of centralization of the thematic discussion processes, greatly interlacing the capillarity of the concepts and content and thus complicating the process of institutionalization.

The axis II, as stated above, consists of four five classes, dealing in tools and methods related to monitoring and evaluation practices, and operational processes involved in building these.

In class four managers stand instruments contributing to the monitoring and evaluation practices, such as GesPública, which is used to improve the performance of management teams and managerial practices in departments. Another relevant instrument was the PlanejaSUS for the strengthening of planning practices in SUS, and consequent aid to qualification management. It is worth noting the low integration in the way information systems were addressed, damaging data extraction and production information. Thus, they highlighted the institution's information and information technology policy as necessary to describe the monitoring and evaluation actions.

The class covers five operational processes inherent in monitoring and evaluation practices highlighting the use of planning tools such as the management report in its quarterly form, linked to the development and qualification of the annual health and the Pact for Health Indicators.

There is a clear need that managers have to make explicit the importance of the integration of instruments and the planning and management process, emphasizing the role of SES and performance of the Secretary of State Planning, as a monitoring body of this practice in the States.

Axis III, Class Three deals with the practical and M&A processes in the federative relations, where the experience of the Pact for Health and Public Action Agreement in the State of Sergipe are seen as concrete expressions of how managers have defined more clearly the health responsibilities among them, those that deal with monitoring and evaluation involving the three federal entities.

Identifies the lines a criticism of the Ministry of Health's position with respect to induction of policies to health care practices at the expense of primary care as the way you relate directly with municipalities, emptying the state's role, as well as how to monitor and evaluate the states and municipalities without undergoing the same evaluation process. Managers, to speak specifically of the dynamics of monitoring and evaluation in the municipalities, said unaware of its existence in a systemic way, highlighted the areas of surveillance and primary care through the SIAB (Warning Information System Basic) where there is still collecting data but not necessarily use for decision making.

With regard to the action of the State in relation to the municipalities, managers highlight the use of the AMQ, an inserted project in the context of institutionalization of evaluation of primary care at the municipal level, which offers tools for assessment, planning and management of the Health Strategy of the family. In the perception of managers, to focus on the evaluation of the work process, the project will force teams to work with the improvement of health indicators, because to achieve an indicator, it is necessary to develop a set of activities by changing patterns and processes. For these and other features, the managers pointed out this process as an Inter practice that allows you to monitor and evaluate.

Another approach carried out, concerns the discussion of macro-functions related to the Essential Public Health Functions (EPHF), which require, according to the managers, are also discussed for municipalities, strengthening the understanding of management practices as an integrated process and not as isolated actions where each plan and other budget, a monitor and assess other. Reinforcing the above, also emphasize that the manager does not plan to have no ability to measure their performance, and who has no ability to measure its performance will never be able to make a qualified assessment, as there will be no reference to it.

The need to find evidence to strengthen the understanding of the monitoring and evaluation practices in the management, especially when the Ministry of Health has them as an object of interest is a point which causes great interest in state teams.

Finally, comes again, the position of the group and personal disability and qualification and quantity, situation identified as an inherent fragility non implementation of labor regulation policy and continuing education. The qualification of the staff of the secretariats in order to follow the established responsibilities and monitor the agreed targets was an indication of the group of managers.

Discussion

The axis of the findings I through Class I and II, which have the management of M&A in the management, indicate lack of clarity between the functions of planning, monitoring, evaluation and audit in the management, which is characterized herein, for carrying out these functions concurrently by different sectors of the same office or in a disjointed manner, which, according to the managers, hinders the understanding of concepts and content, reinforcing the fragmentation of actions and hampering the coordination and harmonization of agendas around the monitoring and evaluation actions. Thus, it is pertinent to the critical position that the existence of a sector or sectors to develop these functions does not guarantee a good practice or efficiency in the management of the shares.

Along the same lines, managers claim that one of the major obstacles in the process is linked to the lack of knowledge about the content inherent to monitoring and evaluation, especially the institutionalization of practices, which led them to work in constant crisis situation.

These findings are consistent with the assertion that in Brazil, the health evaluation presents itself also through new processes, some corporate practices, having almost always a more prescriptive and bureaucratic character. Not part of the institutional culture, appearing unsystematic and systematic manner, and does not always contribute to the decision-making process and the training of professionals, requiring, therefore, technical order and investment policy by the government sectorial¹⁶,

Thus the highlighted aspects point to the need of promoting the "evaluation culture" that the approach carried out by managers, appears in a critical situation, as it requires the qualification of the technical capacity at various levels of the health system, and an effective enabler association of monitoring and evaluation actions, supporting the planning, management and processes decisions¹⁹.

Other points, with respect to the axis I, relate to low-skilled teams, inadequate infrastructure, and the centralization of decision-making, forming a vicious cycle that disrupts the capillarity of the concepts and content, weakening the process of institutionalization of M&A.

Clearly, managers are not able to integrate the monitoring and evaluation practices in an organizational system, influencing their behavior, necessarily relate analytical activities to the management of program interventions, institutionalizing not occur. This requires the development of actions with strategic actors organizations, making them aware of the improvement of evaluation practices in order to qualify the decision making⁶,¹⁰-¹⁵.

Axis II covers aspects of instruments and methods related to the practices of M&A and operational processes involved in building them.

The influence of GesPública as methodology for improving the performance of management teams and managerial practices in departments and PlanejaSUS is perceived, having been used for the strengthening of planning practices in SUS. However, it is worth remembering the low integration of information systems hindering the management of data and the production of information, a fact that highlighted the need for information and computer policy implementation in order to qualify the M&A activities.

The whole approach will find support in the document found recently prepared by the Department of Regulatory Evaluation and Control of the Ministry of Health, which stands as limitations to the development of rating actions in the SUS, related to culture management, which did not press for the use of planning; databases (even if available); the construction of indicators and setting; monitoring and assessment of methodological goals and difficulties of qualitative and even the absence of information systems in health, more interconnected⁷,⁹,¹⁸.

Also in regard to operational processes, emphasizes the use of management report linking it to the annual program and the Pact indicators for Health, making clear the need to integrate the tools and process of planning and management, with an emphasis on role of SES.

This axis has featured as the operational expression of monitoring and evaluation practices and their findings dialogue with the new responsibilities to be assumed by management, requiring the incorporation of evaluation as the planning process component, as an activity capable of supporting decision making and as an element to be considered as a modification indicator or as an initiative aimed at changing the model assistance¹,².

Finally, the axis III presents inter-federative agreements, expressed in the Pact for Health and Public Action Agreement in the State of Sergipe, where managers have defined more clearly the health responsibilities, especially those that deal with monitoring and evaluation. The processes involve necessarily the three federal entities, and are explicit weaknesses of the process in the Ministry of Health's position as inducer of policies health care practices in the form of intervention in relation to the municipalities and how to submit to evaluation process.

The incipient development of monitoring and evaluation practices in a systematic and integrated manner is notorious, being indispensable to think the management of a more articulate way.

Here is visible the lack and/or weakness of a professional management, continuing a labeled "amateurism" in the management, affecting all levels, both by the lack of qualified staff to carry out multiple and complex tasks related to driving, planning, scheduling, auditing, control and evaluation, regulation and management of resources and services, as the persistence of political patronage in the indication of the occupants of positions and management functions at all levels of the system⁹.

Final considerations

The evidence reinforces the need to enhance the pact mechanisms involving the monitoring and evaluation practices incorporated into the daily life of managers and professionals. These practices can not only be bound by the obligation of accountability or measurement results for satisfaction maker’s worker programs and/or sectorial policies.

It is evident that even a position in which the monitoring and evaluation practices are able to mobilize resources, is still fragile reading the daily lives of social actors in the management of health systems teams, making it difficult to build an inter-federative agenda and institutionalization of M&A practices, preventing the adoption of more inclusive practices in the management²³.

Thus, it is clear that it is essential the investment in training and in the provision of human resources; adoption of systemic activities capable of producing learning in the subjects involved and the strengthening of planning practices, regulation, monitoring and evaluation in an integrated manner. Another point is related to the exact definition of technical politics guidelines that guided the process is practical, as well as the definition of strategies and Inter responsibilities, enabling the improvement of management functions, the development of their practices and the consequent taking security to decision and quality³,⁴.

Referências

• ¹
  Felisberto E. Análise da implantação e da sustentabilidade da política nacional de monitoramento e avaliação da atenção básica em saúde no Brasil no período de 2005 a 2008 (dissertação). Recife (PE): Centro de Pesquisas Aggeu Magalhães (CPqAM/Fiocruz); 2008.
• ²
  Brito TA, Jesus CS. Avaliação de serviços de saúde: aspectos conceituais e metodológicos. Rev Digit:(Buenos Aires) [Internet]. 2009 [acesso 28 Jul 2015]; 14(139). Disponível em: http://www.efdeportes.com/
  » http://www.efdeportes.com/
• ³
  Barros NF, Castellanos MEP, Motta MC, Alegre SM, Tovey P, Broom A, et al. A produção de sínteses com evidências qualitativas: uma revisão de métodos e técnicas. In: Onocko RC, Furtado JF, organizadores. Desafios da avaliação de programas e serviços em saúde. Campinas: Editora Unicamp; 2011. p. 121-34.
• ⁴
  Tamaki EM, Tanaka OY, Felisberto E, Alves CKA, Junior MD, Bezerra LCA, et al. Metodologia de construção de um painel de indicadores para o monitoramento e avaliação da gestão do SUS. Cienc Saude Colet. 2012; 17(4):821-8.
• ⁵
  Rua MG. Avaliação de Políticas, Programas e Projetos: notas introdutórias [Internet]. 2010 [acesso 10 Nov 2015]. Disponível em: https://jacksondetoni.files.wordpress.com/2014/05/texto-apoio-05-_-grac3a7as-rua.pdf
  » https://jacksondetoni.files.wordpress.com/2014/05/texto-apoio-05-_-grac3a7as-rua.pdf
• ⁶
  Furtado JP, Laperrièrre H. A avaliação da avaliação. In: Onocko RC, Furtado JF, organizadores. Desafios da avaliação de programas e serviços em saúde. Campinas: Editora Unicamp; 2011. p. 19-39.
• ⁷
  Tanaka OY: Caminhos alternativos para a institucionalização da avaliação em Saúde. Cienc Saude Colet. 2006; 11(3):564-76.
• ⁸
  Paim JS. Avaliação em saúde: uma prática em construção no Brasil. In: Hartz ZMA, Vieira da Silva LM, organizadores. Avaliação em saúde: dos modelos teóricos à prática na avaliação de Programas e Sistemas de Saúde. 2a ed. Salvador: EDUFBA; 2005. p. 9-10.
• ⁹
  Portaria MS/GM nº 545, de 20/05/1993. Estabelece normas e procedimentos reguladores do processo de descentralização da gestão das ações e serviços de saúde, através da Norma Operacional Básica - SUS 01/93. Brasília (DF): Ministério da Saúde; 1993.
• ¹⁰
  Portaria MS/GM nº 2203, de 5/11/1996. Redefine o modelo de gestão do Sistema Único de Saúde, constituindo, por conseguinte, instrumento imprescindível à viabilização da atenção integral à saúde da população e ao disciplinamento das relações entre as três esferas de gestão do Sistema - NOB 01/96. Brasília (DF): Ministério da Saúde; 1996.
• ¹¹
  Portaria nº 373, de 27/02/2002. Aprova a Norma Operacional da Assistência à Saúde – NOAS-SUS 01/2002: regionalização da assistência à saúde: aprofundando a descentralização com equidade no acesso. Brasília (DF): Ministério da Saúde; 2002.
• ¹²
  Ministério da Saúde (BR). Diretrizes operacionais dos Pactos pela Vida, em Defesa do SUS e de gestão-regulamento. Brasília (DF): MS; 2006. (Série Pactos pela Saúde; 2).
• ¹³
  Carvalho ALB, Souza MF, Shimizu HE, Senra IMVB, Oliveira KC. A gestão do SUS e as práticas de monitoramento e avaliação: possibilidades e desafios para a construção de uma agenda estratégica. Cienc Saude Colet. 2012; 17(4):901-11.
• ¹⁴
  Decreto nº 7508, de 28/06/2011. Regulamenta a Lei nº 8080/90. Brasília (DF): Ministério da Saúde, Secretaria de Gestão Estratégica e Participativa; 2011.
• ¹⁵
  Felisberto E. Institucionalização da avaliação. In: Samico I, Felisberto E, Figueiró AC, Frias PG, organizadores. Avaliação em Saúde: bases conceituais e operacionais. Rio de Janeiro: Medbook; 2010. p. 143-61.
• ¹⁶
  Santos Filho SB, organizador. Avaliação e humanização em Saúde: marcos conceituais e metodológicos. Ijui: Editora Unijui; 2009. Avaliação e humanização em Saúde: aproximações metodológicas; p. 27-54.
• ¹⁷
  Contrandriopoulos AP. Avaliando a institucionalização da avaliação. Cienc Saude Colet. 2006; 11(3):705-11.
• ¹⁸
  Denis JL. Institucionalização da avaliação na administração pública. Rev Bras Saude Mater Infant. 2010; 10 Supl 1:229-37.
• ¹⁹
  Figueiredo AMA, Tanaka OY. A avaliação no SUS como estratégia na reordenação da saúde. Cad Fundape. 1996; (19):98-105.
• ²⁰
  Miranda AS. Proposição de escopo para a avaliação da gestão no Sistema Único de Saúde. Relatório final de consultoria prestada ao Ministério da Saúde. Brasília (DF): Ministério da Saúde; 2005.
• ²¹
  Gatti BA. Grupo focal na pesquisa em Ciências Sociais e Humanas. Brasília (DF): Liber Livro Editora; 2005.
• ²²
  Reinert M. Alceste une méthodologie d’analyse des données textuelles et une application: Aurelia De Gerard De Nerval. Bull Methodol Sociol. 1990; (26):24-54.
• ²³
  Shimizu HE. Percepção dos gestores do Sistema Único de Saúde acerca dos desafios da formação das Redes de Atenção à Saúde no Brasil. Physis. 2013; 23(4):1101-22.

Publication Dates

• Publication in this collection
  16 Nov 2016
• Date of issue
  Jan-Mar 2017

History

• Received
  25 Nov 2015
• Accepted
  28 Mar 2016