The Implementation of the Monitoring and Evaluation System of the State Health Secretariat of the Brazilian Federal District (SHS/DF)

Abstract

Through the creation of the Unified Health System (SUS), the planning processes gain intrinsic importance for the creation of public health policies and to subsidize decision-making, implementation of actions and the achievement of results. Since planning tools are fundamental to guidethe management, caution is needed regarding the interface between them,aiming at achieving the integration of health services, of which results are better, more effective and cost-effective for the government. Likewise, continuous monitoring and evaluation (M&E) processes allow the measurement and tracking of strategic information, improving the quality of health information systems and health indicators, as well as the operational and organizational performance. In this article, we present the recent initiative to implement the M&E System in the State Health Secretariat of the Federal District (SHS/DF) and the processes involved in its qualification.

Keywords
Planning; Monitoring; Evaluation; Health; Situation room

Resumo

Por meio da criação do Sistema Único de Saúde (SUS), os processos de planejamento ganham importância intrínseca para a formulação de políticas públicas de saúde e para subsidiar a tomada de decisões, a implementação de ações e a obtenção de resultados. Como as ferramentas de planejamento são fundamentais para orientar a gestão, é necessário o cuidado nas articulações entre elas, a fim de alcançar a integração dos serviços de saúde, cujo resultado é melhor, mais efetivo e mais barato para o governo. Da mesma forma, processos contínuos de monitoramento e avaliação (M&A) permitem medir e rastrear informações estratégicas, melhorando a qualidade dos sistemas de informação em saúde e dos indicadores de saúde, bem como do desempenho operacional e organizacional. Neste artigo, apresentamos a recente iniciativa de implementação do sistema de M&A na Secretaria de Estado da Saúde do Distrito Federal (SES/DF) e os processos envolvidos em sua qualificação.

Palavra-chave
Planejamento; Monitoramento; Avaliação; Saúde; Sala de situação

Introduction

Under the influence of the developmentalist theory of the Economic Commission for Latin America and the Caribbean (ECLAC), aimed at economic growth and the rationalization of the substitutive policy to be operated by the Government, Health Planning based on a methodology for the programming of health resources combined with cost-benefit analyses appeared in Latin America in the 1960s, developed by the Center for Development (CENDES), an entity created in Venezuela in partnership with Universidade Central and supported by the Pan American Health Organization (PAHO)¹.

Even with the advances that were attained, the method was described as overly technocratic, economistic, with low capacity for regulation by the Government, lack of definition regarding sponsorship and dominated by private interests¹.

For Testa², the weakness of the public sector and the disregard for the political and macro-institutional aspects involved in the policy and planning creation processes have partially invalidated CENDES’ proposal. As an alternative, there appears the set of strategic planning (SP) aspects that proposes the adherence of other actors and the political feasibility of the plans.

According toTesta³, although there is an emphasis on political actors, the founding matrices and the varied political currents that encouraged the debate between currents of situational and strategic planning in Latin America have been based on the marxist conception of social class structures.

The rise of this conception of planning historically converges with the construction of the public health reform movement in Latin countries and the struggle for the redemocratization of countries that were under dictatorial regimes.

Based on the critique of planning as resulting from a single actor, the Government, a more pluralist view emerges, according to which the state and social intervention dynamics presuppose situations of power sharing, which raises the need of planning at times of conflict and cooperation between the actors. Conditions of exception (authoritarian regimes) exacerbated the conflict and the distrust relationship reinforced strategic contexts.

The first milestone in this evolution is represented by the document ‘Health Policy Formulation’ (1975) of the Pan American Center for Health Planning (CPPS). It perceives planning as a process that, although dominated by the Government, involves the mobilization of several actors, and introduces the need for political feasibility and strategy analysis. According to the two greatest advocates of this current, Testa and Matus, normative planning (NP) gives way to strategic planning (SP)²,⁴.

With democratization, the institutionalization and legitimation of the spaces for the participation of society are guaranteed at the interface between civil society, politics and administrative power. Matus⁵ emphasizes that in a context in which several projects of society/actors are in constant confrontation, in addition to the recognition of the conflict and its thematization, it is necessary to strengthen the other’s capacity to listen, interact and negotiate. Therefore, one considers that a pluralistic and communicative conception of planning has greater applicability.

In this context, according to Rivera & Artmann⁶, two major models can be identified: the first one based on the problem-solution approach, corresponding to the strategic-situational and total quality planning, which considers a planning model that goes from the present to the future, and the second, that deals with the SP based on scenarios, predicting future planning for the present, using Godet’s⁷ strategic perspective as a model.

Considering the aspects of politics, planning and health management in Brazil, Paim and Teixeira⁸mention the historical division proposed by Levcovitz et al.⁹: the first period (1974-1979) analyzed the health system and its economic, political and social determinants. The second (1980-1986) discusses the spread of the reform and socio-political association. The third (1987-1990) discusses the juridical-legal order of the principles and guidelines of the reform project. The fourth (1991-1998) seeks to define the role of each sphere of government and the creation of management tools.

The fifth period (1999-2000) deals with the regulation of funding and the organization of the management and care models of the regionalized service network, whereas the sixth and last period deals with the search for funding stability (constitutional amendment n. 29) and the development of primary care, with the expansion of the Family Health Program (FHP). During this same period, four other health planning / management currents were identified⁹.

From the 1990’s onward, the main role of the central planning and strategic modalities for planning by goals and targets under a monitoring and evaluation (M&E) system changed.

According to Merhy¹⁰, to evaluate is essentially to estimate the value of an intervention or any of its components. Evaluation is an essential tool to support management due to its capacity of improving the quality of decision-making. However, its use is still incipient in health service management¹¹,¹².

Habicht et al.¹³ emphasizes that monitoring is an action that allows the observation, measurement and continuous evaluation. It is essential for the routine monitoring of priority information, both for the process of program implementation and for assessing its operational and ultimate performance. Based on this assumption, institutionalizing the evaluation and monitoring within the structure of national, regional, municipal or local management becomes crucial, aiming to integrate it into an organizational system.

Considering this scenario, the planning, monitoring and evaluation process of the SHS/DF prioritizes the integration between the planning and management tools: the Pluriannual Plan (PPA), the Annual Health Programming (PAS), the District Health Plan (PDS), the Governor’s Management Agreement (AR), the Regional Management Agreement (AGR) and the Inter-federation Pact.

For that purpose, the SHS/DF has been implementing an M&E system, understood as a set of articulated, systematic and formalized activities of production, registration, monitoring and analysis of strategic information that, in addition to an integral view of the SP, allows qualifying the decision-making process.

In this article, we present the recent initiatives for the implementation of the SHS M&E system and the processes involved in the M&E qualification.

The decentralization of the Health System of the Federal District

In 2016, the DF population was estimated at 2,977,216 inhabitants¹⁴. However, considering the Integrated Development Region of the Surrounding Area of the Federal District (RIDE-DF), created by Complementary Law N. 94, of February 19, 1998 and regulated by Presidential Decree N. 7,469, dated of May 4, 2011, it has an estimated population of 1,293,768, totaling 4,208,598 inhabitants¹⁵.

Hartz¹⁶ emphasizes that, unlike the other states of the country, it is a Federation Unit (FU) with peculiar characteristics, having the political-administrative functions of state and municipality. As a State, it has the responsibility of organizing the planning, coordination, monitoring and evaluation activities of the entire health policy, while as a municipality, it has direct contact with the population, by providing basic, medium and high-complexity health services.

Aiming at improving the administration, the Federal District was divided into 31 Administrative Regions (RA), of which only 19 have defined areas. Garcia¹⁷ emphasizes that these elements act as a guide for the definition of public health policies where actions must be considered not only for the Brazilian population as a whole, but also for the Surrounding Areas, which exert strong pressure on several sectoral areas: health, education, and housing.

The SHS is in charge of the SUS management within the Federal District, in addition to having a central role in the discussions on health actions and services within RIDE. Up to 2015, the SHS/DF maintained a vertical management model focused on Central Administration (ADMC). Since the publication of Decree N.36,918 of 11/26/2015,and subsequent changes in its structure, the Secretariat started to comprise: the Central Administration (ADMC), Assistance Reference Units (URA), Reference Units (URD) and Superintendences of the seven Health Regions (Central, Midsouth, West, South, Southwest, North and East) which are the smallest management units of the territory. In 2017, the Internal Regulation of the Superintendences of Health Regions was published, which defined the limits of the responsibility of each organizational unit¹⁸.

In order to coordinate the management model decentralization efforts, the Regional Management Program (PRS) was developed, where each region is represented by a superintendence body that achieves a Regional Management Agreement (AGR), which is a formal agreement tool between the ADMC and each of the seven superintendences¹⁹.

During this process, ADMC, in addition to ensuring the regions the necessary conditions for the development of activities, must develop institutional incentive strategies aiming to implement a results-based management model¹⁸.

To guide the decision-making processes of the regional managers, and to improve the monitoring and evaluation process of the AGRs, the M&E methodology to be used was defined, by agreeing on a matrix of goals and indicators, whereas the points of health care, the list of the actions and services of each point of care, the authorized services, the annual costs, the total cost per unit were identified, and also, the matrix of responsibilities was created.

Another initiative, focused on strategic alignment, management quality and performance improvement, was the creation of the Office of Processes, which aims at institutionalizing routines that promote the organization, monitoring, evaluation and achievement of results in the different organizational processes developed by its employees to meet their institutional mission. Their competences are to develop, recommend and multiply methodologies and best practices of process management; to know and support the mapping of the organizational processes developed by the institution and to make available the information about them, promoting their standardization and description in manuals; to continuously foster the performance M&E of the organizational processes through the creation of indicators; and to implement process improvements aiming at increasing efficiency, efficacy and effectiveness.

This initiative is aligned with the methodology adopted by the Governorship, State Secretariat of the Chief of Staff, State Secretariat of Planning, Budget and Management and of Institutional and Social Relations. Albeit recent, all processes in the technical areas have already been mapped and are undergoing the validation and appreciation process¹⁹.

Alignment of SHS/DF planning tools

Planning in the SUS is a management function that, in addition to being a legal requirement, is one of the relevant mechanisms to warrant unity and constitutional principles. It expresses both the responsibilities of the managers of each sphere of government in relation to the health of the territory population and the integration of the systemic organization. The task of planning requires technical knowledge that is expressed in tools and devices developed in work processes, aiming to guide and integrate health actions and services²⁰.

Gottems et al.²¹ emphasize that management planning tools should be interconnected and in line with the budget planning tools. Blumm et al.²² point out that in 2012, the SHS/DF’s Strategic Situational Planning (PES) was not compatible with the PPA (2012-2015) and the PDS (2012-2015) regarding its strategies and goals, generating an incongruity between the goals and their indicators, inadequate use of resources and distorted analysis of results.

Considering this picture, in the period of 2015 to 2016, the area of planning of the Health Secretariat promoted workshops with the participation of managers and professionals from the different areas of assistance, to create the SP, PDS, PPA and PES, using a participatory strategy to build the alignment of tools for health planning and programming (Figures 1, 2 and 3), subsidizing the creation of the PAS and the Annual Budgetary Law Proposal (PLOA)²³.

Figure 1
Process of creation of the Planning Tools.

Figure 2
Alignment of the Planning Tools.

Figure 3
SHS/DF monitoring and evaluation cycle.

Starting in May 2017, these initiatives were reinforced through the implementation of processes of results analysis and evaluation through specific and periodic meetings between the top management managers, called Results Evaluation Meetings (RARs)²⁴. These meetings contribute to the construction of references and evidence that subsidize the decision-making regarding the maintenance, improvement or reprogramming of the planned actions, reinforcing the importance of the PDCA (Plan-Do-Check-Act; or sometimes, Plan-Do-Check-Adjust) process in the institution’s strategic agenda.

This follow-up model was designed to instrumentalize the technical areas with the perspective of systematically organizing the monitoring and follow-up of results agreed upon in the SP.

The M&E tools, information management and dissemination of the SHS/DF

Monitoring is crucial for the routine follow-up of a schedule priority information, both for operational performance and its ultimate performance. The evaluation, in turn, provides the assessment of value appraisal that assists in decision-making, being able to subsidize changes in the creation and/ or implementation of health programs, projects or policies²⁵.

According to Felisberto²⁶, the dissemination of an evaluative culture and its inclusion into the routines qualify the actions of decision-makers (health professionals, planners, managers and representatives of civil society organizations), yielding changes in these processes.

Carvalho et al.²⁷comments that despite the availability of several information systems aimed at the operation of health care establishments, service management, investigation or control of several diseases, its use has been advocated for the planning of interventions on the sanitary reality.

However, the use of strategic and qualified health information remains a challenge for managers, since often the numerous information systems being used admit duplicate data, exhibit access difficulties and do not count with managerial analysis reports²⁸.

Information technology (IT) tools have been driving forward monitoring strategies, expanding the possibilities for health situation assessment. The use of these tools has allowed the integration of different information systems and contributes to the health monitoring process. In this scenario, to carry out the SP monitoring, the SHS/DF adopted a set of measures that allow the follow-up of the implementation of its results and assists in decision-making.

In 2016, a management tool of the Planning Tools called the SHS/DF Planning System (SESPLAN) was developed. This system allows the recording of data from the different areas of the secretariat, sharing the information of the planning cycle, broadening the understanding of the macro processes, from the ultimate to the sustaining ones, and the analysis of the results. It consists of six modules:

1. Goals and Indicators: presents the agreements, monitoring and results with detailed analysis;

2. Annual Health Program (PAS): allows the monitoring of the implementation of actions;

3. Budget Operation (LOA): presents the schedule and detail of expenses, considering the financial availability;

4. Government Monitoring System (SAG Stage): follows the production and budget resources;

5. Production / Activities: allows the registration of information complementary to the agreed Strategic Objective (SO);

6. Reports: provides the contextualization of information and results of agreements and subsidizes the accountability of the SHS/DF.

SESPLAN has shown to be efficient and has filled a gap in the integration of M&E and planning tools. However, the increasing volume of data, accesses and number of users using the Secretariat intranet network has sometimes caused the loss of data records and information reliability. One of the limitations in the system use is due to the fact that it was developed using an Excel platform.

The informatization of SESPLAN started in 2018, using free technology and a web platform, with a relational database. This warrants data integrity and user access through any web browser. With the system implementation, only authorized users have access to the data, allowing the log of the performed activities, ensuring the integrity and auditing of the recorded information. The results of the indicators, previously recorded manually in the system, are, for the most part, automatically calculated by the tool, making it possible the analysis throughout the monitoring cycle and in a timely manner for the evaluation. The results are displayed in an interactive monitoring panel, which allows one to detail the recorded information.

Another performed monitoring activity is related to the accreditation and enabling of health services, through data registration using Excel spreadsheets, carried out in a fragmented way by the different technical areas involved, thus making it difficult to share the information and monitoring and integral evaluation process by managers.

To overcome this deficiency, the Accreditation and Qualification Management System (SIGECH) was built, a computerized system in a web platform that allows simultaneous access of several users, the recording of information about current decrees, qualified services and those with qualifying potential, action plans being performed for resolution of “nonconformities” indicated by sanitary surveillance, in addition to the monitoring of the deadlines established for the accreditation and qualification of health services. Through reports and graphical panels, all the steps/stages of the process are visualized, which allows the monitoring and detailed evaluation aiming to meet the defined deadlines.

Indicator management

According to Santos²⁹, the challenge of performing more effective health actions to meet the users’ needs shows the necessity to build and use M&E indicators. Indicators can be understood as quantitative and / or qualitative measures, with intentionalities, based on records of facts or reality phenomena. To know the reality will always be a challenge, resulting in representations that may be more or less distant from it.

The construction and use of indicators – able to provide more precise and essential information on the reality aspects, supporting decision-making aimed at the rights and well-being of all – is still not a widely adopted practice in the scenario of governments, managers, non-governmental entities that interact with SUS and even of social control in all of its dimensions.

Santos²⁹ emphasizes that health indicators, when generated on a regular basis in a dynamic system, can be valuable tools for the management and evaluation of the health situation and actions at all Public Health levels.

For that purpose, it is important to establish an M&E process of public health policies, since the SUS management has used concepts, methodologies and the most diverse tools for more than two decades, Berreta et al.³⁰ point out that this modus operandi, while allowing the dissemination and expanding of this practice in strategic areas, requires a more systemic view, in order to contribute to the policy and program implementation process.

Therefore, indicators with systematic measurements were selected to assist in the monitoring and performance of the health system, subsidizing strategic and operational analyses, accountability and the attainment of the objectives and goals agreed in the 2016 – 2019 management.

Occurring concomitantly with the selection of indicators was the creation of a technical data sheet for the qualification of indicators, with predefined fields that detail the main characteristics of each indicator, including its calculation memory, interpretation and limitations. A catalog is being prepared, to be made available at the SHS/DF Health Situation Room, containing all the indicators monitored by the management tools.

Another ongoing initiative is the use of a cooperative monitoring and evaluation approach to risk situations and social vulnerability of the Territory, under the perspective of making it Healthy and Sustainable, having as reference the 2030 Agenda for Sustainable Development Goals (SDGs).

For that purpose, a methodology for the assessment of indicators related to SDG Number 3 is being created: “Ensure healthy lives and promote well-being for all at all ages” using an interactive system of analysis, monitoring and evaluation of the Territory, with the participation of the local community and the activation of socio-technical networks.

It has also been proposed to establish a collaborative space in the Portal of the Situation Room in Health, to disseminate the information of the Networks. The first site in which the methodology is being tested as a Project is known as the Structural City, also referred to as the Structural Village, which comprises the administrative region of the Complementary Industry and Supply Sector (SCIA) in the Federal District.

Health Situation Room (HSR) of the SHS/DF

The issue of the M&E of public policies meets a demand from society for a high-quality and planned public management, a new model of management of public resources, which has been strengthened since mid-1990s³¹. An important component of this process is the dissemination of information aiming at increasing management transparency and social regulation.

The permanent availability of information helps managers in the design and operationalization of planning activities, in decision-making support and should make it clear the intentionality of policies, programs and projects³². The information and analysis model must be accessible and within society’s reach, giving M&E a pedagogical and reflective role in the planning process³³.

The SHS/DF SSS is a physical and virtual space, constituting an integrating data and information tool of the main health information systems being used in SHS, allowing the democratization of strategic information, such as indicators of the health status of the population. The SSS helps in the analysis of sanitary data and in the situational diagnosis, allowing the identification of the population’s priority problems in their territories and the local health needs, subsidizing the management in health action planning.

Based on the principles and guidelines of the National Policy of Information and Informatics in Health (PNIIS)³⁴, the Law of Access to Information (LAI)³⁵ and of Social Control, which indicate the democratization of health information, the SHS / DF, in 2017, formalized an agreement with Fundação Oswaldo Cruz (Fiocruz) that establishes support for the qualification and improvement of M&E, information management and the implementation of an SSS.

It is essential to substantiate the qualification process of the produced information, to know the source of the data and to evaluate whether they are valid and reliable, a fundamental condition for the objective analysis of health conditions, as well as to make evidence-based decisions.

It is also essential to know the population’s main demands for information, sent through the Ombudsman’s Office or the Electronic System of the Citizen’s Information Service (e-SIC), with the purpose of making them available in the Situation Room Portal and thus increase SES active transparency.

Therefore, in terms of uses and functions, the SSS of the SHS/DF is prepared not only to provide qualified information to support the process of planning, monitoring and evaluation of health actions, but also to increase the institution’s active transparency, in total alignment with the international Open Government Partnership initiative, which aims to disseminate and globally encourage government practices related to government transparency, access to public information and social participation (Figure 4).

Figure 4
Situation Room as an integrating element of the SHS/DF Monitoring and Evaluation System.

In this sense, research and studies are being carried out on information and communication technologies (ICT) available at the national and international markets and their possibilities of use in the Situation Room and other components of the M&E system, with the objective of updating the processes and improving the organization of the current infrastructure, considering the needs of SHS/DF.

We highlight the use of free software with standards, technologies, procedures and control mechanisms necessary to meet the conditions of dissemination and sharing of data and public information using the Open Data model, in accordance with the Government Electronic Interoperability standards (e-PING).

The SSS integrates the M&E system of SHS/DF and contributes to the monitoring of the several management tools agreed upon and their main indicators, with the purpose of subsidizing strategic analyses, accountability, achievement of objectives and goals agreed upon in the 2016-2019 management.

The training of employees involved in the planning, monitoring and evaluation process is another sustainability element related to a good-quality M&E process. By believing this is true, the SHS/DF will promote in the year 2019, through a partnership with Fiocruz, an EAD course of specialization (100 openings) and another one at the level of professional Master’s Degree (35 opening), both in Health Evaluation.

Final considerations

The SHS/DF SP aims to improve the management and quality of health actions, programs and services, and for that purpose, it has set as one of its priorities, the implementation of an M&E system, using SSS, to disseminate strategic and good-quality information, in an organized, systemic way, with a technical-scientific basis, made available as tables, graphs, maps, technical documents or strategic reports, making it easier for the manager, the worker and the citizen who uses SUS to know the reality of the DF population’s health care.

Otherwise, meetings for evaluations of results (RARs), another activity that integrates the M&E system of the SHS/DF and contribute to the construction of references and evidence that support decision-making, constitute spaces for discussion that allow the creation of knowledge in the organizational scenario, as they enable the interaction of tacit and explicit knowledge of each of the participants and, therefore, contribute to institutional maturation.

A continuous evaluation by the managers regarding the mechanisms and tools available in the M&E system of the SHS/DF and their capacity to demand and produce information on the performance and results of health programs, aiming at their use, contributing to institutional learning and the improvement of public policies is expected.

Referências

• ¹
  Organización MS, Organización PS. Problemas conceptuales y metodológicos de la programación de la salud Washington: World Health Organization; 1965. Publicación Cientifica nº 111.
• ²
  Testa M. Estrategia, coherencia y poder en las propuestas de salud – Parte I. Cuadernos Médico Sociales 1987; (38)4:24.
• ³
  Testa M. Estrategia, coherencia y poder en las propuestas de salud. Parte II. Cuadernos Médico Sociales 1987; (39)3:28.
• ⁴
  Matus C. Política, planejamento e governo Brasília: IPEA; 1993.
• ⁵
  Matus C. Adeus, senhor Presidente. Governantes governados São Paulo: Edições Fundap; 1997.
• ⁶
  Rivera FJU, Artmann E. Planejamento e gestão em saúde: histórico e tendências com base numa visão comunicativa. Cien Saude Colet 2010; 15(5):2265-2274.
• ⁷
  Godet M. Manual de prospectiva estratégica: da antecipação à ação Lisboa: Cepes; 1993.
• ⁸
  Paim JS, Teixeira CF. Política, planejamento e gestão em saúde: balanço do estado da arte. Rev Saude Publica 2006; 40(n. spe):73-78.
• ⁹
  Levcovitz E, Baptista TWF, Uchôa SAC, Nespoli G, Mariani M. Produção de conhecimento em política, planejamento e gestão em saúde e políticas de saúde no Brasil (1974-2000) Brasília: Organização Pan-Americana da Saúde; 2002. (Série Técnica Projeto de Desenvolvimento de sistemas e Serviços de Saúde, 2).
• ¹⁰
  Merhy EE. O SUS e um dos seus dilemas: mudar a gestão e a lógica do processo de trabalho em saúde (um ensaio sobre a micropolítica do trabalho vivo). In: Fleury S, organizadora. Saúde e democracia: a luta do CEBES São Paulo: Lemos Editorial; 1997.
• ¹¹
  Brousselle A, organizador. Avaliação: conceitos e métodos Rio de Janeiro: Editora Fiocruz; 2011.
• ¹²
  Contandriopoulos AP. Avaliando a institucionalização da avaliação. Cien Saude Colet 2006; 11(3):705-711.
• ¹³
  Habicht JP, Victora CG, Vaughan JP<italic>.</italic> Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol 1999; 28(1):10-18.
• ¹⁴
  Instituto Brasileiro de Geografia e Estatística (IBGE). População, projeção. Rio de Janeiro: IBGE; 2016. [acessado 2018 Maio 20]. Disponível em: https://www.ibge.gov.br/apps/populacao/projecao/
  » https://www.ibge.gov.br/apps/populacao/projecao/
• ¹⁵
  Instituto Brasileiro de Geografia e Estatística (IBGE). População, projeção. Rio de Janeiro: IBGE; 2015. [acessado 2018 Maio 20]. Disponível em: https://www.ibge.gov.br/apps/populacao/projecao/
  » https://www.ibge.gov.br/apps/populacao/projecao/
• ¹⁶
  Hartz ZMA. Meta-avaliação da gestão em saúde: desafios para uma nova saúde pública. Cien Saude Colet 2012; 17(4):829-837.
• ¹⁷
  Garcia RC. Subsídios para organizar avaliações da ação governamental. Planejamento e Políticas Públicas 2001; 23:7-70.
• ¹⁸
  Brasil. Decreto nº 7.469, de 4 de maio de 2011. Regulamenta a Lei Complementar no 94, de 19 de fevereiro de 1998, que autoriza o Poder Executivo a criar a Região Integrada de Desenvolvimento do Distrito Federal e Entorno - RIDE e instituir o Programa Especial de Desenvolvimento do Entorno do Distrito Federal. Diário Oficial da União 2011; 5 maio.
• ¹⁹
  Brasil. Decreto nº 38.017, de 21 de fevereiro de 2017. Diário Oficial do Distrito Federal 2017; 22 fev. Republicação: Diário Oficial do Distrito Federal 2017; 23 fev.
• ²⁰
  Distrito Federal. Secretaria de Saúde (SS). [Internet]. Brasília: SSDF; 2012. [acessado 2018 Maio 20]. Disponível em: http://www.saude.df.gov.br/wp-conteudo/uploads/2017/11/RAG_2012-Parte_I.pdf
  » http://www.saude.df.gov.br/wp-conteudo/uploads/2017/11/RAG_2012-Parte_I.pdf
• ²¹
  Gottems LB, Oliveira A, Roque MV, Neves MJ, Oliveira CFDS. O programa de gestão regional da saúde no Distrito Federal. X Congresso Consad, 2017, 13p.
• ²²
  Blumm MHN, Bauer CAB, Gama GMS, Costa MKA, coordenação; Almeida MO colaboração. Manual de gestão de processos da Secretaria de Saúde do Distrito Federal Brasília: Subsecretaria de Planejamento, Regulação, Avaliação e Controle; 2014.
• ²³
  Brasil. Ministério da Saúde (MS). Manual de planejamento no SUS Brasília: Fiocruz, MS; 2016.
• ²⁴
  Universidade Federal do Maranhão (UFMA). Gestão pública em saúde: programação anual de saúde e relatório anual de saúde São Luís: UNA-SUS/UFMA; 2016. (Guia de Gestão Pública em Saúde, Unidade V).
• ²⁵
  Distrito Federal. Secretaria de Saúde (SS). Planejamento e gestão em saúde Brasília: SSDF; 2012. [acessado 2018 Maio 30]. Disponível em: http://www.saude.df.gov.br/planejamento-e-gestao-em-saude/
  » http://www.saude.df.gov.br/planejamento-e-gestao-em-saude/
• ²⁶
  Felisberto E. Da teoria à formulação de uma Política Nacional de Avaliação em Saúde. Cien Saude Colet 2006; 11(3):553-563.
• ²⁷
  Carvalho ALB, Souza MF, Shimizu HE, Senra IMVB, Oliveira KC. A gestão do SUS e as práticas de monitoramento e avaliação: possibilidades e desafios para a construção de uma agenda estratégica. Cien Saude Colet 2012; 17(4):901-911.
• ²⁸
  Conselho Nacional de Secretários de Saúde (CONASS). Os sistemas de informação em Saúde. In: Conass. Coleção Progestores – Para entender a Gestão do SUS. Sistema Único de Saúde Brasília: CONASS; 2007. p.74-101.
• ²⁹
  Santos LC. Crítica a modelos de avaliação de desempenho de sistemas de saúde [tese]. São Paulo: Universidade de São Paulo; 2015.
• ³⁰
  Berreta IQ, Lacerda JT, Calvo MCM. Modelo de avaliação da gestão municipal para o planejamento em saúde. Cad Saude Publica 2011; 27(11):2143-2154.
• ³¹
  Brasil. Decreto nº 7.508, de 28 de junho de 2011. Regulamenta a Lei no 8.080, de 19 de setembro de 1990, para dispor sobre a organização do Sistema Único de Saúde – SUS, o planejamento da saúde, a assistência à saúde e a articulação interfederativa, e dá outras providências. Diário Oficial da União 2011; jun 29.
• ³²
  Rede Interagencial de Informação para a Saúde (Ripsa). Rede Interagencial de Informação para a Saúde Indicadores básicos para a saúde no Brasil: conceitos e aplicações 2ª ed. Brasília: Organização Pan-Americana da Saúde; 2008.
• ³³
  Brasil. Ministério da Saúde (MS). Planejamento estratégico do Ministério da Saúde: 2011 – 2015: resultados e perspectivas Brasília: Editora do Ministério da Saúde; 2013.
• ³⁴
  Brasil. Ministério da Saúde (MS). Política Nacional de Informação e Informática em Saúde Brasília: Editora do Ministério da Saúde; 2016.
• ³⁵
  Brasil. Lei nº 12.527, de 18 de novembro de 2011. Regula o acesso a informações previsto no inciso XXXIII do art. 5o, no inciso II do § 3o do art. 37 e no § 2o do art. 216 da Constituição Federal; altera a Lei no 8.112, de 11 de dezembro de 1990; revoga a Lei no 11.111, de 5 de maio de 2005, e dispositivos da Lei no 8.159, de 8 de janeiro de 1991; e dá outras providências. Diário Oficial da União 2011; 18 nov.

Publication Dates

• Publication in this collection
  27 June 2019
• Date of issue
  June 2019

History

• Received
  15 June 2018
• Reviewed
  06 Feb 2019
• Accepted
  26 Mar 2019