Evaluation Model for developing effective indicators for health surveillance actions in Brazil

Abstract

Objective:

to present an evaluation model for developing effective indicators for the Vigilância Sanitária (VISA) (Sanitary and Health Surveillance) actions in Brazil.

Methods:

an exploratory, evaluative study from primary sources data - electronic form by key informants and reports on national, international and secondary benchmarking visits - documental analysis.

Results:

the diversity of VISA management practices made it possible to define the evaluation object: Health Surveillance Actions - as an organized action system. The components: management; regulation; sanitary and health control; sanitary and health risk monitoring and information, communication and health education were derived from the aims of the intervention and composes a central structure of the Theoretical Model and the Logical Model Actions of VISA in the Sistema Nacional de Vigilância Sanitária (SNVS) (National Sanitary and Health Surveillance System) .

Conclusions:

The Vigilância Sanitária (VISA) (Sanitary and Health Surveillance) accumulates experience with the process indicators for monitoring actions, being incipient are capable of evaluating its impact. The theoretical resources in the evaluation area support the information management in the VISA field and besides contributing for studies on social determinants and the incorporation of analyses in historic series.

Key-words:
Health evaluation; Health surveillance; Health indicators

Resumo

Objetivos:

apresenta-se a modelagem avaliativa para a construção de indicadores de efetividade das ações de vigilância sanitária (VISA) no Brasil.

Métodos:

estudo avaliativo exploratório, com dados de fontes primárias - formulário eletrônico com informantes-chave e relatórios de visitas de benchmarking nacional e internacional e secundárias - análises documentais.

Resultados:

a diversidade das práticas nas instâncias gestoras da VISA possibilitou a definição do objeto de avaliação: Ações de Vigilância Sanitária -, enquanto um sistema organizado de ação. Os componentes: gestão; regulação; controle sanitário; monitoramento do risco sanitário e informação, comunicação e educação para a saúde originam-se dos objetivos da intervenção e compõem a estrutura central do Modelo Teórico e do Modelo Lógico das Ações de VISA no âmbito do Sistema Nacional de Vigilância Sanitária.

Conclusões:

a VISA acumula experiência com indicadores de processos para monitoramento das ações, sendo incipientes aqueles capazes de avaliar o seu impacto. Os recursos teóricos da área da avaliação apoiam a gestão da informação no campo da VISA, para além da contribuição de estudos sobre os determinantes sociais e a incorporação de análises de séries históricas.

Palavras-chave:
Avaliação em saúde; Vigilância sanitária; Indicadores de saúde

Introduction

The definition on concepts and operations using 'theory-based evaluation' has been the aim of the most intense international debate since the early 2000s returning to the idea originated in the 1970s on their potential support for formulation and the theory of interventions testing for evaluation purposes.^1-3 Although some authors do not address these in a very systematic or precise fashion, as Coryn et al.³ argue in their 2011 American Journal of Evaluation article, theory-driven evaluation practices are increasingly being used by specialists, theoreticians and evaluators.^1-3

The theory-based evaluation increases the capacity to estimate the impact of programs that are not conducive to robust experiment design based on predefined standards.⁴ The Theoretical modeling for evaluation aims to understand factors that may interfere with the expected effects and extrapolate the object of the study itself. It can be undertaken to answer a number of questions regarding to how an intervention was conducted, the changes it brings to, possible improvements, and the relation to costs and benefits, providing an understanding of why an intervention worked in a particular way.² The process involves establishing premises that seek to relate intervention outcomes with the devices it produced, a system of values, norms and guidelines that are associated to causal links between undertaken actions and the outcome of such action. The context in which these occur, it also considers in creating a structure to facilitate value judgments; discussion among formulators, implementers and evaluators regarding the appropriateness of the intervention and even the evaluation itself and the way it is used.^2,4-8 It may also contribute with the research most used by managers in the decision making, since it starts out from the aims and guidelines of the interventions and provides easiness in understanding them through greater knowledge of the object of study, insofar as references of different approaches in theories and methods.²

In search of measuring effectiveness in the public sector aims to gauge the outcomes of actions that benefit the population. In the public health system as for in any other organized system of action, the desired results are the consequence of a carefully planned and well-executed processes.⁹ In the health evaluation field, it has become a commonplace to approach the concepts of impact and effectiveness, in other words, the effect of an intervention in real, not experimental situations, particularly those directed towards specific populational groups should be measured over long periods of time.^10-12 However, this challenge should be encounter by the acknowledgement of the inherent limitations in the evaluation designs and that these will not be overcome starting by innovations in the statistical approaches.¹³

The Vigilância Sanitária (VISA) (Sanitary and Health Surveillance) area in Brazil has accumulated a great amount of experience regarding to the development of indicators process for monitoring planned actions (management, administrative and technical) and this has contributed to the work organization at the Sistema Nacional de Vigilância Sanitária (SNVS) (National Sanitary and Health Surveillance System).¹⁴ However, the indicators structures which can evaluate the impact in its action is still not incipient. The theoretical resources in the health evaluation field can support the VISA information management and contribute to the decision making at the SNVS, whose practices are based on the principles of promoting and protecting the population's health.¹⁴ On the other hand, it is important to consider the inexistent description of a specific VISA policy in Brazil and this limits the use of the classic evaluation model of the policy cycle, in which it considers as phases of formulation, implantation and evaluation.^14-16 Therefore, considering the indicators that intend to measure the effectiveness of the VISA actions should bear in mind the importance of the debate regarding the institutionalization of evaluation and organizational improvement strategies.^12,14

This article presents an evaluation model for developing effective indicators for health surveillance in Brazil and proposes a systematization of elements that constitute and form the basis of the highest VISA priorities being developed on by various sectors of the SNVS

Methods

This is an exploratory evaluative study was carried out between June 2016 and September 2017, which adopted the principals of the triangulation of methods for interpretative articulation of data. These were collected by primary sources, by an electronic form with key informants and the analysis of the reports on national and international benchmarking visits, and data from secondary sources by using documental analysis. The data collection contemplated the proposition by the health surveillance and regulation managers of the field of themes to be approached in the evaluation and besides involving the identification of the initiatives, institutional practices, consolidated projects and the availability of databases and other existing sources of information.

The electronic forms were filled in by managers of the technical sectors of the Agência Nacional de Vigilância Sanitária (Anvisa) (National Sanitary and Health Surveillance Agency), the Secretaria de Vigilância em Saúde do Ministério da Saúde (SVS/MS) (Secretary Surveillance in Health of the Brazilian Ministry of Health) and by the members of the Grupo de Trabalho Tripartite da Vigilância Sanitária (GTVisa) (Tripartite Sanitary and Health Surveillance Work Group) and they aimed to identify the highest priority in health surveillance items and working processes to be considered in the model structure. The reports on the national benchmarking visits were referred to initiatives in health surveillance monitoring that made use of national data and local health information systems in the States of Goiás, Minas Gerais, Santa Catarina, São Paulo and Paraná, and the two city capitals: Belo Horizonte and Curitiba. The international visits were conducted at institutions in two countries specifically chosen from those with which Brazil engages in technical cooperation in the field of health surveillance and regulation. In Portugal, the Autoridade Nacional do Medicamento (National Authority on Medications) and the Produtos de Saúde (Infarmed) (Health Products) and the Direção Geral de Saúde (DGS) (General Director in Health), and the United Kingdom, the Medicines & Healthcare Products Regulatory Agency (MHRA) and Public Health England (PHE). These visits designed subsidize to foster more in-depth discussion in monitoring and evaluating the effectiveness in the sanitary and health surveillance actions. The documental analysis was carried out on the following sources: (i) the 2007, Plano Diretor de Vigilância Sanitária (PDVISA) ¹⁷ (Sanitary and Health Surveillance Director Plan); (ii) the 2015, Textos de Referência dos Ciclos de Debates em Vigilância Sanitária¹⁸ (Reference Texts from the Sanitary and Health Surveillance Debates Cycles); and (iii) the 20162019 cycle, the Planejamento Estratégico da Anvisa¹⁹ (Anvisa Strategic Plan).

To define the object and the delimitation focus that scopes the theoretical model with a view to facilitate the description and identification of the underlying theory in which eight workshops were conducted between February and September 2017, involving technical and managers from the Assessoria de Planejamento (Assistant Planning Board) and various managing sectors from Anvisa and consultants specializing in health surveillance and regulation, public planning, public administration, epidemiology and health evaluation. The meetings were held at Anvisa offices in Brasília, DF. The workshop discussions were centered in the choice of referral theorist to be used and in the principles and guidelines that rule VISA actions. The parameter adopted for the model was a national level analysis involving policy-making, technical cooperation, planning, SNVS coordination, co-funding, skills development and action taken at State or Municipal level, or directly and/or in complementary way by the national regulatory agency. The information obtained was also triangulated in a comparison form among the answers in electronic forms, national and international benchmarking visit reports, the workshops and among these, the findings of the documental analysis.

This article is in compliance with the provisions of Article 1, single paragraph, Item VII, of the Resolution nº. 510, on April 7, 2016 at the National Health Board which concerns specific ethical considerations applicable on researches in Social and Human Sciences.

Results

The Theoretical Model to Evaluate the Health Surveillance Actions in the SNVS scope

The analysis on the diversity of practices developed at the sub-national mangers stage and the national regulatory agency and adding the knowledge to their different profiles made it possible to determine the evaluation object of which would come to serve as the guiding thread of the model. This object called 'Sanitary and Health Action' is understood to refer to an intervention, in other words, an organized system of action is constituted of the most representative priority actions being undertaken by the System. For the purposes of this model, these were grouped into components, in accordance with those adopted by various institutional Anvisa documents as well as those produced and institutionalized by the SNVS tripartite management group within the Sistema Único de Saúde (SUS) (Brazilian National Health System). These components: management, regulation, sanitary and health control, sanitary and health risk monitoring and information, and communication and health education were derived from the intervention aims and sought to provide systematization that offers the clarity needed for building up the logical model.¹⁴

The articulation that predicts the connection between the intervention and expected results may suggest evaluations of processes and/or more immediate, intermediate and final results or impacts, insofar as these contribute to the final aims. These may be affected by crosscutting external and/or internal factors to different dimensions of the intervention, which form part of the context and are affected by the existing situation in which the intervention occurs. The contextual factors influence the installation and the effects of the intervention and may be of a political, organizational or socioeconomic nature. The health conditions and sanitary risks are also elements of the context that might influence and be influenced by the intervention.

The effects of activities in one component are the direct outcomes most affected by the diverse interactions within the SNVS scope, constituting the most immediate and intermediate consequences of the model. With the final outcomes or impacts, they are related to the effect of the intervention on the overall population. For the proposed model, the predicted impacts were determined to have a reduction in morbidity and mortality, health promotion and protection and social recognition from VISA. These are constantly subject to the influence of contextual factors. The model also suggests that the evaluation itself is a tool that promotes changes in the behavior of the professionals at the SNVS, contributing for its cultural and organizational incorporation in the evaluation of the SNVS and the SUS and its eventual institutionalization.

Figure 1 shows the various theoretical elements that facilitates the understanding on these relations.

Figure 1
Theoretical Model to Evaluate VISA Actions.

The Logical Model of the Health Surveillance Actions

The logical model presents the most objective elements that enable a stronger correlation between the components previously defined in the theoretical model. These were then divided into subcomponents with the exception of the Sanitary and Health Risk Control component. The proposed subdivision (Table 1) enables to enhance the systematization of the activities defined as priority to achieve the objectives.

Table 1
The components and subcomponents on the logical model of VISA Actions.

Components	Subcomponents
Management	Planning Skills development Knowledge management
Regulation	Legislation and legal framework Sanitary and Health licensing Regularization on products and services Market regulation
Sanitary and Health Risks Control	Sanitary and health risks control Products Services
Monitoring on Sanitary and Health Risks	Antimicrobials resistance Patient's safety VISA emergencies
Information, communication and health education	Relationship with the society Cross-sector partnerships

Source: Brazil, 2018.

The subcomponents of Planning, Skills Development and Knowledge Management which make up the Management component allowing to group together in activities related to the attributions of the system manager who should assure SUS principles and guidelines, foremost among which is decentralization as a strategy for strengthening the SNVS. The subcomponents of Legislation and Regulatory Framework, Sanitary and Health Licensing, Regulation on Products and Services and Market Regulation which make up the Regulation component systemizing to group together in activities that promote desirable behavior and deter undesirable behavior as far as the population's health is concerned in terms of the production and circulation of goods. The activities most closely related to oversight by the SNVS organizations, principally those concerning the control of production of goods and services, once these have been made available to the consumer they are grouped together in the Sanitary and Health Risk Control component.

The post market surveillance activities seek to meet the SNVS challenges regarding to minimize health risks and promotion a better quality of life. In view of the scope and cross-sector nature of these activities, they were divided into five subcomponents: Products, Services, Antimicrobials Resistance, Patient's Safety and VISA Emergencies. These make up the Sanitary and Health Risk Monitoring component, which is the most complex of the settings in which the SNVS operates institutional, since it requires constant upgrading of technology and organization of the necessary operational working processes. The final component, covering the subcomponents Relationship with the Society and Cross-Sector Partnerships to group together in activities related to information management that represent participatory actions in the fields of communications, mobilization, social control and education.

Tables 2 and 3 present schematically outlined the relations described above and their plausibly causal relation to the intended outcomes, represented as: (i) immediate effects - which are related to each component and generated directly by the developed activities; (ii) intermediate effects - related to a set of components; and (iii) impacts - to which Sanitary and Health Surveillance Actions contributes in synergy with other SUS actions developed and within the various contexts represented in the theoretical model. The last two are verifiable by identification of the indicators affected by each of the components established in the evaluation model. It is important to note that an intermediate effect may be a consequence of one or more immediate effects, just as an immediate effect may contribute to more than one intermediate one. Table 3 presents a clearer visualization of the logical model, not showing how predicted intermediate effects and impacts may be causally related to various immediate effects and activities.

Table 2
Components, Subcomponents, Activities and Immediate Effects on the Logical Model of VISA Actions.

Components	Subcomponents	Activities	Immediate Effects
Management	Planning	• Mapping on sanitary and health situation in establishments subject to VISA	• Increased capacity for local action
		• Coverage establishment and goal performance	• Improved compliance with established sanitary and health standards
		• Articulation of the VISA actions with Health Plan priorities at three management levels	• Higher rational work processes in VISA
	Skills Development	• Professional qualification (training and up skilling)	• Increase on scientific research in VISA
	Knowledge Management	• Promotion on VISA studies and researches
		• National and international cooperation in science and technology
		• Knowledge exchange (events, seminars, collaborative networks, etc.)
		• Production and publication information on decision-making
Regulation	Legislation and Legal Framework	• Development and upgrading of normative framework of good regulatory practices
		• Analysis on regulation impact	• More predicable impact on the installation of sanitary and health norms
	Sanitary and Health Licensing	• Authorization to work	• More performance on clinical research by the regulated sector
		• Issuing sanitary and health license	• Reduction on shortages on high priority medications
		• Certificate for good practices
		• Training at pharmaceutical center
		• Training and accreditation for labs
	Regularization on products and services	• Product registration	• Improve quality requirements and safe pharmaceutical materials, medication and products
		• Post-registry alterations
		• Recording and reporting on low-risk products
		• Pharmacopeia Updated
		• Approval on imported and exported goods and products
		• Approval on clinical research
	Market Regulation	• Monitoring on medication cost
		• Assessment on cost of new products and new versions of medications
		• Monitoring on medication shortages
Sanitary and Health Risk Control	Sanitary and Health Risk Control	• Fiscal analyses to evaluate product quality	• Greater consistency of products and services available for the consumer
		• Sanitary and health inspection	• Reduce number of infractions
		• Sales control on antimicrobials	• Increase sanitary control of products and services at entry areas into the country
		• Sanitary and health administration process	• Reduce epidemic outbreaks during large-scale events
Monitoring on Sanitary and Health Risk	Products	• Monitoring food	• Reduce % on food sample with potential acute health risk
		• Monitoring agrotoxins levels in food	• Reduce levels of sugar, sodium, fat and agrotoxins in processed food
		• Monitoring technical complaints and adverse events related to products	• Reduce reports of ineffective drugs
		• Monitoring the quality of water for human consumption	• Reduce reports of false medication
		• Monitoring the quality of drugs available in the market	• Reduce in underreporting of adverse events
		• Monitoring product advertising	• Reduce in underreporting of technical complaints
	Services	• Monitoring health services	• Increase number of satisfactory results on medications evaluation
		• Monitoring services related to health	• Reduce number of health services classified as medium-high or high risk
	Antimicrobials Resistance	• Monitoring antimicrobial waste in food	• Increase number of drinking water samples
		• Monitoring antimicrobials resistance	• Increase number of satisfactory reports regarding maximum limits for waste in water for human consumption
	Patient's Safety	• Monitoring levels of healthcare-related infection	• Reduce cases of microorganisms infection resistant to antimicrobials
		• Monitoring adverse healthcare events
	VISA Emergencies	• Investigation on VISA emergencies	• Reduce outbreaks of bacterial infection caused by multiresistant microorganisms in the ICUs
		• Management information on VISA emergencies	• Reduce bed sores in long-stay patients
			• Reduce falls among hospitalized elderly patients
			• Reduce errors regarding administration of medication
			• Reduce infection resulting from catheter reuse
I n f o r m a t i o n , Communication and Health Education	Relationship with the Society	• Communication on sanitary and health risks	• Greater access to VISA information for the population
		• Meeting spontaneous demands of the population and management on resulting information	• Greater visibility for VISA actions
		• Liaising with social control organizations at various levels
	Intersectoral partnerships	• Liaising with teaching institutions to include content related to VISA at various levels of formal education
		• VISA campaigns and events

Source: Brazil, 2018.

Table 3
Intermediate Effects and Impacts on the Logical Model of VISA Actions.

Intermediate Effects	Impactos
• Improve the quality in VISA decision-making	• Reduce morbidity and mortality
• Widen safe access for products and services subject VISA	• Protection and promotion of health
• Better knowledge about VISA	• Social recognition for VISA Actions
• Reduce sanitary and health risk when introducing products and services
• Reduce cases of diarrheia, influenza, and food-borne diseases in confined environments (ships, oil rigs)
• Reduce cases of acute intoxication by agrotoxic waste in food
• Reduce cases of cancer related to agrotoxins exposure in rural workers
• Improve quality, safety and effectiveness of products and services
• Improve nutritional value in processed foods
• Reduce severe adverse events associated to medication use and health products
• Better quality of water for human consumption
• Reduce levels antimicrobials resistance
• Reduce hospital infections
• Reduce indiscriminated antimicrobials consumption
• Widen the participation of the society in formulating norms
• Improve consumption of products and services
• Improve public satisfaction with VISA Actions
• Improve SNVS communication scope

Source: Brazil, 2018.

Discussion

The debate regarding the professionalization of results based on public administration in the field of sanitary and health surveillance began in Brazil in 1999 with the creation of Anvisa and the qualifications required by the Laws nº 9,782/1999 and nº 13,411/2016.^20-21

On the other hand, important forums for discussion became possible to define the VISA policy guidelines centered on planning, monitoring and evaluation.^17-18.²² These are laid out in the recommendation of the I Conferência Nacional de Vigilância Sanitária (National And health Conference) in 2001, as follows - to define an integrated set of indicators, to create a methodology for self-evaluation in sanitary and health surveillance and also to propose a national project to evaluate the impact of these actions in terms of improving the population's quality of life²² and in the PDVISA-2007 guidelines - The development in planning, monitoring, evaluation and auditing should be continuously shared at all three levels of government for further support in the process of attending our responsabilities.¹⁷

In recent years, some movements and institutional accords have mobilized managers to value monitoring, health,evaluation actions and management.^23-24 However, the indicators related to VISA have focused on operational work processes and these have performed poorly in terms of promoting greater capacity to undertake actions. Examples of this are "The percentage of municipalities with the agreed santitary and health surveillance strategies" and "The percentage of medication companies inspected by the sanitary and health surveillance officers per year", which are included in the Pacto pela Saúde (Pact for Health).¹⁴

The report on the Cycle of Sanitary and Health Surveillance Debates in 2015, which included one national, one international and five regional forums, noted that one of the most important challenges is to share information about the actions with VISA. This is because of outdated information management technology along with poor quality databases and limited interoperability between existing information systems hindering monitoring, evaluation, planning and communication both within the SNVS, and with other similar organizations and the population.²⁵ The present study corroborates this, with the States and municipalities reporting, on benchmarking visits, that system maintenance is hampered by the fragmentation of national health information systems and a slow uptake on new information technology in the public administration. This not only difficult the identification of relevant information to the SNVS management, but also undermines the effective fulfillment of the obligations handed down to by the Ministry of Health.

The international benchmarking visits (Portugal and the United Kingdom) pointed out the importance of a more advanced structuring of information management for the development of sanitary and health regulation. The strategy of international alignment of the sanitary and health regulation in Europe has promoted good practices and led to the establishment of parameters for measuring the effectiveness of regulatory action. It has also strengthened institutional integration through the alignment and assimilation between the regulator and other health system operators and helped to build up information networks spanning various regulatory institutions across the continent. These visits also noted advances in the structuring and availability of healthcare databases that help researchers in their quest for scientific evidence on which to base regulators' decision-making. These advances are brought about by financial incentives that aim to improve the quality, coverage and comprehensiveness for information by investing in the training of those who report information, the creation of web platforms and the introduction of internal and external auditing. Evaluation by the society, the performance of state regulatory agencies by the public inquiries is also presented as an important strategy for improving oversight.¹⁴

As we have observed, international advances in the regulation have been strengthened on information management in the installation of the evaluation processes and, in Brazil, this has been a recurrent theme in recent decades and a priority among the SNVS professionals. This shows the current importance and urgent need for a VISA evaluation model that favors the development of strategies for overcoming the SNVS's information management challenges and thereafter the identification and measurement of indicators that evaluate the outcome of its actions. The Theoretical Evaluation Model and the Logical Intervention Model described here outline a method of evaluation whose implementation requires consistent institutional support for the structuring of information management firmly grounded in reliable, sensitive and wide-ranging sources.¹⁴

The incipient nature of the structured SNVS evaluation practices that go beyond measuring coverage demonstrates the need of processes based on consistent levels of evidence. These may arise from the institutionalization of evaluation practices that aim to strengthen organizational learning and privilege outcome based assessment, while continuing to explore the mechanisms underlying processes that result in effective actions.²⁵ the theory based on the evaluation modeling is a strategic tool for decision-making and changing practices and may help to improve the safety and quality control of the products and services which is the ultimate action of the sanitary and health surveillance in SUS.

In short, as communities broaden their capacity to produce and this generates imbalances in the market and new technological risks, there is a need for direct intervention in various areas of collective and individual interest and in the relation between health, disease and quality of life. Establishing equilibrium requires planning to be a strategy for prioritizing actions and this is an indispensable component of any adequate management of health risks in Brazil.²⁶ Thus, the sanitary and health surveillance becomes more powerful if it incorporates concepts in planning, development, control and evaluation and moves away from a purely supervisory and punitive perspective towards a more integrated mode of operating and the production of more easily deliverable outcomes.²⁷

The growing demand for added value in the production and provision of services by public organizations in the field of healthcare requires more systematic understanding of the process of applying knowledge.⁹ It is clear, however, that the complexity of the organizational and natural environments that VISA covers requiring the development of tools and conditions that aid managers in making better decisions based on the production, dissemination and appropriation of knowledge. This in turn helps to strengthen interaction and cooperation among policy-makers, managers, technicians and healthcare professionals, teachers and researchers, and others who contribute to advance knowledge and improve sustainable actions.²⁵

However, just as VISA cannot be confined to the SUS, evaluation should also receive feedback from other sources of information and evidence regarding health surveillance and healthcare that derive from the cross-sector interactions and practices within which VISA operates. The institutionalization of evaluation also presupposes the willing of participating institutional leaders, the development of skills evaluation and the creation of strategies for publishing results and, above all, viable reviews on policies and practices as a result of the evaluation.²¹

The effectiveness and impact in healthcare interventions are affected by the complexity and interdependence of the factors that determine health and disease in human populations.¹⁷ These relate to the degree of satisfaction and the value added to the changes brought about.²⁸ The various institutions that make up the health system carry out complementary intertwined actions that seek to achieve favorable outcomes. The measurement of population level indicators reveals the effect of these actions as a whole, but makes it difficult, if not impossible, to ascertain the exact portion each institution involved has contributed to improve healthcare indicators. Studies on social determinants and historic time series can be expected to light and broaden the base of evidence by supporting inferences regarding the effects of the interventions.^10,11

However, it should be understood that the prime role of any theory of evaluation is that the light on the relations between the object of evaluation and the elements of which it is composed, without pretending to do so, is exhaustive. The model outlined here constitutes a first step towards consolidation of the incorporation of evaluation into a daily work of the SNVS and its institutionalization as a strategy to strengthen sanitary and health surveillance in Brazil as a whole.

References

• ¹
  Christie CA. Advancing empirical scholarship to further develop evaluation theory and practice. Can J Progr Eval. 2011; 26(1): 1-18.
• ²
  Van der Knaap P. Theory-based evaluation and learning: possibilities and challenges. Evaluation. 2004; 10(1): 1634.
• ³
  Coryn CLS, Noakes LA, Westine CD, Schröter, DC. A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Eval. 2011; 32: 199-226.
• ⁴
  Lam TCM. Theory-based evaluation and objective-based evaluation: an integration of the two approaches. European Evaluation Society Conference; 2002 oct 12; Seville (Spain).
• ⁵
  Rossi PH, Freeman HE, Lipsey MW. Evaluation. A systemic approach. 7 ed. Sage Publications; 2004.
• ⁶
  Stufflebeam DL., Shinkfield AJ. Evaluation theory, models & applications. San Francisco: Jossey-Bass; 2007.
• ⁷
  Rogers PJ. Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation. 2008; 14(1): 29-48.
• ⁸
  Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z. Avaliação: Conceitos e Métodos. Rio de Janeiro: Editora Fiocruz; 2011.
• ⁹
  Goldsteen RL, Goldsteen K, Graham DG. Introduction to public health. New York: Springer Pub.; 2011.
• ¹⁰
  Vieira-da-Silva LM. Avaliação de Políticas e Programas de Saúde. Coleção Temas em Saúde. Rio de Janeiro: Editora Fiocruz; 2014.
• ¹¹
  Champagne F, Brousselle A, Contandriopoulos A-P, Hartz Z. A Análise dos Efeitos. In: Brousselle A, Champagne F, Contandriopoulos A-P, Hartz Z, organizadores. Avaliação: conceitos e métodos. Rio de Janeiro: Editora Fiocruz; 2011. p. 159-82.
• ¹²
  Samico I, Felisberto E, Figueiró AC, Frias PG. Avaliação em Saúde: bases conceituais e operacionais. 1 ed. Rio de Janeiro: Medbook; 2010.
• ¹³
  Hyder AA, Puvanachandra P, Morrow RH. Measuring the health of populations: explaining composite indicators. J Public Health Res. 2012; 1(3):222-8.
• ¹⁴
  Brasil. Ministério da Saúde. Avaliação das Ações de Vigilância Sanitária: uma proposta teórico-metodológica / Ministério da Saúde, Agência Nacional de Vigilância Sanitária, Hospital Alemão Oswaldo Cruz. - Brasília, DF; 2018. 96p. ISBN 978-85-334-2608-5
• ¹⁵
  Haney SR, Gonzales-Block M, Buxton MJ, Kogan M. The Utilization of Health Research in Policy-making: concepts, examples and methods of assessment. Health Research Policy and Systems [periódico on line]. 2003. [acesso em jan 2018]. Disponível em: http://www.health-policy-systems.com/content/l/1/2
  » http://www.health-policy-systems.com/content/l/1/2
• ¹⁶
  Secchi L. Políticas Públicas: conceitos, esquemas de análise, casos práticos. São Paulo: Cengage Learning; 2011.
• ¹⁷
  Brasil. Agência Nacional de Vigilância Sanitária. Plano Diretor de Vigilância Sanitária - PDVISA. Brasília, DF; 2007.
• ¹⁸
  Brasil. Agência Nacional de Vigilância Sanitária. Ciclo de Debates em Vigilância Sanitária: Desafios e Tendências. Brasília, DF; 2016.
• ¹⁹
  Brasil. Agência Nacional de Vigilância Sanitária. Planejamento Estratégico - 2016-2019 - Gestão Estratégica. Brasília, DF; 2016.
• ²⁰
  Brasil. Presidência da República. Lei nº. 9.782, de 26 de janeiro de 1999. Define o Sistema Nacional de Vigilância Sanitária e cria a Agência Nacional de Vigilância Sanitária. Brasília: Diário Oficial da União 27 jan 1999.
• ²¹
  Brasil. Presidência da República. Lei nº 13.411, de 28 de dezembro de 2016. Altera a Lei nº 6.360, de 23 de setembro de 1976, que dispõe sobre a vigilância sanitária a que ficam sujeitos os medicamentos, as drogas, os insumos farmacêuticos e correlatos, cosméticos, saneantes e outros produtos, e dá outras providências, e a Lei nº 9.782, de 26 de janeiro de 1999, que define o Sistema Nacional de Vigilância Sanitária, cria a Agência Nacional de Vigilância Sanitária, e dá outras providências, para dar transparência e previsibilidade ao processo de concessão e renovação de registro de medicamento e de alteração pós-registro. Brasília, DF: Diário Oficial da União 29 dez 2016.
• ²²
  Brasil. Agência Nacional de Vigilância Sanitária. I Conferência Nacional de Vigilância Sanitária: Relatório Final. Brasília, DF; 2001.
• ²³
  Brasil. Ministério da Saúde. Portaria GM nº 399, de 22 de fevereiro de 2006. Divulga o Pacto pela Saúde 2006 - Consolidação do SUS e aprova as Diretrizes Operacionais do Referido Pacto. Brasília, DF: Diário Oficial da União 22fev 2006.
• ²⁴
  Brasil. Ministério da Saúde. Portaria GM nº 1.580, de 19 de julho de 2012. Afasta a exigência de adesão ao Pacto pela Saúde ou assinatura do Termo de Compromisso de Gestão, de que trata a Portaria nº 399/GM/MS, de 22 de fevereiro de 2006, para fins de repasse de recursos financeiros pelo Ministério da Saúde a Estados, Distrito Federal e Municípios e revoga Portarias. Brasília, DF: Diário Oficial da União 19 jul 2012.
• ²⁵
  Felisberto E, Samico I, Bezerra LCA, Hartz Z. Institucionalizando a avaliação nas organizações e agências de pesquisas: um estudo de caso exemplar. Saúde em Debate. [periódico on line] . 2017. [acesso em 7 Maio 2018] 41: p.387-99. Disponível em: <http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0103-11042017000 500387&lng=en&nrm=iso>.
  » http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0103-11042017000 500387&lng=en&nrm=iso
• ²⁶
  Alves SMC. Processo de participação da sociedade civil nas consultas públicas realizadas pela Agência Nacional de Vigilância Sanitária - Anvisa (2000-2006). [dissertação]. Brasília, DF: Instituto de Ciências Humanas da Universidade de Brasília; 2008.
• ²⁷
  Flexa RGC, Silva Júnior JB, Brito RL, Sousa AIA, Araújo FF, Martins MAF. Planejamento estratégico em vigilância sanitária: aplicação do Balanced Scorecard (BSC). Vigil Sanit Debate. 2017; 5(2): 13-23.
• ²⁸
  Brasil. Ministério do Planejamento, Desenvolvimento e Gestão. Indicadores de programas: Guia Metodológico. Brasília, DF; 2010. p. 128.

Publication Dates

• Publication in this collection
  Jul-Sep 2018

History

• Received
  11 July 2018
• Accepted
  10 Aug 2018