Methods and challenges for the health impact assessment of vaccination programs in Latin America

ABSTRACT OBJECTIVE To describe methods and challenges faced in the health impact assessment of vaccination programs, focusing on the pneumococcal conjugate and rotavirus vaccines in Latin America and the Caribbean. METHODS For this narrative review, we searched for the terms “rotavirus”, “pneumococcal”, “conjugate vaccine”, “vaccination”, “program”, and “impact” in the databases Medline and LILACS. The search was extended to the grey literature in Google Scholar. No limits were defined for publication year. Original articles on the health impact assessment of pneumococcal and rotavirus vaccination programs in Latin America and the Caribbean in English, Spanish or Portuguese were included. RESULTS We identified 207 articles. After removing duplicates and assessing eligibility, we reviewed 33 studies, 25 focusing on rotavirus and eight on pneumococcal vaccination programs. The most frequent studies were ecological, with time series analysis or comparing pre- and post-vaccination periods. The main data sources were: health information systems; population-, sentinel- or laboratory-based surveillance systems; statistics reports; and medical records from one or few health care services. Few studies used primary data. Hospitalization and death were the main outcomes assessed. CONCLUSIONS Over the last years, a significant number of health impact assessments of pneumococcal and rotavirus vaccination programs have been conducted in Latin America and the Caribbean. These studies were carried out few years after the programs were implemented, meet the basic methodological requirements and suggest positive health impact. Future assessments should consider methodological issues and challenges arisen in these first studies conducted in the region.

In the last thirty years, scientific and technological advances have resulted in the development and marketing of several new vaccines, increasing the opportunities to prevent morbidity and mortality related to infectious diseases of public health importance. 55 In the 2000s, global and regional initiatives and commitment to immunization reduced prices of these new vaccines, which became accessible for low-and middleincome countries. Consequently, national immunization programs have been offering new vaccines. 55  Once a new vaccine is introduced into routine immunization, it is necessary to monitor vaccine coverage, vaccine effectiveness and safety as well as the health impact of the vaccination program. Country differences in burden of disease, serotype and genotype distribution, health services organization and access, clinical practices, and surveillance systems prevent the use of international evidence as a guarantee of good results after implementing a program. Furthermore, vaccination programs may result in complex effects, changing the average age of infection, seasonal patterns of disease and genotype or serotype distribution.
Published studies use conflicting terms to describe different types of effects. 24 Vaccine effectiveness is defined as the ability of a vaccine to protect against disease when used under field conditions (routine practice). 24 Vaccine effectiveness refers to the protection conferred by individual immunization on vaccinated persons. 24 Vaccination programs affect all people, even if only part of the population is vaccinated. When many people are immunized, the pathogen transmission decreases, which reduces the disease incidence and, consequently, protects the unvaccinated ones (indirect effect or herd protection). The health impact of a vaccination program refers to the total effects of the program, meaning the total (direct and indirect) effect on the vaccinees and the indirect effect on unvaccinated persons. 24

INTRODUCTION
case-control studies. The health impact of a vaccination program is estimated by comparing all individuals of the population affected by the vaccination program with a reference population unaffected by any program, usually the same population before and after program implementation. Different methodological approaches of differing complexity may be used. 24 Countries with national health information systems, academic expertise in health services research, disease burden measurement and technology assessment in health care, policy makers, epidemiological surveillance, and immunization program professionals with experience in vaccine evaluations might have a more favorable context to conduct a health impact assessment (HIA) of vaccination programs. 51 Nevertheless, introducing a new vaccine may be the opportunity for countries to create conditions for this evaluation, and others to follow, particularly if international organizations stimulate and support these initiatives.
The World Health Organization recommended that all national immunization programs offer rotavirus vaccine and pneumococcal conjugate vaccine (PCV), 56,57 which have been introduced in Latin America and the Caribbean (LAC) from 2006 and 2008, respectively (Table 1). Hence, LAC countries have already had time to conduct HIA of rotavirus and pneumococcal vaccination.
The objective of this study was to describe the methodological approaches (study design, data sources and outcomes of interest) used and the challenges to conduct HIA of PCV and rotavirus vaccination programs, with focus on LAC countries.

METHODS
This is a narrative literature review of HIA of PCV and rotavirus vaccination programs, with focus on LAC countries. A search in Medline and LILACS, using the terms "rotavirus", "pneumococcal", "conjugate vaccine", "vaccination", "program", and "impact", was conducted on June 10, 2013 (PCV) and September 20, 2013 (rotavirus) and repeated on April 30, 2014. The review was supplemented with a search in Google Scholar to assess grey literature such as articles published in non-indexed journals, guidelines, and technical reports. There were no limits for publication year. Studies published in English, Spanish and Portuguese were eligible.
Three reviewers screened the identified abstracts and full texts and selected original articles that assessed the health impact of vaccination programs in LAC countries. Economic evaluations, mathematic models, vaccine efficacy or effectiveness studies and impact reviews were excluded. The references of all included articles were cross-checked and a hand search was carried out to identify further articles.
Data were extracted by one reviewer using a template developed specifically for this study and checked for accuracy by a second one. Data extracted from each study included: author, year, country, study design, data sources, clinical syndrome of interest, outcomes, and main results. Differences between reviewers were solved by discussion.

ANALYSIS OF RESULTS
We initially identified 207 articles in the search: 92 on pneumococcal and 115 on rotavirus vaccination programs. After applying the exclusion criteria based on title and abstract reading, and checking for duplicates, we read 37 articles on PCV and 60 articles on rotavirus vaccine in full. The search update added seven articles to the set. Finally, we reviewed 33 studies on HIA of vaccination programs conducted in LAC, 25 of which assessed rotavirus and eight, PCV vaccination programs.
Of the 20 Latin American countries, 14 have introduced rotavirus vaccine in their immunization programs since 2006, and we identified at least one published HIA for eight of them (Table 1). 12,27 In the Caribbean, only three of 25 countries have introduced rotavirus vaccine in their immunization programs since 2009 and no published HIA was identified. Since 2008, 17 Latin American countries have introduced PCV in their immunization programs and we identified at least one published HIA for five of them. Since 2009, five countries in the Caribbean have introduced PCV in their immunization programs and no published vaccination HIA was identified (Table 1). 12 The 33 included LAC studies are described in Tables 2, 3 and 4 according to vaccine, data sources, country, and study design. In most LAC countries, rotavirus vaccines have been introduced earlier than PCV. Consequently, more HIA of rotavirus vaccination programs have been performed and published (25/33, 75.8%). Most studies were conducted in Brazil (15/33, 45.5%), mainly on rotavirus (13/25, 52.0%). Three of the eight studies on pneumococcal vaccine were conducted in Uruguay, one of the first LAC countries to implement a PCV childhood vaccination program. 25,26,41,42 Ecological studies (interrupted time series analyses and other studies comparing pre-and postvaccination periods) were the most frequent (25/33, 75.7%). Cohorts (3), case series (3) and cross-sectional (2) studies were also conducted. Data sources were mainly secondary epidemiological or administrative databases (16/33, 48.5%). Surveillance data (8/33, 24.2%) and primary data collection (7/33, 21.2%) were also used. Two studies mixed data from both surveillance and health information systems. 6,11 The study design was tied to data characteristics.

Study design
Ecological studies are frequently used to evaluate epidemiological impact, especially when using large non-disease-specific databases, as they allow tracking population disease trends over time in relation to the timing of interventions. They allow both short-and long-term assessment of the vaccination program in a general population, but the establishment and measurement of causal relationship are limited because changes in disease incidence after vaccine introduction cannot be attributed exclusively to the intervention. Natural variations and secular trends affect disease incidences in the absence of vaccination. 11 Changes in social and health conditions and improvement in access to healthcare system during the study may also influence the results of before-after studies. 11 Strategies to control the effects of possible confounding factors include study design (comparison with other diseases or similar countries) and analyses (statistical methods to estimate the expected occurrence of the outcome using patterns before vaccine introduction, for example).
The rates of diarrhea-related hospitalizations and deaths of under-five children have been declining in LAC countries in the last three decades due to safe water supply, improvements in sanitation and hygiene, breastfeeding promotion, better nutrition, enhanced access to health care, and proper treatment of diarrhea, including oral rehydration therapy. 14 This decline may be misinterpreted, overestimating the impact of vaccination. In some LAC studies, this decrease was already evident before vaccine introduction. 14,22,29 Few studies adjusted for these secular trends appropriately in the analyses. 9,11,15,43 A Brazilian study used a generalized linear model to compare the postvaccination years with expected rates estimated from prevaccination years adjusted for secular and seasonal trends. 9 Two studies, in Mexico, used all-cause hospitalization to control these secular trends. 15,43 A neighbor and similar country that had not implemented rotavirus vaccination was used as a control for possible secular trends in a HIA of rotavirus vaccination in four LAC countries. 11 Rotavirus disease classically shows natural year-to-year variation, making it difficult to determine to which extent changes in disease trends are related to vaccination or to natural changes. Biennial increase in rotavirus activity has been reported in the postvaccine era. 55 Unimmunized susceptible children accumulate during seasons with low rotavirus activity and the higher number of susceptible individuals facilitates transmission during a subsequent season. 53 Temporal variability in rotavirus genotype distribution also occurs naturally, independent of vaccination. 4,10 Proper assessment of vaccination impact requires monitoring for longer periods and careful interpretation. 10 There is evidence of decreasing trends in pneumonia incidence and mortality in low-and middle-income countries from 2000 to 2010, attributed to economic and social developments, reduction in the prevalence of risk factors, expansion and improvement in case management and also the implementation of PCV and Haemophilus influenzae type b (Hib) childhood vaccination programs. 46 Pneumonia also has a seasonal pattern and the observation period must last at least a year to consider these variations. Two LAC studies adjusted for possible secular trends in pneumonia rates using nonrespiratory and diarrhea events as controls (Table 4). 1,7

Data sources
Health information system databases were the main data source for HIA of vaccination programs in LAC. Thirteen studies assessed the impact of rotavirus vaccination based on health information systems, mainly mortality and hospitalization data at national level (Table 2). 6,9,11,14,15,17,20,22,29,34,[43][44][45] One Brazilian study on the HIA of pneumococcal vaccination on hospitalizations used health information systems. 1 The identification of hospitalizations or deaths in health information system databases relied on International Classification of Diseases (ICD) codes in discharge summaries or death certificates. Health information system databases are increasingly being used in research and health assessment. Some advantages are the broad coverage, lower costs of data collection, easy access to data, and the possibility of longitudinal follow-up. Major disadvantages are lack of standardization in data collection, which affects the quality of the information, time and space variation in coverage, and lack of important information for the analysis. There may also be delays on database availability. Changes in coding practices during the studied period may also affect the analyses. 9 Failure to assign codes for gastroenteritis is a major reason for underestimating the burden of rotavirus disease and the impact of the vaccination program when using these data sources. 31 Underdiagnosing or underreporting may also be an issue when estimating burden of disease and HIA of the vaccination program, leading to the development of models that use international data when adequate local data are unavailable. 51 Important issues to be considered when using health information system databases are the completeness and reliability of the available data, coverage (proportion of population included), representativeness (whether persons included are similar to those not included), and sustainability (if the database is part of the health system and will be maintained long enough to monitor the effects of the program, independent of specific sponsoring). When using administrative databases, it is essential to consider the rules that govern the system and possible changes over time. 58 In general, mortality databases are more reliable than morbidity or health services utilization databases because death is a single event and data are less affected by administrative or economic conditions. On the other hand, it measures only the effect on severe disease, and changes in less severe conditions will not be identified.
Deaths occurring outside the health system, particularly in impoverished or rural areas, may not be registered in mortality systems. 6 Access to hospitalization information systems is increasing and they have been considered very useful as data sources in vaccination program impact analysis (Table 2). 9,15,17,44 A study in Goiania, Midwestern Brazil, used database linkage of secondary administrative hospitalization data and primary population-based surveillance data and found similar hospitalization rates for community-acquired pneumonia in children. 50 In many countries, burden of pneumococcal and rotavirus disease estimates are based on national sentinel-based surveillance data and the HIA of vaccination program relies on these data. Seven LAC studies evaluated the impact of rotavirus vaccination based on surveillance data (Table 2). 10,13,32,33,37,39,59 Information on the catchment population of the sentinel hospitals is unavailable for most sites, precluding incidence rates estimation, which constitutes a limitation. 32,59 Depending on the number of sentinel sites and their location, these data cannot be generalized for the entire population. 32,59 The World Health Organization has proposed a sentinel surveillance system for rotavirus and invasive bacterial diseases, a but some LAC countries established population-based surveillance for diarrhea with data collection for hospitalizations and outpatient visits at public health facilities. These population-based systems were used in HIA of rotavirus vaccination in Nicaragua and El Salvador, the latter in combination with sentinel hospital data. 37,59 Population-based surveillance data were also used in HIA of a PCV program in Uruguay. 25,26 The sensitivity of surveillance systems may change over time: variations in methods, case definitions, population under surveillance and reporting patterns may affect the results of before-after studies. In the era of PCV and rotavirus vaccines, most countries strengthened their surveillance systems to inform for decisions on vaccine policies. 54 Furthermore, vaccine introduction increases disease awareness, testing and reporting. 23 In LAC, invasive bacterial diseases surveillance was first organized as a laboratory-based surveillance system, the Sistema de Redes de Vigilancia de los Agentes Bacterianos Responsables de Neumonia y Meningitis (SIREVA II -Surveillance Network System for the Bacterial Agents Responsible for Pneumonia and Meningitis), created in 1993, initially in six countries (Argentina, Brazil, Chile, Colombia, Mexico, and Uruguay). 19 The SIREVA II Regional 2012 Report contains data on pneumococcal serotypes and antibiotic resistance from 19 Latin American countries and the Caribbean Epidemiology Center. b SIREVA II is a voluntary reporting system and its coverage varies a lot among countries, and caution is advised when using it to estimate invasive pneumococcal disease incidence and impact of PCV. In general, laboratory-based data lacks demographic and clinical information, which limits the analyses. 52 Furthermore, laboratory procedures to identify the pathogen may change over time. 3,28 Despite these limitations, SIREVA II is the best source of data on pneumococcal serotype distribution in the region and may allow assessing serotype replacement after vaccine introduction. Understanding serotype replacement is critical in low-and middle-income countries, where most deaths from pneumococcus occur, with greater diversity of serotypes causing disease and nasopharyngeal colonization early in infancy. 16 Reports of laboratory-confirmed rotavirus infections from clinical microbiology laboratories that constitute a national-or sentinel-laboratory surveillance system were also used. 10,13,33,39 These data allow assessing the impact of vaccination on rotavirus-confirmed diarrhea and genotype distribution.
Local secondary data including the hospital discharge summary database and medical records of a single hospital have been used as data sources for HIA of vaccination programs. 34,41,42 The major limitation of these data is that study results cannot be generalized to the entire population. 30,35 Primary data collection was conducted in HIA of PCV 50 and rotavirus 2,8,21,30,48 vaccination programs in LAC countries (Tables 3 and 4). Primary data collection may be particularly useful in settings where health information system databases are unavailable or unreliable and the surveillance system has not been appropriately implemented. Also, it can provide information unavailable on other data sources, such as rotavirus genotype circulation (Table 3). 2,48 Limitations of studies based on primary data include the small sample size collected in just one or few sites, precluding generalizing the results to the whole population. 2,8,30 Additionally, prospective design may be quite expensive, hampering the sustainability of the study and long-term HIA of the vaccination program.

Study outcome definition
Choosing syndrome of interest and hospitalization or mortality rates as outcomes. Only two LAC studies that used population-based surveillance data assessed the impact of rotavirus vaccination on outpatient care (number of healthcare visits). 37,59 The etiological diagnosis of rotavirus gastroenteritis requires laboratory tests, which are rarely performed in clinical practice since they do not alter the treatment. 17,59 Rotavirus testing is done at the discretion of the physician, based on institutional practices, which may change over time. 53 Although more specific and precise, using rotavirus-related diarrhea in studies based on secondary data may underestimate the true burden of disease and the impact of the vaccination program. Furthermore, "measuring impact on all-cause diarrhea may be more valuable to decision makers and the public health community because it provides an estimate of the preventable fraction of diarrhea deaths and admissions attributable to rotavirus". 9 Most of the eight HIA of PCV in LAC evaluated pneumonia, 1,7,25,26,35,41,42 two evaluated invasive pneumococcal disease 38,50 and one evaluated meningitis. 1 We did not identify any LAC study evaluating the impact of PCV on acute otitis media.
Diagnosing invasive pneumococcal diseases require laboratory tests. In some countries, such as the USA, blood cultures (BC) are performed in routine care for every child with fever without a focus in both hospital and outpatient care, whereas in others, such as most LAC countries, BC are limited to severely ill hospitalized children. BC practices may affect the burden of disease estimates (invasive pneumococcal disease incidence increases parallel to the number of BC samples in a population), the relative frequency of clinical syndromes (higher frequency of bacteremia without focus in countries with higher frequency of BC samples) and the serotype distribution. Previous use of antibiotics before sample collection also affects diagnostic sensitivity. 40,54 Changes in medical practices may also influence the results of before-after studies.
A study of invasive pneumococcal disease before and after the PCV7 program implementation in England and Wales evaluated control pathogens that also depend on blood culture practices and reporting, but for which there had been no public health intervention (Escherichia coli and non-pyogenic streptococci). 18 Similar trends (increasing rates) for invasive pneumococcal disease and the control pathogens suggested that the sensitivity of surveillance was increasing prior to vaccine introduction. Ignoring prevaccination trends could lead to underestimating the reduction in invasive pneumococcal disease and overestimating the degree of replacement disease. 18 Technical developments may increase the sensitivity of diagnostic tests. 54 Introduction of polymerase chain reaction (PCR) in the cerebral spinal fluid for diagnosing bacterial meningitis may increase the number of laboratory-confirmed pneumococcal meningitis cases. 47 This must be considered when analyzing changes in diagnostic test results throughout the period evaluated.
Pneumonia definition is a challenge. 46 Bacteremia occurs only in a small proportion of cases. The etiological diagnosis of non-bacteremic pneumonia by current tests is insufficiently sensitive and specific, and rarely performed in clinical practice. Due to difficulties to isolate the etiological agent, most studies focused their analyses in all-cause pneumonia. 1,25,26,35,41,42 Some of them also evaluated pneumococcal pneumonia (PP). 41,42 Definitions of pneumonia vary among studies. In general, studies based on secondary data used the diagnosis given by the attending physician, but diagnostic criteria vary among clinicians, health services and health information system databases. 35 Prospective cohorts used more standardized criteria, mainly radiologically-confirmed pneumonia. 25,26 Although the clinical diagnosis of acute otitis media does not require additional exams, and the collection of material to isolate pathogens is easier than for pneumonia, LAC countries lack high quality data on acute otitis media incidence and health resource use. 5 Generally, acute otitis media is treated in outpatient services, for which registered information is limited in LAC, hindering the HIA of pneumococcal vaccination on this disease.

Main results of the health impact assessment of rotavirus and pneumococcal vaccination programs in Latin America
Most studies showed decreased rates of diarrhea-related deaths, hospital admissions and healthcare visits after rotavirus vaccination implementation (Tables 2, 3 and 4). Blunting or delay of seasonal peaks of diarrheal disease after vaccine introduction has also been reported. 15,17,32,48 Two studies in Mexico and another in Brazil assessed the impact of rotavirus vaccination on diarrhea mortality or hospitalization rates according to the socioeconomic level or human development index of the region. 15,17,20 The Mexican studies observed comparable reduction in diarrhea-related deaths and hospitalization in all areas, whereas the Brazilian study showed great reduction in hospitalization rates of under-five children in the least developed areas. 15,17,20 All eight HIA of pneumococcal vaccination programs in LAC showed reduction in the events of interest, mainly hospitalization, after PCV introduction (Table 4). Nonvaccine serotypes increased in Colombia and Uruguay after vaccine introduction. 38,41,42 Table 5 presents a summary of advantages and limitations of study design, data sources, and outcomes used in HIA of vaccination programs.
Despite the strategies to access grey literature, country reports and other local documents may not be included in this review. The classification of epidemiological study designs was heterogeneous and we used the authors' classification. Furthermore, some studies lack methodological information. These two limitations may affect our analysis and synthesis of knowledge production on HIA of PCV and rotavirus vaccination programs.
The challenges in conducting HIA of vaccination programs are easier to face in countries with reliable and sustainable health information systems and surveillance data as well as expertise in health evaluation. However, LAC countries have managed to do a lot in HIA of pneumococcal and rotavirus vaccination programs in a relatively short time after program implementation. Almost all met basic methodological requirements for HIA and More reliable than morbidity data Measures only results of severe clinical syndromes; thus, changes in less severe conditions will not be identified. Difficulty in discriminating effects of changes in the incidence or treatment of conditions suggested a positive health impact. High-quality studies have been conducted in small countries without tradition in research that have prioritized surveillance and registers. Future HIA of vaccination programs should consider the methodological issues and challenges that arose in these first studies conducted in the region as well as in studies from other regions. HIA of vaccination programs should be considered essential in the planning phase of vaccine introduction, with the definition of outcomes, data sources, and responsibilities for data collection and resources. They can also contribute to the validation and methodological development of vaccine cost-effectiveness studies.

AUTHORS' CONTRIBUTIONS
AMCS, AFN, PCS and HMDN were responsible by the conception and design of the study. AMCS, AFN and TYY conducted data collection, analysis and interpretation. AMCS drafted the article. AFN, PCS, TYY and HMDN critically reviewed the paper contents, and all authors approved the submitted version.