Methods to evaluate COVID-19 vaccine effectiveness, with an emphasis on quasi-experimental approaches

Abstract

The evaluation of vaccine effectiveness is conducted with real-world data. They are essential to monitor the performance of vaccination programmes over time, and in the context of the emergence of new variants. Until now, the effectiveness of COVID-19 vaccines has been assessed based on classic methods, such as cohort and test-negative case-control studies, which may often not allow for adequate control of inherent biases in the assignment of vaccination campaigns. The aim of this review was to discuss the study designs available to evaluate vaccine effectiveness, highlighting quasi-experimental studies, which seek to mimic randomized trials, by introducing an exogenous component to allocate to treatment, in addition to the advantages, limitations, and applicability in the context of Brazilian data. The use of quasi-experimental approaches, such as interrupted time series, difference-in-differences, propensity scores, instrumental variables, and regression discontinuity design, are relevant due to the possibility of providing more accurate estimates of COVID-19 vaccine effectiveness. This is especially important in scenarios such as the Brazilian, which characterized by the use of various vaccines, with the respective numbers and intervals between doses, applied to different age groups, and introduced at different times during the pandemic.

Key words:
Coronavirus; Causal inference; Immunization

Resumo

A avalição da efetividade de vacinas é feita com dados do mundo real e é essencial para monitorar o desempenho dos programas de vacinação ao longo do tempo bem como frente a novas variantes. Até o momento, a avaliação da efetividade das vacinas para COVID-19 tem sido baseada em métodos clássicos como estudos de coorte e caso controle teste-negativo, que muitas vezes podem não permitir o adequado controle dos vieses intrínsecos da alocação das campanhas de vacinação. O objetivo dessa revisão foi discutir os desenhos de estudo disponíveis para avaliação de efetividade das vacinas, enfatizando os estudos quase-experimentais, que buscam mimetizar os estudos aleatorizados ao introduzir um componente exógeno para atribuição ao tratamento, bem como suas vantagens, limitações e aplicabilidade no contexto dos dados brasileiros. O emprego de métodos quase-experimentais, incluindo as séries temporais interrompidas, o método de diferença em diferenças, escore de propensão, variáveis instrumentais e regressão descontínua, são relevantes pela possibilidade de gerar estimativas mais acuradas da efetividade de vacinas para COVID-19 em cenários como o brasileiro, que se caracteriza pelo uso de várias vacinas, com respectivos número e intervalos entre doses, aplicadas em diferentes faixas etárias e em diferentes momentos da pandemia.

Palavras-chave:
Coronavírus; Inferência causal; Imunização

Introduction

During development and licensing, vaccines go through a series of stages to evaluate their safety and efficacy. The use of a new vaccine becomes possible following its approval by regulatory bodies. The regulatory decisions required to approve their use in the population focus on the balance between the risk and benefit, expressed in safety and efficacy measures. However, they do not capture all of the information required to guarantee their continued use in public health¹.

Efficacy data derives from phase III pre-licensing studies, usually randomized clinical trials, and provide the measure of a proportional reduction of the risk of infection, or disease, in the vaccinated group, compared to one that has usually received a placebo. Although randomized clinical trials have a strong internal validity, and provide robust evidence of the direct biological effects of the vaccine on an individual level, they cannot be generalized to a population-level vaccination programme².

From the time when the vaccine starts to be used in the population, its effectiveness needs to be evaluated from two main objectives: verifying if the levels of efficacy registered in the phase III studies were maintained in the real world, and continuing to monitor its safety, which involves several thousand people and, therefore, may enable rare, but severe, events to be confirmed. Thus, “efficacy” is usually defined as the performance of an intervention in ideal, controlled circumstances, while “effectiveness” refers to its performance under conditions of use in the real world³.

In the case of COVID-19, we have a scenario in which various highly effective vaccines were developed in a short space of time, and have been applied to millions of people. On the other hand, the virus has been demonstrating the capacity to mutate quickly, which has affected some of its characteristics, such as transmissibility. The possibility of mutations that affect the vaccine`s capacity for protection is possible, although this has not been significantly documented. A further important issue is the duration of the effect of the vaccine, i.e., the effectiveness may reduce in just a few months⁴. In addition, there is the fact that the various vaccines are technologically different, having varying effects on the immunological system and, consequently, their efficacy⁵. Lastly, the lack of clear correlates of protection for COVID-19 should be highlighted (i.e., usually measurable antibodies which may serve markers of individual protection), although if found, may be different for specific vaccines.⁶ These immunological correlates of protection enable individuals, or the fraction of the population which is adequately protected, to be quickly identified through the use of surveys on population samples, with biological sample tests. In their absence, or complementary to these, we have effectiveness studies, which are population-based studies, either purely observational or quasi-experimental, and aim to estimate the effect of the vaccine (effectiveness) during its use in the real world.

In this context, vaccine effectiveness (VE) studies should include appropriate methods to mimic the process of allocating intervention, seeking the most adequate balance between those who did, or did not, receive the intervention. The methods may be adaptations of classic observational studies (i.e., time-series, case-control and cohorts), or those called quasi-experimental. The quasi-experimental methods, also called `natural experiments`, are resources which are less used and disseminated, but that have a great potential of providing adequate and efficient solutions to solve the problem of imbalance on allocating the intervention among treated and control groups, and producing valid VE estimates.

Therefore, this manuscript has the aim of providing an overview of the methods used when assessing VE, with a greater emphasis on quasi-experimental studies. Also, we aim to emphazise the ecosystem of existing and accessible epidemiological data, to discuss the potential and limitations for conducting VE studies in Brazil.

Methods to evaluate vaccine efficacy and effectiveness

Halloran et al.³ define four types of effects which may be considered in studies to evaluate vaccine effectiveness. An examination of these effects involves considering not only administration of the vaccine to individuals, but also the context of vaccination programmes, and the different levels of vaccination coverage in populations in particular. Schematically, considering two extreme alternatives, in which in one there is a population with a vaccination programme under development and, in the other there is no vaccination programme at population-level, the following effects of the vaccines may be defined:

• (1) the direct effect is obtained by comparing measures of the occurrence of the disease (e.g., incidence of the infection or disease, hospitalisation or death) in vaccinated and unvaccinated individuals in the same population;

• (2) the indirect effect is obtained by comparing measures of the occurrence of the disease in those who were unvaccinated in the population in which there is a vaccination programme, and the unvaccinated, or a population where there is no vaccination programme;

• (3) In the total effect, occurrence of the event among those vaccinated in the population in which there is a vaccination programme are compared with the incidence in those unvaccinated in the population where there is no vaccination programme; and

• (4) the overall, or global effect, is estimated by comparing measures of the disease occurring in populations both with and without a vaccination programme, or, generically, with varying levels of vaccination coverage. The overall effect examines the impact on the level of the population, considering both the direct effect of the vaccination on individuals who have been vaccinated, and the indirect effects arising from the reduction in the levels of community transmission due to group immunity (collective or herd) - this is a measure of the global benefit of vaccination for public health.

Randomized clinical, cohort, and case-control studies are mainly used to examine the direct effects of a vaccine, while community-based intervention, or cluster randomized trials, can be used to evaluate the overall, total, or indirect effects. More recently, quasi-experimental methods have been used to evaluate the effects of vaccination programmes. They may also be used to evaluate indirect and total effects if there is adequate data on the vaccination status of individuals within a cohort of those eligible for a certain programe⁷.

Observational epidemiological methods and their application in the evaluation of COVID-19 vaccine effectiveness

Case-control (test-negative) studies

In the last twenty years, a modification in the traditional case-control study design, called the “test-negative case-control” has been widely used in observational studies to evaluate the post-licensing effectiveness of influenza vaccines^8,9. More recently, the World Health Organization (WHO) proposed application of this study to evaluate COVID-19 vaccine effectiveness¹⁰.

In COVID-19 studies, a case is defined as a patient who seeks health care and tests positive for SARS-CoV-2 infection through a reverse transcription polymerase chain reaction (RT-PCR) and/or antigen test. The control is a patient who followed the same case process, but receives a negative result for SARS-CoV-2 infection. VE is calculated in a similar way to traditional case-control studies; in other words, VE = (1 - the ratio of chance of vaccination among cases and controls) × 100%^4,11.

Ranzani et al.¹¹ conducted a test-negative case-control study in the elderly aged ≥70 from the State of São Paulo who received the CoronaVac vaccine. Vaccine effectiveness for symptomatic infection, adjusted for age and comorbidities, was 18.2% (CI 95% 0.0 to 33.2) and 41.6% (CI 95% 26.9 to 53.3) in a 0 to 13 and ≥14 day period, respectively, following the 2^nd dose¹¹. Hitchings et al. (2021)⁴ also evaluated the effectiveness of the CoronaVac in health professionals in Manaus, State of Amazonas, Brazil. Vaccination with a minimum of one dose presented effectiveness, adjusted by other variables, of 49.6% (CI 95% 11.3 to 71.4) for symptomatic SARS-CoV-2 infection in the ≥14 day period following the first dose. However, low effectiveness was confirmed (36.8%; CI 95% 54.9 to 74.2) in the ≥14 day period following the second dose⁴. In the United Kingdom, Bernal et al.¹² evaluated the effectiveness of Comirnaty and Vaxzevria vaccines with a test-negative case-control study. The results were similar for both vaccines, with over 70% effectiveness for the Alpha, and over 65% for the Delta variant¹².

Cohort studies

Cohort studies have characteristics similar to those of clinical trials, except that they do not involve manipulating interventions. The longitudinal structure allows observation of the time sequence of events, and outcome exposure, which facilitates the causal inference process, and direct calculation of incidence and mortality measures.

A prospective cohort study was conducted on a national level in Chile to evaluate CoronaVac vaccine effectiveness, including approximately 80% of the population.¹³ Using an extension of the Cox proportional-hazards model, considering vaccination status as a dependent time variable, vaccination effectiveness was estimated, associated with partial immunization (≥14 days following receipt of the first dose, and before receiving the second dose), and full immunization (≥14 days following receipt of the second dose). Among the fully immunized people, the adjusted effectiveness of the vaccine was 65.9% for SARS-CoV-2 infection, 87.5% for hospitalisation, 90.3% for ICU admission, and 86.3% for death. The results were maintained in the subgroup analyses by age, mainly among people of an age equal or superior to 60¹³.

The SIREN study is based on a prospective multicentric cohort of public hospital employees in the United Kingdom, immunized with the BNT162b2 vaccine (Comirnaty)¹⁴. Risk factors, vaccination status, and symptoms, were registered at two week intervals, in addition to all of the RT-PCR and SARS-CoV-2 antibody test results. A Poisson mixed-effects proportional hazard model was used to calculate hazard ratios, to compare the time to infection in unvaccinated and vaccinated participants, and thereby estimate the impact of the vaccine on symptomatic and asymptomatic infections. Vaccination coverage in the period studied was 89%. Significantly lower coverage was associated with previous infection (OR 0.59; CI 95% 0.54-0.64), women (OR 0.72, CI 95% 0.63-0.82), those aged under 35, belonging to minority ethnic groups (especially black people) (OR 0.26, CI 95% 0.21-0.32), porters/security guards (OR 0.61, CI 95% 0.42-0.90), or midwives (OR 0.74, CI 95% 0.57-0.97), and employees living in more vulnerable neighbourhoods (OR 0.75, CI 95% 0.65-0.87)¹⁴. Vaccine effectiveness was 72% (CI 95% 58-86) 21 days following the first dose, and 86% (CI 95% 76-97) seven days after two doses¹⁴.

A prospective cohort study conducted in Scotland¹⁵, with a hospitalisation and mortality database of 5.4 million people, estimated the effectiveness of the first doses of Pfizer-BioNTech (Cominarty) and Oxford-AstraZeneca (Vaxzevria) vaccines against COVID-19 related hospital admissions. The time-dependent Cox model and Poisson regression models were used for this. The first dose of the BNT162b2 (Cominarty) vaccine was associated with 85% (CI 95%; 76-91) vaccination effectiveness to prevent COVID-19-related hospitalisation, 28-34 days after vaccination¹⁵. Vaccine effectiveness in the same time interval for the ChAdOx1 (Vaxzevria) vaccine was 94% (CI 95%; 73-99). The results of combined vaccine effectiveness to prevent COVID-19-related hospitalisation were comparable, by restricting the analysis to those at an age equal to or above 80 (81%; CI 95% 65-90), 28-34 days following vaccination)¹⁵.

A longitudinal study was conducted in Brazil, to evaluate the effectiveness of CoronaVac and Vaxzevria in four different outcomes: SARS-CoV-2 virus infection, hospitalisation, ICU admission, and death. Due to the lack of data on the unvaccinated population, the period between the date of the first dose and the 13^th day was used as a reference to estimate VE. Full vaccination (14 days after the 2^nd dose) with Vaxzevria or CoronaVac displayed 78% and 53% vaccination effectiveness against SARS-CoV-2 infection, respectively. VE against hospitalisation, ICU admission, and death, was 91.4% (CI 95% 90.1-92.5), 91.1 (CI 95% 88.9-92.9) and 92.3% (CI 95% 90.5-93.7) respectively for Vaxzevria, and 71.2% (70.0-72.4), 72.2% (70.2-74.0) and 73.7% (72.1-75.2) for CoronaVac. The lower protection found for older adults is also highlighted, especially those over the age of 80, particularly with CoronaVac⁵.

Quasi-experimental methods and their applications in evaluating vaccine effectiveness

Observational studies (such as cohort, or case-control studies), which compare individuals who have been vaccinated with those who have not, can be used to evaluate the direct effect of the vaccines. However, individuals who have been vaccinated may systematically differ from those who have not, and it is often difficult, or impossible, to separate the effects of these differences from those related to the vaccine. On the other hand, quasi-experimental studies, while they are also fundamentally observational, since they stem from records of interventions conducted outside of the researcher`s control, however seek to mimic experimental studies by exogenous attribution to eligibility of a treatment, in this case, the vaccine, thereby avoiding endogenous sources of bias¹⁶.

There is a wide range of scenarios in which quasi-experimental methods can be used to evaluate vaccines¹⁶. Firstly, when a vaccine has already been implemented, and, therefore, the use of randomized trials is no longer a viable option, or in circumstances where it would be unethical not to provide a vaccine to a specific group (control), such as during a pandemic. Among the quasi-experimental methods, we can cite the following: interrupted time series, difference-in-differences, propensity scores, instrumental variables, and regression discontinuity.

Interrupted time series

Interrupted Time Series (ITS) are indicated to evaluate a large-scale intervention applied at population-level, and with a well-defined start date,^17,18 where multiple pre- and post-intervention observations are used to examine a change in the tendency of the outcome following the intervention. ITS studies have been increasingly frequent to evaluate the effect of public health interventions, highlighting the impact of rotavirus¹⁹ and pneumonia²⁰ vaccination programmes.

ITS studies can be constructed using two approaches: single and multiple groups. In single group studies, there is no comparison group, and the effect can be estimated from the change in the pre- and post-intervention tendency, so that the pre-intervention tendency is counterfactual. However, in multiple group studies, there is at least one comparison group, and the change in the intergroup outcome tendency (i.e., in the pre- and post-intervention period), and between the groups (comparison and control group) can be evaluated^17,21.

A minimum of three variables is required to analyse ITS regression: the time (t) elapsed since the start of the study with the unit that represents the frequency with which the observations are made; the variable that indicates the pre-intervention (Xt) period; and the outcome in each period of time (Yt)^18,21.

In order to prepare an interrupted time series, a number of assumptions should be fulfilled. Initially, that there is a clear differentiation between the pre- and post-intervention period needs to be confirmed, requiring a timeframe for the start of the intervention. These studies require a sequential measure of the outcomes before and after the intervention, they can be binary, continuous, or counts, and are indicated when the change occurs in a short space of time. In addition, sequential measures of the outcome over time should be similarly distributed both before and after the intervention¹⁸.

ITS has already been used in Brazil to evaluate rotavirus²² and pneumococcal conjugate (PCV10)²³ vaccine effectiveness. However, no data is available on the use of ITS in COVID-19 vaccine effectiveness until this time.

ITS studies are useful when randomization is impossible, allowing a large-scale longitudinal evaluation. In addition, the possibility of working with aggregated population data contributes towards greater external validity¹⁶. Quick and convenient access to routinely collected health data on database, and use of ITS, enables the production of information on vaccination effectiveness in real time¹⁸. A further important point to be raised is the presentation of graphic and numerical results, which are easily understood by health professionals and managers, and may assist with decision-making, and the reallocation of resources.

On the other hand, there are limitations. The analysis may be subject to residual confounders, due to the unavailability of important covariables not registered on the databases used. Variables which change quickly in time need to be identified, and treated through multivariate models, since they may bias the association found¹⁸. Seasonality and the unequal distribution of units of time before and after the intervention are also factors that need to be taken into consideration¹⁸.

Difference-in-differences

The difference-in-differences (DD) method has been used to evaluate the impact, considering interventions that vary in time^24-26, due to its relative convenience, and wide use in evaluations of public health interventions²⁷. In order to measure the effect, DD considers that the treatment group estimate (counterfactual) is equal to the pre-intervention value of the treatment group, added to the control group post/pre-difference²⁸. Although DD identifies the average effect in those treated, its meaning and identification conditions differ between data types²⁸.

Use of DD depends on supporting premises. Firstly, the value of a stable treatment unit and premises of “parallel tendencies”^28-30; in other words, there should not be a spillover effect among the treatment and control groups, since the treatment effect will not be identified. Additionally, the control variables at an individual and/or aggregated level should be exogenous, and unaffected by the treatment. A typical approach is the use of covariables prior to the intervention^28,29.

Raymond et al.²⁶ used DD methodology to evaluate the effect of the provision of human papillomavirus (HPV) vaccination coverage in children, on infection by the virus in young women. The effect was estimated using linear probability regressions, adjusted for race/ethnicity, age, income, head of the family`s level of education, and family employment²⁶. To the best of our knowledge, no reports on the use of DD to evaluate COVID-19 vaccine effectiveness are available.

All of the DD analyses should carefully consider any possible violations of assumptions, many of which appear to be probable, due to the COVID-19 dynamic, such as record quality, the non- linearity that arises from person-to-person transmission, and the probability that control policies at population-level have different effects over time³¹. By including control groups, DD provides important advantages over methods such as before and after ITS comparisons.³¹ In addition, the graphic and parametric tools developed for DD in recent years allow an evaluation of the plausibility of assumptions.

Propensity score

The propensity score (PS) is the probability of allocating treatment, conditional to the confounding variables observed. This score could be used to control confounding, making treatment groups more similar in relation to the confounding variables observed³². In VE evaluation studies, the confounding variables that require control are, for example, age, sex, occupational profile, health conditions, confidence, and availability of the vaccine in specific contexts, among other aspects. Thus, this information may be used to estimate the probability of an individual being vaccinated.

The PS is commonly estimated using logistics regression, being the status of treatment predicted by the set of confounding variables observed³². The selection of independent variables that will be included in the PS estimation model is an important step in the use of this methodology. It is recommended that both exposure-related variables and the outcome being investigated (confounding variables) are included^32,33.

Following score estimation, they can be used in the analysis to control the confounding variables observed in four main forms: (i) model adjustment, where the PS is included as an adjustment variable in a regression model that relates the outcome and exposure of interest; (ii) analysis stratification, where the PS is used to divide the study population into groups, or lower, more similar strata in relation to the confounding variables observed; (iii) participant matching, which involves matching an exposed with an unexposed individual who has a similar PS; and (iv) model weighting, where PSs are used to calculate the statistical weights for each individual, so that when considering the study population, the exposed and unexposed groups become similar in relation to the confounding variables observed, allowing an impartial estimate of the relation between the exposure and outcome.^32,33 The last two have been highlighted in literature in recent years, especially in studies that evaluate VE.

The main assumptions to estimate the causal effect in PS based methods include (i) ignorability, which means that no confounders are observed³⁴; (ii) positivity, which means that every individual should have a probability other than zero to receive any treatment³⁴; (iii) the correct specification of the model used to calculate the PS, considering that its calculation is based on measured characteristics (variables). If unmeasured factors influence the selection of treatment, the resultant PS will not remove all of the bias from the confounding variables. In some cases, the residual bias could remain, and incorrect specifications could increase the bias.³⁴

The use of PS has been widely implemented in observational studies, particularly to evaluate the effectiveness of various medicines, highlighting VE studies for influenza^35-37 and rotavirus.³⁸ Therefore, use of PS could be an important methodological alternative to evaluate COVID-19 vaccine effectiveness. However, since the method controls exclusively for the confounding variables observed, it requires a significant number of covariables that are associated both with the vaccination (e.g., distribution, vaccination coverage, prioritization criteria - such as age, comorbidities, profession, and vulnerable populations, among others), for the outcome of interest, whether infection, hospitalisation, or death by COVID-19.

Until this time, the use of PS has only been identified in one study, which paired individuals receiving a minimum of one dose of any of the vaccines with unvaccinated individuals, according to demographic data, location (post code), and the number of previous PCR SARS-CoV-2 tests.³⁹ The authors highlighted that the administration of two doses of the COVID-19 vaccine (mRNA-1273 [Moderna], or BNT162b2 [Pfizer/BioNTech]), was 88.7% effective in preventing SARS-CoV-2 (CI 95%: 68.4-97.1%) infection. In addition, the administration of a minimum of one dose of the vaccines reduced hospital admission rates by 14 days for patients who were subsequently diagnosed with COVID-19 following immunization, in relation to those who were not vaccinated (3.7% vs. 9.2%; relative risk: 0.4; p-value: 0.007).³⁹ Therefore, it is emphasised that evaluation of the effectiveness of COVID-19 vaccines using the PS method is limited in literature.

Instrumental variables

Instrumental variables have been used increasingly as a strategy to control confounding in non-randomized study designs. Unlike the propensity score, instrumental variables use an exogenous form of variation, or “instrument”. The instrument should: (1) be the cause or “proxy” for exposure; (2) only be outcome related through exposure; and (3) not be associated with any confounding variable not measured in the study⁴⁰.

In instrumental variable analysis, the effect of the intervention on the outcome of interest is measured by comparing the magnitude of the association between the instrument and outcome with that between the instrument and intervention exposure⁴⁰. A sequence of two regression analyses is generally used, which depends on the nature of the exposure and the outcome. Selection of an instrument is a fundamental part of the design, and is not simple. Even if the three above assumptions are met, the association between the instrument and intervention exposure should be strong. Inadequate instruments which establish a weak association with the intervention may produce inflated estimates, with broad confidence intervals⁴⁰.

The use of instrumental variables has not been widely used in the context of evaluating VE. We highlight the study by Wong et al.⁴¹, which evaluated the effectiveness of the influenza vaccine in the hospitalisation and death of elderly patients, using influenza vaccination coverage among people aged 65 or over as the instrument. A study by Chakrabarti et al.⁴², also evaluated the effect of supplementary immunization activities in routine health service vaccination coverage. The authors evaluated if the child had obtained the main routine vaccination, or not, using the child`s age in the first supplementary immunization campaign as the instrument.⁴² Use of methodologies using instrumental variables to evaluate COVID-19 vaccination effectiveness has not been documented in literature until this time.

When applied to evaluating COVID-19 vaccine effectiveness as instrumental variables, they could have great analytical power, given the difficulty in obtaining individualized data that contains clinical information and comorbidities, strongly associated with hospitalisation and mortality caused by the disease. Exogenous sources of variation, which could be used as instruments include age, the location of residence (municipality or state), and vaccination coverage, as conducted in the study by Wang et al.⁴¹.

The advantage of this method is due to the fact that the selection of a good instrument allows a non-biased estimate of association between intervention and outcome, without the need to measure all the confounding variables. This becomes crucial when conducting studies with secondary bases, where the data has already been collected. However, the main methodological limitation related to the instrumental variable approach, is the difficulty in finding good instruments, which may reduce the capacity to detect small effects, even in studies with large databases⁴³.

Regression discontinuity design

Regression discontinuity design (RDD) has been used increasingly in research in the area of epidemiology and public health.^44-47 It allows the effect of an intervention to be measured when the eligibility rules are based on a previously defined threshold (cut-off point). In other words, in regression discontinuity exposure to a treatment, or intervention, is determined by a continuous variable, such as the income or age of an individual, called the attribute variable, and the cut-off point for this variable defines intervention eligibility (or treatment). Generally speaking, RDD assumes that the intervention attribute (or treatment) in a small neighbourhood around the cut-off point is ignorable, and the potential response could be assumed independent of the attribute, which occurs in a randomized study.

Regression discontinuity design can be described in a deterministic or probabilistic way. For example, if all of the individuals above the cut-off point receive the vaccine, and nobody receives the vaccine under the cut-off point, then regression discontinuity is considered deterministic, known as sharp regression discontinuity. In contrast, if the probability of receiving the vaccine on one side of the cut-off point is greater in relation to the other, the design is said to be probabilistic, known as fuzzy regression discontinuity. Thus, in sharp RDD the intervention (or treatment) attribute rule perfectly determines the exposure, while in fuzzy RDD, the attribute rule causes a discontinuous change in the probability of exposure around the cut-off point, and this discontinuity is used to estimate the local causal effect of the change on the policy, or intervention, among the individuals who are around the cut-off point^46,48.

RDD has assumptions which can make its use inadequate, if they cannot be verified. The assumptions associated with discontinuous regression design are: 1) the existence of discontinuity in the probability of exposure; 2) non-manipulation of attribute variable values; 3) interchangeability; and 4) continuity in the probability of the response at the cut-off point. The first assumption evaluates if there is a discontinuous change in the probability of exposure around the cut-off point. The assumption of non-manipulation of the attribute variable suggests that the individuals in the study did not alter the information on this variable. Manipulation of this information would violate the assumption that the groups are attributed to the intervention (or treatment), which occurs in randomized studies. The assumption of interchangeability suggests that the exposure groups could be interchanged around the cut-off point. If this assumption is valid, it is expected that the individuals around the cut-off point are similar in relation to the distribution of all the baseline variables. The last assumption of RDD could be seen as an extension of the assumption of interchangeability, and suggests that any discontinuity in the probability of the response could solely be attributed to exposure. We highlight that these assumptions are considered weak, and can be confirmed through graphic analyses, or statistical tests^46,47.

For example, in studies that evaluate VE, the criteria (or eligibility rule) which makes an individual eligible to receive the vaccine, is usually age. Thus, individuals who are over a previously-defined age limit, are considered eligible for the intervention. Therefore, in this type of study, age would be our attribute variable. Use of discontinuous regression in studies on VE is still in the early stages, but we can cite some recent work which made use of this type of study design. For example, Frio and França⁴⁹ used fuzzy RDD to evaluate if the HPV vaccination affected the start of a sex life in girls in the age range close to the cut-off point (14 years old), which is defined by the public vaccination campaign⁴⁹. The authors also investigated if the teenagers who had already started to have a sex life, stopped using condoms since they had been vaccinated⁴⁹. The studies by Van Ourti and Bouckaert (2020)⁵⁰ and Anderson et al.⁵¹, used RDD to evaluate vaccine effectiveness against the influenza virus. In Van Ourti and Bouckaert`s⁵⁰ study, the authors estimated the impact of the Dutch vaccination programme on the use of medication, outpatient consultations, hospitalisation and mortality at the age of 65 (cut-off point), and concluded that there was an increase in the vaccination rate, but they did not find a relation with a possible reduction in hospitalisation, or mortality rates in the population analysed. In the study by Anderson et al.⁵¹, the authors evaluated vaccine effectiveness in the reduction of hospitalisations and mortality among older adults in the 65 year old age range (cut-off point), and concluded that there was an increase in vaccination rates, but they did not find a relation with a possible reduction in hospitalisation, or mortality rates, in the population analysed.

Application of RDD to evaluate the effectiveness of COVID-19 has been discussed⁵², but its application is still in the early stages^53,54. RDD was used in 2020 to evaluate if the BCG vaccination, applied to protect against tuberculosis, could reduce COVID-19 infection.⁵⁵ However, an association between greater BCG vaccination coverage, and a lower chance of age-specific COVID-19 infection, was not found using data in the five countries applying this design⁵⁵. In England, effectiveness of the first dose of the COVID-19 vaccine was evaluated using age, defined as the priority criterion for the vaccine in this country, as a treatment attribute variable⁵³. Another study evaluated the effectiveness of the COVID-19 vaccination in hospitalisation rates for the disease in New York, also applying the age criterion as an attribute variable⁵⁴. However, we highlight that the rather broad age ranges may violate RDD assumptions, and should be verified.

Vaccines in use and with potential use in Brazil

The COVID-19 vaccines developed until this time (using data updated on 8th September, 2021) are based on four main technologies: 1) inactivated SARS-CoV-2 virus vaccines (CoronaVac); 2) recombinant vaccines which use adenovirus viral vectors, expressing the SARS-CoV-2 Spike protein (S) (the Janssen vaccine uses a human adenovirus, and Vaxzevria uses a chimpanzee adenovirus); 3) messenger RNA (mRNA) vaccines, which codify the SARS-CoV-2 (Comirnaty) S protein; and 4) protein subunit vaccines, which use SARS-CoV-2 S protein nanoparticles, or fragments of this protein (Novavax)⁵⁶.

Chart 1 presents characteristics of the four vaccines authorised by the National Sanitary Surveillance Agency (Anvisa) for emergency or definitive use in the country until September 2021. They are, Vaxzevria (ChAdOx1-S; Oxford-AstraZeneca-Fiocruz, produced in partnership with the Serum Institute of India), CoronaVac (Butantan/Sinovac Biotech), Comirnaty (Pfizer/BioNTech/Wyeth), and Janssen-Cilag (Johnson & Johnson). They are all indicated for people aged 18 or over, except for Comirnaty, which is authorised for those over the age of 12, and have a two dose vaccination scheme, except for Janssen, which is applied in a single dose. The administration of Vaxzevria was temporarily interrupted for pregnant women and those who have recently given birth.

Chart 1
Characteristics of vaccines in use in Brazil in September 2021.

Vacine	Doses applied (%)	Interval between doses (weeks)	Seroconversion (%)	Efficacy (% of protection)	Overall effectiveness
CoronaVac	68,697,598 (33.8)	2 to 4	92 (14 days) and 97 with 28 days	77.96% (for symptomatic cases with outpatient or hospital care)	N/A
Vaxzevria	91,054,329 (44.8)	Up to 12	>98 following the 1st and >99 following the 2nd dose	73.43% in the general population and among people with comorbidities	N/A
Comirnaty	38,820,038 (19.1)	12	N/A	92.6% following the 1st, and 95.0% following the 2nd dose	Health workers = 80% 1st dose, and 90% 2nd dose The elderly >70 years old = 80% (reduction in hospitalisation) and 85% (reduction in deaths) General population = 97% (symptomatic cases, need for hospitalisation and death)
Janssen	4,674,271 (2.3)	Single dose	N/A	66.9% after 14 days, and 66.1% after 28 days. Efficacy was 76.7% after 14 days, and 85.4% after 28 days in the prevention of serious cases.64	NA

• Sources: National Plan to Operationalise COVID-19 Vaccination56^,64.

Until now, the simultaneous administration of COVID-19 vaccines with others on the national immunization calendar is not recommended, to improve monitoring of adverse events following vaccination, and due to a lack of knowledge on the possibility of antigenic competition.

The administration of additional doses, or interchangeability between different vaccines has been discussed, and may be recommended in specific situations (e.g., pregnant women and the immunosuppressed). Although attractive as a public health policy, a combination of vaccines awaits proof of efficacy and effectiveness. The Center for Disease Control Prevention (CDC/EUA) and Public Health England only advocate this in exceptional situations, or if there is a lack of vaccines. In Brazil, based on the possible association of viral vector vaccines with rare thrombotic events⁵⁷, a combination is indicated for pregnant women who receive a first dose of Vaxzevria, with the second dose of Comirnaty or CoronaVac. For the same reason, a number of European countries, and Canada, have recommended the second dose with mRNA vaccines for people under the age of 55 who received their first dose with Vaxzevria^58,59. Considering the change in guidelines in these countries, data on combined VE will be available in the near future.

Existing and accessible data in Brazil - Limitations and possibilities

In a context of increasing vaccination, there is a growing need for rapid answers on crucial questions, in order to orientate vaccination programmes. Among these, we highlight (1) the effect of new variants of the virus on vaccine effectiveness; (2) the optimum interval between doses; (3) the effect of the vaccines on asymptomatic infection, against serious disease; (4) the decline in immunity over time, and (5) the need, or usefulness, of combining different vaccines, or booster doses. In this context, the use of data originating from large databases of data routinely collected by the health system, associated with appropriate methods of analysis, has been one of the solutions presented in various countries. For any chosen investigation, both classic observational studies, and the use of quasi-experimental designs, require high quality information, whether related to the vaccination status of each individual, health outcomes, or the covariables required.

At the start of the pandemic, Scotland built a large integrated data system⁶⁰, which has been widely used for epidemiological and COVID-19 clinical studies and is currently employed to evaluate BNT162b2 vaccine effectiveness⁶¹. In Israel, a follow-up study used the databases of four health organizations, which cover more than half of the population, to combine health history information, RT-PCR test results, outpatient, hospitalisation and vaccination data, to estimate the effectiveness of the first and second doses of the BNT162b2 vaccine at protecting from infection, symptomatic COVID-19, hospitalisation, serious disease, and death.⁶² Outside of the circle of developed countries, Chile has presented a major study on the effectiveness of the main vaccine in use in the country (CoronaVac), also utilising data which is routinely collected on national databases¹³. In Brazil, two effectiveness studies based on routine data were conducted using the case-control approach. The first, of health professionals from Manaus⁴, and the other with the elderly population in the state of São Paulo¹¹, both evaluating the effectiveness of CoronaVac, or CoronaVac and Vaxzevria, throughout the country⁵.

In Brazil, a series of databases register and provide complete or partial data that is potentially useful to study COVID-19 vaccine effectiveness. Of specific interest are databases with individualized, identified, or pseudonymised data, allowing analyses that bring together different markers for exposure and outcomes, and relevant covariables. Unfortunately, we do not have a unified system with all of the relevant, required information, since it is dispersed throughout different records.

We will now provide a brief description of the databases available until this time. In the first group are databases which were created, or extended, to register COVID-19 related events, while the others are for regular use, and include important information for COVID-19 related studies. They all provide dictionaries and open data on the OpenDataSUS site.

Databases produced for COVID-19 related events

• a) National COVID-19 Vaccination Campaign: Contains demographic (age, sex, race/skin colour, and place of residence), and vaccination data (date the vaccine doses were administered, type of vaccine, batch, and place vaccine was administered) (Available at: https://opendatasus.saude.gov.br/dataset/COVID-19-vacinacao).

• b) Notifications of Influenza-like Illness: The data originates from the e-SUS NOTIFICA (e-SUS NOTIFIES) system, which was developed to register cases of Influenza-Like Illness (ILI) suspected of being COVID-19. Sociodemographic (age, sex, race/colour and occupation, according to the Brazilian Occupation Classification, which is only mandatory for health professionals) and clinical-epidemiological data (type of test conducted and the results, evolution, final classification, type of symptom, and associated clinical conditions) is recorded. There is also information on the patient’s place of residence and notification location (state and municipality), and dates that the symptoms started, notification, and tests conducted (Available at: https://opendatasus.saude.gov.br/dataset/casos-nacionais).

• c) 2020 and 2021 Severe Acute Respiratory Syndrome (SARS) Database: the notification of severe acute respiratory syndrome (SARS) hospitalisations and deaths is mandatory in Brazil, and records are stored on the SIVEP-Gripe (Influenza Epidemiological Surveillance Information System) computerised database. The database includes sociodemographic (date of birth, sex, race/colour, and level of education) and clinical-epidemiological data (signs and symptoms, associated clinical conditions, type of test conducted and its result, evolution, final classification, use of mechanical ventilation, and admission to an ICU bed, among others). There is also information on the patient’s place of residence and notification location (state and municipality), the health centre used, and details such as dates of the start of symptoms, admission, release, or death, notification and tests conducted (Available at: https://opendatasus.saude.gov.br/dataset/bd-srag-2021; https://opendatasus.saude.gov.br/dataset/bd-srag-2020).

Regularly collected data that could be the source of COVID-19 outcomes

a) SUS Hospital Information System (SUS/HIS): Access to Hospitalisation Authorisations (HA) requested and authorised during the period of study are obtained through this system;

b) Mortality Information System (MIS): access to Death Certificates (DC) registered on the MIS on a monthly basis, independent of any review or criticism, registered during the period of study. The update could be monthly, through access to files in the DataSUS native dissemination format (.dbc), with the full details registered on the DC. This information is important to monitor the reduction in deaths as a result of vaccination. However, the registration of revised mortality experiences a delay of at least six months, which imposes a limitation on its use on evaluating vaccination effectiveness.

Data linkage

Within the context of the Brazilian health data ecosystem, the isolated use of the above-mentioned databases is, in general, inadequate for a valid assessment of vaccine effectiveness, since the relevant variables are dispersed throughout different databases. For example, we have a database that provides information on the vaccine doses of every individual, their age, and sex, among other details, but other databases are required to discover if this individual was infected, or became a COVID-19 clinical case. Therefore, linkage strategies for these different databases allow for the capture and incorporation of relevant information which is available on only one of the databases, or which have a varying quality of records and level of completeness between them. This process provides new possibilities for investigation with national databases⁶³ and prospects for investigation into health, particularly in VE evaluation studies.

The objective of linking databases is finding records for the same individual on different databases, in order to combine the various items of information for each of them. In the presence of an unmistakable identification variable between the databases, the deterministic method is used, and records are related by comparing this variable on the different databases. In the absence of a variable of this nature, the probabilistic record linkage method is used, which involves estimating the probability of agreement and disagreement between the common variables on the databases for the record pairing process. While the deterministic method involves a single stage of comparison, the probabilistic method may involve a series of stages, such as standardisation, blocking, and matching.

Conclusions and Recommendations

In the complex and dynamic context that has characterized the development of this pandemic, vaccination effectiveness studies are gaining importance, as part of the resources which are able to produce evidence that subsidises the relevant decisions that are required to control the pandemic.

Although it is important to highlight the results of the efficacy of COVID-19 vaccines, understanding VE is a more complex task, particularly in a pandemic context with a reduced number of doses available, various types of vaccines, and the emergence and circulation of different strains of the SARS-CoV-2 virus. Classic observational studies, particularly test-negative case-control and longitudinal studies are approaches which have been used frequently, due to convenience, their designs are widely known, and analytical methods are standardized on various software programs. While quasi-experimental (or natural experiments) studies are strictly observational, they include a diversified series of designs, which seek to reduce the chance of biases introduced by non-randomized vaccination, since the decision to be vaccinated, or not, in the real world depends on various, non-random factors.

There are a wide range of scenarios in which quasi-experimental studies can be used for vaccine evaluation. The possibility of using data with exogenous attributes, primarily without the requirement of high quality data, avoids sources of bias, and makes quasi-experimental designs an excellent choice to evaluate COVID-19 vaccine effectiveness. Methods such as ITS, using aggregated data, PS, DD, Instrumental Variable and RDD on an individual level, may be used to evaluate vaccine effectiveness. However, their widespread use faces a series of limitations. For example, the main limitation of the PS is that it only controls measured variables, and is dependent on the availability of databases with covariables, which is not always the case. Instrumental variables may not necessarily be good analysis options, since the assumptions can easily be violated, given the difficulty of obtaining good instruments in a context of constant changes in eligibility criteria, and the rapid expansion of vaccination coverage.

The evaluation of COVID-19 vaccine effectiveness, although estimated by quasi-experimental methods, should also take other sources of variation into account, which may hinder the attainment of unbiased measures, such as 1) the introduction of different strains of SARS-CoV-2; 2) the administration of different vaccines and multiples immunization strategies (number, time interval between doses, and combination of vaccines); 3) stage of the epidemic in each location evaluated; and 4) difficulties with gathering records in different Brazilian municipalities.

In this review, we have presented the immense variety of options that exist for any researcher who is interested in following VE in populations, particularly COVID-19 vaccines. The ecosystem of data available is crucial for selecting the best evaluation strategies. It should be clarified that whichever option selected, there will be limitations that, possibly, may not be able to satisfy all of the assumptions. The researcher is responsible for selecting the analytical strategies that are best suited to the context, are the most robust possible, and always include sensitivity tests that support (or not) their findings.

Acknowledgments

We would like to thank the teams of Rede CoVida and Vigivac.

Referências

• ¹
  Clemens J, Brenner R, Rao M, Tafari N, Lowe C. Evaluating new vaccines for developing countries: Efficacy or effectiveness? J Am Med Assoc 1996; 275:390-397.
• ²
  Saadatian-Elahi M, Horstick O, Breiman RF, Gessner BD, Gubler DJ, Louis J, Parashar UD, Tapia R, Picot V, Zinsou JA, Nelson CB. Beyond efficacy: The full public health impact of vaccines. Vaccine 2016; 34:1139-1147.
• ³
  Halloran ME, Haber M, Longini Jr IM, Struchiner C. Direct and Indirect Effects in Vaccine Efficacy and Effectiveness. Am J Epidemiol 1991; 133:323-331.
• ⁴
  Hitchings MDT, Ranzani OT, Torres MSS, Oliveira SB, Almiron M, Said R, Borg R, Schulz WL, Oliveira RD, Silva PV, Castro DB, Sampaio VS, Albuquerque BC, Ramos TCA, Fraxe SHH, Costa CF, Naveca FG, Siqueira AM, Araújo WN, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of CoronaVac among healthcare workers in the setting of high SARS-CoV-2 Gamma variant transmission in Manaus, Brazil: A test-negative case-control study. Lancet Reg Heal - Am 2021; 9:100025.
• ⁵
  Cerqueira-Silva T, Oliveira V de A, Pescarini JM, Bertoldo Júnior J, Machado TM, Flores-Ortiz R, Penna G, Ichihara MY, Barros JV, Boaventura VS, Barreto ML, Werneck GL, Barral-Netto M. Influence of age on the effectiveness and duration of protection in Vaxzevria and CoronaVac vaccines. medRxiv 2021.
• ⁶
  Krammer F. COVID-19 vaccination and immune thrombocytopenia. Nat Med 2021; 27:1145-1153.
• ⁷
  Halloran ME, Struchiner C, Longini Jr IM. Study Designs for Rvauating Different Efficacy and Effectiveness Aspects of Vaccines. Am J Epidemiol 1997; 146: 789-803.
• ⁸
  Sofia Arriola C, El Omeiri N, Azziz-Baumgartner E, Thompson MG, Sotomayor-Proschle V, Fasce RA, Horoch MV, Olalla JEC, Almeida WAF de, Palacios J, Palekar R, Couto P, Descalzo M, Ropero-Álvarez AM. Influenza vaccine effectiveness against hospitalizations in children and older adults - Data from South America, 2013-2017. A test negative design. Vaccine X 2019; 3:1-8.
• ⁹
  Castilla J, Martínez-Artola V, Salcedo E, Martínez-Baz I, Cenoz MG, Guevara M, Alvarez N, Irisarri F, Morán J, Barricarte A. Vaccine effectiveness in preventing influenza hospitalizations in Navarre, Spain, 2010-2011: Cohort and case-control study. Vaccine 2012; 30:195-200.
• ¹⁰
  World Health Organization (WHO). Evaluation of COVID-19 Vaccine Effectiveness. Geneva: WHO; 2021.
• ¹¹
  Ranzani OT, Hitchings M, Dorion Nieto M, D'Agostini TL, Paula RC de, Paula OF de, Villela EF de M, Torres MSS, Oliveira SB de, Schulz W, Almiron M, Said R, Oliveira RD de, Silva PV da, Araújo WN de, Andrews JR, Cummings DAT, Ko AI, Croda J. Effectiveness of the CoronaVac vaccine in the elderly population during a P.1 variant-associated epidemic of COVID-19 in Brazil: A test-negative case-control study. BMJ 2021; 374:1-43.
• ¹²
  Lopez Bernal J, Andrews N, Gower C, Gallagher E, Simmons R, Thelwall S, Stowe J, Tessier E, Groves N, Dabrera G, Myers R, Campbell CNJ, Amirthalingam G, Edmunds M, Zambon M, Brown KE, Hopkins S, Chand M, Ramsay M. Effectiveness of Covid-19 Vaccines against the B.1.617.2 (Delta) Variant. N Engl J Med 2021; 385:585-594.
• ¹³
  Jara A, Undurraga EA, González C, Paredes F, Fontecilla T, Jara G, Pizarro A, Acevedo J, Leo K, Leon F, Sans C, Leighton P, Suárez P, García-Escorza H, Araos R. Effectiveness of an Inactivated SARS-CoV-2 Vaccine in Chile. N Engl J Med 2021; 385:875-884.
• ¹⁴
  Hall V, Foulkes S, Saei A, Andrews N, Oguti B, Charlett A, Wellington E, Stowe J, Gilson N, Atti A, Islam J, Karagiannis I, Munro K, Khawan J, Chand MA, Brown CS, Ramsay M, Lopez-Bernal J, Hopkins S. Effectiveness of BNT162b2 mRNA vaccine against infection and COVID-19 vaccine coverage in healthcare workers in England, multicentre prospective cohort study (the SIREN Study). SSRN Electron J 2021.
• ¹⁵
  Vasileiou E, Simpson CR, Robertson C, Shi T, Kerr S, Agrawal U, Akbari A, Bedston S, Beggs J, Bradley D, Chuter A, Lusignan S de, Docherty A, Ford D, Hobbs R, Mark J, Katikireddi SV, Marple J, McCowan C, McGagh D, McMenamin J, Moore E, Murray JLK, Pan J, Ritchie L, Shah SA, Stock S, Torabi F, Tsang RSM, Wood R, Woolhouse M, Sheikh A. Effectiveness of First Dose of COVID-19 Vaccines Against Hospital Admissions in Scotland: National Prospective Cohort Study of 5.4 Million People. SSRN Electron J 2021.
• ¹⁶
  Lopez Bernal JA, Andrews N, Amirthalingam G. The Use of Quasi-experimental Designs for Vaccine Evaluation. Clin Infect Dis 2019; 68:1769-1776.
• ¹⁷
  Linden A. Conducting interrupted time-series analysis for single- and multiple-group comparisons. Stata J 2015; 15:480-500.
• ¹⁸
  Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: A tutorial. Int J Epidemiol 2016; 0:1-8.
• ¹⁹
  Hungerford D, Vivancos R, Read JM, Iturriza-Gómara M, French N, Cunliffe NA. Rotavirus vaccine impact and socioeconomic deprivation: An interrupted time-series analysis of gastrointestinal disease outcomes across primary and secondary care in the UK. BMC Med 2018; 16:1-14.
• ²⁰
  Silaba M, Ooko M, Bottomley C, Sande J, Benamore R, Park K, Ignas J, Maitland K, Mturi N, Makumi A, Otiende M, Kagwanja S, Safari S, Ochola V, Bwanaali T, Bauni E, Gleeson F, Knoll MD, Adetifa I, Marsh K, Williams TN, Kamau T, Sharif S, Levine OS, Hammitt LL, Scott JAG. Effect of 10-valent pneumococcal conjugate vaccine on the incidence of radiologically-confirmed pneumonia and clinically-defined pneumonia in Kenyan children: an interrupted time-series analysis. Lancet Glob Heal 2019; 7:e337-e346.
• ²¹
  Handley MA, Lyles CR, McCulloch C, Cattamanchi A. Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research. Annu Rev Public Health 2018; 39:5-25.
• ²²
  Carmo GMI Do, Yen C, Cortes J, Siqueira AA, Oliveira WK de, Cortez-Escalante JJ, Lopman B, Flannery B, Oliveira LH de, Carmo EH, Patel M. Decline in diarrhea mortality and admissions after routine childhood rotavirus immunization in Brazil: A time-series analysis. PLoS Med 2011; 8:11.
• ²³
  Andrade AL, Minamisava R, Policena G, Cristo EB, Domingues CMS, Brandileone MC de C, Almeida SCG, Toscano CM, Bierrenbach AL. Evaluating the impact of PCV-10 on invasive pneumococcal disease in Brazil: A time-series analysis. Hum Vaccines Immunother 2016; 12:285-292.
• ²⁴
  Greene SK, Kulldorff M, Lewis EM, Li R, Yin R, Weintraub ES, Fireman BH, Lieu TA, Nordin JD, Glanz JM, Baxter R, Jacobsen SJ, Broder KR, Lee GM. Near real-time surveillance for influenza vaccine safety: Proof-of-concept in the Vaccine Safety Datalink Project. Am J Epidemiol 2009; 171:177-188.
• ²⁵
  Uddin MJ, Shamsuzzaman M, Horng L, Labrique A, Vasudevan L, Zeller K, Chowdhury M, Larson CP, Bishai D, Alam N. Use of mobile phones for improving vaccination coverage among children living in rural hard-to-reach areas and urban streets of Bangladesh. Vaccine 2016; 34:276-283.
• ²⁶
  Raymond S, Li L, Taioli E, Nash D, Liu B. The effect of the Affordable Care Act dependent coverage provision on HPV vaccine uptake in young adult women, National Health and Nutrition Examination Survey 2007-2016. Prev Med (Baltim) 2021; 148:1-8.
• ²⁷
  Wing C, Simon K, Bello-Gomez RA. Designing Difference in Difference Studies: Best Practices for Public Health Policy Research. Annu Rev Public Health 2018; 39:453-469.
• ²⁸
  Fredriksson A, Oliveira GM de. Impact evaluation using Difference-in-Differences. RAUSP Manag J 2019; 54:519-532.
• ²⁹
  Athey S, Imbens GW. Identification and Inference in Nonlinear Difference-in-Differences Models. Econometrica 2006; 74:431-497.
• ³⁰
  Lee M jae, Kang C. Identification for difference in differences with cross-section and panel data. Econ Lett 2006; 92:270-276.
• ³¹
  Goodman-bacon A, Marcus J. Using Difference-in-Differences to Identify Causal Effects of COVID-19 Policies. Surv Res Methods 2020; 14:153-158.
• ³²
  Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011; 46:399-424.
• ³³
  Austin PC. Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: A systematic review and suggestions for improvement. J Thorac Cardiovasc Surg 2007; 134:1128-1135.
• ³⁴
  Thoemmes F, Ong AD. A Primer on Inverse Probability of Treatment Weighting and Marginal Structural Models. Emerg Adulthood 2016; 4:40-59.
• ³⁵
  Coles C, Zell E, Chen W-J, Richardson A, Arnold J, Fairchok M, Schofield C, Clair KJS, Danaher P, Rajnik M, Mcdonough E, Mor D, Ridore M, Burgess T, Millar E, Jackson HM. Estimates of Influenza Vaccine Effectiveness Among Highly Immunized United States Military Members andBeneficiaries: Utilization of Propensity Score Matching. Open Forum Infect Dis 2015; 2:S149.
• ³⁶
  Nation ML, Moss R, Spittal MJ, Kotsimbos T, Kelly PM, Cheng AC. Influenza vaccine effectiveness against influenza-related mortality in Australian hospitalized patients: a propensity score analysis. Clin Infect Dis 2021; 72:99-107.
• ³⁷
  Walzer P, Estève C, Barben J, Menu D, Cuenot C, Manckoundia P, Putot A. Impact of influenza vaccination on mortality in the oldest old: A propensity score-matched cohort study. Vaccines 2020; 8:1-9.
• ³⁸
  Hungerford D, Vivancos R, Read JM, Bonnett LJ, Bar-Zeev N, Iturriza-Gómara M, Cunliffe NA, Frech N. Mitigating bias in observational vaccine effectiveness studies using simulated comparator populations: Application to rotavirus vaccination in the UK. Vaccine 2018; 36:6674-6682.
• ³⁹
  Pawlowski C, Lenehan P, Puranik A, Agarwal V, Venkatakrishnan AJ. FDA-authorized COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system. Med (NY) 2021 13(8):979-992.
• ⁴⁰
  Cousens S, Hargreaves J, Bonell C, Armstrong B, Thomas J, Kirkwood BR, Hayes R. Alternatives to randomisation in the evaluation of public-health interventions: Statistical analysis and causal inference. J Epidemiol Community Health 2009; 65:576-581.
• ⁴¹
  Wong K, Campitelli MA, Stukel TA, Kwong JC. Estimating influenza vaccine effectiveness in community-dwelling elderly patients using the instrumental variable analysis method. Arch Intern Med 2012; 172:484-491.
• ⁴²
  Chakrabarti A, Grépin KA, Helleringer S. The impact of supplementary immunization activities on routine vaccination coverage: An instrumental variable analysis in five low-income countries. PLoS One 2019; 14:1-11.
• ⁴³
  Staiger D, Stock JH. Instrumental Variables Regression with Weak Instruments. Econom Soc 1997; 65:557-586.
• ⁴⁴
  Bor J, Moscoe E, Mutevedzi P, Newell ML, Bärnighausen T. Regression discontinuity designs in epidemiology: Causal inference without randomized trials. Epidemiology 2014; 25:729-737.
• ⁴⁵
  Sekhon JS, Titiunik R. Understanding Regression Discontinuity Designs As Observational Studies. Obs Stud 2017; 3:174-182.
• ⁴⁶
  Moscoe E, Bor J, Bärnighausen T. Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: A review of current and best practice. J Clin Epidemiol 2015; 68:132-143.
• ⁴⁷
  Oldenburg CE, Moscoe E, Bärnighausen T. Regression Discontinuity for Causal Effect Estimation in Epidemiology. Curr Epidemiol Reports 2016; 3:233-241.
• ⁴⁸
  Smith LM, Lévesque LE, Kaufman JS, Strumpf EC. Strategies for evaluating the assumptions of the regression discontinuity design: A case study using a human papillomavirus vaccination programme. Int J Epidemiol 2017; 46:939-949.
• ⁴⁹
  Frio GS, França MTA. Human papillomavirus vaccine and risky sexual behavior: Regression discontinuity design evidence from Brazil. Econ Hum Biol 2021; 40:1-10.
• ⁵⁰
  Van Ourti T, Bouckaert N. The Dutch influenza vaccination policy and medication use, outpatient visits, hospitalization and mortality at age 65. Eur J Public Health 2021; 30:275-280.
• ⁵¹
  Anderson ML, Dobkin C, Gorry D. The effect of influenza vaccination for the elderly on hospitalization and mortality an observational study with a regression discontinuity design. Ann Intern Med 2020; 172: 445-452.
• ⁵²
  Basta NE, Halloran ME. Evaluating the effectiveness of vaccines using a regression discontinuity design. Am J Epidemiol 2019; 188:987-990.
• ⁵³
  Bermingham C, Morgan J, Ayoubkhani D, Glickman M, Islam N, Sheikh A, Sterne J, Walker AS, Nafilyan V. Estimating the effectiveness of first dose of COVID-19 vaccine against mortality in England: a quasi-experimental study. medRxiv 2021.07.12.21260385; doi: https://doi.org/10.1101/2021.07.12.21260385
  » https://doi.org/10.1101/2021.07.12.21260385
• ⁵⁴
  Greene SK, Levin-Rector A, McGibbon E, Greene SK, Levin-Rector A, McGibbon E, Baumgartner J, Devinney K, Ternier A, Sell J, Kahn R, Kishore N. Reduced COVID-19 Hospitalizations among New York City Residents Following Age-Based SARS-CoV-2 Vaccine Eligibility: Evidence from a Regression Discontinuity Design. medRxiv 2021.06.30.21259491; DOI http://doi.org/10.1101/2021.06.30.21259491
  » http://doi.org/10.1101/2021.06.30.21259491
• ⁵⁵
  Fukui M, Kawaguchi K, Matsuura H. Does TB Vaccination Reduce COVID-19 Infection? No Evidence from a Regression Discontinuity Analysis. medRxiv 2020.04.13.20064287; DOI: https://doi.org/10.1101/ 2020.04.13.20064287
  » https://doi.org/10.1101/ 2020.04.13.20064287
• ⁵⁶
  Brasil. Ministério da Saúde (MS). Plano Nacional de Operacionalização da Vacina contra a Covid-19. Brasília: MS; 2021.
• ⁵⁷
  Brasil. Trata-se de atualização das referentes a vacinação contra a covid-19 em gestantes e puérperas até 45 dias pós-parto. Nota Técnica no 2/2021-SECOVID/GAB/SECOVID/MS. 1.1. Rio de Janeiro: Ministério da Saúde (MS); 2021.
• ⁵⁸
  Reuters. German experts say under 60s should not get second AstraZeneca vaccine. 2021. [cited 2021 Set 7]. Available from: https://www.reuters.com/article/us-health-coronavirus-germany-astrazenec-idUSKBN2 BO6KZ
  » https://www.reuters.com/article/us-health-coronavirus-germany-astrazenec-idUSKBN2 BO6KZ
• ⁵⁹
  Reuters. France recommends second mRNA COVID-19 dose for under- 55s who got first dose with Astra. 2021. [cited 2021 Set 7]. Available from: https://www.reuters.com/article/us-health-coronavirus-germany-astrazenec-idUSKBN2BO6KZ
  » https://www.reuters.com/article/us-health-coronavirus-germany-astrazenec-idUSKBN2BO6KZ
• ⁶⁰
  Simpson CR, Robertson C, Vasileiou E, McMenamin J, Gunson R, Ritchie LD, Woolhouse M, Morrice L, Kelly D, Stagg HR, Marques D, Murray J, Sheikh A. Early Pandemic Evaluation and Enhanced Surveillance of COVID-19 (EAVE II): Protocol for an observational study using linked Scottish national data. BMJ Open 2020; 10:1-7.
• ⁶¹
  Vasileiou E, Simpson CR, Shi T, Kerr S, Agrawal U, Akbari A, Bedston S, Beggs J, Bradley D, Chuter A, Lusignan S de, Docherty AB, Ford D, Hobbs R, Joy M, Katikireddi SV, Marple J, McCowan C, McGagh D, McMenamin J, Moore E, Murray JLK, Pan J, Ritchie L, Shah AS, Stock S, Torabi F, Tsang RSM, Wood R, Woolhouse M, Robertson C, Sheikh A. Interim findings from first-dose mass COVID-19 vaccination roll-out and COVID-19 hospital admissions in Scotland: a national prospective cohort study. Lancet 2021; 397:1646-1657.
• ⁶²
  Haas EJ, Angulo FJ, McLaughlin JM, Anis E, Singer SR, Khan F, Brooks N, Smaja M, Mircus G, Pan K, Southern J, Swerdlow DL, Jodar L, Levy Y, Alroy-Preis S. Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data. Lancet 2021; 397:1819-1829.
• ⁶³
  Sanni Ali M, Ichihara MY, Lopes LC, Barbosa GCG, Pita R, Carreiro RP, Santos DB dos, Ramos D, Bispo N, Raynal F, Canuto V, Almeida B de A, Fiaccone RL, Barreto ME, Smeeth L, Barreto ML. Administrative data linkage in Brazil: Potentials for health technology assessment. Front Pharmacol 2019;10:1-20.
• ⁶⁴
  Sadoff J, Gray G, Vandebosch A, Cárdenas V, Shukarev G, Grinsztejn B, Goepfert PA, Truyers C, Fennema H, Spiessens B, Offergeld K, Scheper G, Taylor KL, Robb ML, Treanor J, Barouch DH, Stoddard J, Ryser MF, Marovich MA, Neuzil KM, Corey L, Cauwenberghs N, Tanner T, Hardt K, Ruiz-Guiñazú J, Le Gars M, Schuitemaker H, Van Hoof J, Struyf F, Douoguih M. Safety and Efficacy of Single-Dose Ad26.COV2.S Vaccine against Covid-19. N Engl J Med 2021; 384:2187-2201.

Edited by

Chief editors:

Romeu Gomes, Antônio Augusto Moura da Silva

Publication Dates

• Publication in this collection
  26 Nov 2021
• Date of issue
  Nov 2021

History

• Received
  21 Sept 2021
• Accepted
  29 Sept 2021
• Published
  01 Oct 2021