The Challenge of Measuring Vaccine Effectiveness

Table 1. WHO guidance on conducting VE evaluations using breakthrough case data outlines potential reasons for crude VE estimates to differ from expectation based on clinical trial and observational study results. Differing crude estimates may have valid or invalid reasons that require more rigorous studies to examine.

Controlled studies can yield more reliable measurements of VE. 

More rigorously controlled studies can often account for a wider variety of potential confounding factors. For example, such studies can be designed from the outset to gather more granular data—often to the individual or household level—to help tease the competing strands of explanation apart. Controlled studies use a variety of techniques and analyses to help provide a complementary and overall, more complete picture of which factors are driving changes in vaccine performance at a given time. These studies are generally large enough and have data detailed enough to test possible explanations for VE estimates that differ from expected values.

• Studies can be designed to separate populations by their time since completing a primary series or booster. This clarifies timing when subjects are transitioning from the broad categories of “fully vaccinated with primary series” to “boosted.”
• Stratifying outcomes by time period or variant predominance can more precisely account for differences in vaccine protection due to the ebb and flow of new variants, since VE in the time of Delta can be very different from VE in the time of Omicron.
• They can look at VE for different vaccine products and dose combinations, since the different doses, molecular compositions, and platforms may have different outcomes, and different populations may have received different vaccines.
• Outcomes such as asymptomatic infection, symptomatic infection, hospitalization, critical illness, or death can be more precisely defined, measured, and separately assessed.
• An increasingly complex population immunity landscape comprising those with varying levels of vaccine-induced immunity (from which vaccine products, number of doses, time elapsed since vaccination) as well as infection-induced immunity (prior infection by which variant and how much time has elapsed) can be disentangled.

The original vaccine trials were randomized controlled trials, where some subjects were given the vaccine and others were given a placebo to determine vaccine efficacy. The U.S. Centers for Disease Control and Prevention (CDC) has ongoing systems in place to monitor infection in long-term care facilities and Covid-19 cases, deaths, and hospitalizations in the general population, comparing these outcomes between vaccinated and unvaccinated individuals. Case studies and clinical observations can be used to highlight initial findings, such as the reports from an early outbreak of the Delta variant in Massachusetts. Information gleaned from these observations cannot reliably estimate vaccine effectiveness because of confounding factors and a lack of controls, but they can be used to inform more carefully controlled studies.

Several kinds of studies can be used to more precisely examine real-life vaccine effectiveness. Household contact studies are designed to determine the level of transmission that occurs from infected individuals to their close contacts (and to see if vaccination affects the level of transmission). Case-control studies (including test-negative designs) can identify similarly matched individuals with and without Covid-19 and compare their vaccination status. Cohort studies can look at individuals with different levels of vaccination and follow those people to see if they go on to experience the designated outcome (such as infection, symptomatic illness, hospitalization, or death).

Table 2. Key vaccine effectiveness studies and monitoring systems used in the US in an interactive list that is sortable and searchable by name, outcome, population, or study design.

Controlled studies can also produce misleading results.

Even within these more rigorous study designs, the devil can be in the details.

• Sample sizes can be small, particularly when looking at many subgroups or when there has been less time to vaccinate a group (as with younger children most recently becoming eligible for vaccination).
• Outcomes can be rare, which is why data on deaths in younger populations is often too sparse to draw strong conclusions.
• Confidence intervals, which show a range of values that are consistent with the data from a study, can be very wide. They can also overlap, meaning comparison groups may not be significantly different.

Because of these issues, study findings showing only small differences or wide confidence intervals without a strong evidence base are sometimes treated conservatively with an honest “We don’t know right now.” Any new information, such as the newly emerging trends described above, should be scrutinized closely and compared with other sources. It may be a trend that indicates something real and significant or might just be a blip on our pandemic data landscape.

There are many things to consider when interpreting VE studies.

It may be tempting to read a news headline—or even a tweet—to draw conclusions about what a VE study reports, but it is important to find out more about how the study was conducted. Some characteristics of studies that can affect how they may be interpreted include:

• Study design: Different study designs can have advantages and disadvantages.
• Sample size: A small sample size is more likely to show less conclusive and more wide-ranging results.
• Subject characteristics: The age of subjects can have a large effect on VE. Other characteristics might include immunocompromised status, comorbidities, access to testing or healthcare, or behavioral characteristics, such as occupational exposure or level of risk tolerance.
• Vaccine product and dose: It is important to note the vaccine manufacturer and type, along with the number of doses and interval between doses. The composition of “vaccinated” and “unvaccinated” populations might be affected by the inclusion of partially vaccinated individuals, or those who received a booster.
• Prior infection: As the number of infections rises, the number of previously infected individuals grows. A prior infection can affect vaccine effectiveness, sometimes acting as an additional dose.
• Endpoint and outcome definition: VE can be measured against an array of outcomes of increasing severity, including: infection (which may be asymptomatic), transmission, symptomatic illness, hospitalization, ICU admission, or death. How these particular outcomes are defined can also lead to clear differences in VE assessments, even against outcomes that could fall in the same broad category.
• Reference group definition: VE can also be measured relative to different reference groups for comparison. Most are measuring the benefit of vaccination against a fully unvaccinated group, but the value of booster doses could be compared against those who received a primary vaccine series. An even newer example is the evaluation of 4th dose effectiveness compared to primary series plus boosted (totaling 3 mRNA doses) recipients in Israel.
• When the study was conducted: The initial vaccine trials performed in 2020, when the ancestral SARS-CoV-2 strain dominated, yielded different results from newer VE studies performed when the Delta variant was dominant. Those, in turn, will differ from the newest studies against Omicron due to different transmissibility, virulence, and immune evasion properties of these new variants.
• Time since vaccination: Vaccine-induced immunity wanes over time, which is particularly evident with Omicron as the dominant circulating variant.

The vaccines remain highly effective.

No single study (out of more than 1,000 documented by the WHO) can determine vaccine effectiveness. Systematic reviews of these studies, with a critical eye to the above considerations, offer a more complete assessment of vaccine effectiveness. The best-controlled studies, however, consistently show that boosters significantly improve VE against both symptomatic disease and more severe outcomes, even during Omicron’s predominance.