Ten years ago, I published a peer-reviewed article with Florida State University professor Jeffrey Lacasse entitled, What Does It Mean for an Intervention to “Work”?—where we explored different factors that went into a scientific article claiming a health intervention was “effective” or “successful.” Then, like now, a great deal of public health messaging promotes various interventions as effective for children and adults alike—with significantly less attention going to what that word actually means. Jeff and I were hoping to promote a more thoughtful conversation about exactly that.
Over recent months, we’ve likewise heard many reports and commentaries celebrating how effective the leading coronavirus vaccine candidates have turned out to be. In the same spirit of promoting thoughtful inquiry about these crisis interventions, I walk through a similar analysis below of the most recent data from leading vaccine clinical trials – asking the same question posed in the British Medial Journal this last fall, “What [does] it mean exactly when a vaccine is declared ‘effective’?”
I take as my primary texts the published research reports of the three leading coronavirus vaccines (of 44 being tested clinically, and another 151 in preclinical development), each of which released new updates within the last month. Hyperlinks to the full text of each are available below, including:
- Moderna’s most recent Phase III findings in partnership with the National Institute of Allergy and Infectious Diseases (NIAID): Lindsey Baden, et al. Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine (December 30, 2020, New England Journal of Medicine).
- BioNTech-Pfizer’s most recent Phase III findings: Fernando Polack, et al. Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine (December 10, 2020, New England Journal of Medicine).
- Oxford-Astrazeneca’s most recent Phase III findings: Merryn Voysey, et al. Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomized controlled trials in Brazil, South Africa, and the UK (December 8, 2020, Lancet).
I’ve had experience reviewing other clinical trials over the years and led my own randomized-controlled trial of a mindfulness-based intervention for youth several years back. While I’m familiar with some of the many variables that go into these kinds of rigorous, controlled studies, I’m not pretending to be an immunology or vaccine expert here. Instead, this is offered as a lay review of publicly available data written for an audience of people, like me, who aren’t experts in these matters—but who want to know enough to feel confident in their health decisions.
In doing so, I acknowledge a fraught, pandemic climate wherein any question raised can be feared as potentially “undermining public trust” or even “spreading misinformation.” That, of course, is not my intention. Like everyone else, I care about promoting truth and preserving public trust. And surely, space to continue exploring matters carefully together is protective to us all, and to the precious lives and health we are all trying to preserve. Indeed, if we have reached a point where sincere questions and honest concerns about popular views are automatically cast as “dangerous,” the peril of this moment is larger than we might realize.
In October 2020, Oxford scholar Dr. Susanne Hodgson and colleagues asked similar questions in a review entitled, “What defines an efficacious COVID-19 vaccine? A review of the challenges assessing the clinical efficacy of vaccines against SARS-CoV-2.” In that piece, they referenced the unprecedented “warp speed” of COVID-19 vaccine development to date, “with some candidates beginning phase 3 studies within 4 months of the start of vaccine development”—thanks to modifications such as conducting usually sequential animal and human trials simultaneously, and combining Phase 2 and 3 of trials. Given this speed, these scholars argue for the social value of heightened scrutiny of the clinical results, noting that:
Assessment of the efficacy of a vaccine is complex for many diseases but particularly so in the case of SARS-CoV-2, where the fundamental understanding of the pathogen is evolving. … In keeping with the development of any novel medical intervention, but particularly so in this [charged] context, it is imperative that efficacy outcomes for a SARS-CoV-2 vaccine are critically appraised with scientific rigour to understand their generalisability and clinical significance.
Adding to this kind of healthy critical appraisal necessary for a productive public discussion is my hope. (Special thanks to the review from four expert reviewers – two researchers, and two MD’s – and their thoughtful feedback that helped refine the piece). I summarize the exploration below in three sections, each taking up one aspect of what many would assume it means for a health intervention like vaccinations to be deemed as “effective.”
1. Sustainability: Effective over a certain period of time. When any health intervention is presented in news reports as effective, most people take for granted this means its success has proven to endure over some sustained period of time. It’s common, though, for many to be surprised in learning that most new pharmacological treatments are approved by the FDA on the basis of RCTs (Randomized Controlled Trials) with between 8 and 12 weeks of follow-up.
These Phase III vaccine trials fall right in that same range, with a median duration of 8 weeks follow-up after the second vaccine dose for all trials (and a subset of the Pfizer sample having another 6 weeks of follow-up after that). This is a limitation that investigators have acknowledged and been open about. As the Moderna researchers note, “Key limitations of the data are the short duration of safety and efficacy follow-up.” Referencing this same time limitation, Pfizer researchers state that “both the occurrence of adverse events more than 2 to 3.5 months after the second dose and more comprehensive information on the duration of protection remain to be determined.”
These kinds of acknowledgments are important since most of the more serious adverse effects sometimes witnessed in a subset of people taking other vaccines (such as auto-immune neurodevelopmental and chronic conditions) would take much longer than a few months to manifest.
It’s partly these shorter-term follow-up periods common in controlled trials that prompted UCLA scholar David Cohen to raise concern years ago with the “inability of conventional clinical trials to provide a true picture” of a health intervention’s full effects.
In the absence of more serious and systematic longer-term research, of course, there are some definitive things that can still be said about a vaccine’s more immediate effects. Namely, in this case, that people who received this vaccine across trials were less likely to experience COVID-19 any symptoms of COVID-19 in the 2-3 months after, compared with those who did not get the vaccine (that includes “infections even with only mild symptoms” which “qualify” as a “COVID-19 event” in all Phase 3 trials).
That’s what the “90%” and “94%” and “95% successful” figures refer to, different than what some presume (I’ve noticed many assuming these stats meaning something like this: “hey cool – there’s a 95% chance this vaccine will be effective in making sure I don’t get COVID-19!”). The statistical meaning of these numbers is more nuanced, namely: that out of the total number of people vaccinated (compared with the total not vaccinated), the percentage of people who showed symptoms of COVID-19 were that much less in the treatment group. Specifically:
- Out of 15,210 participants in the Moderna placebo group, 185 people showed some symptoms of COVID-19, compared with 11 people in the vaccinated group.
- Out of 21,728 participants in the Pfizer placebo group, 162 people showed some symptoms of COVID-19, compared with 8 people in the vaccinated group.
That’s what the numbers mean. They also mean that for every 20,000 people willing to get vaccinated, we can expect just under 200 of them will be less likely in the months post-vaccination to experience COVID-19 symptoms. Although it would have been helpful to have a large enough study to investigate whether the vaccine prevents serious illness, hospitalization and death, the size and cost of that study would have been too large.
That’s why, as summarized by Peter Doshi, associated editor of the British Medical Journal, “none of the trials currently under way are designed to detect a reduction in any serious outcome such as hospital admissions, use of intensive care, or deaths.” As he explained, “Hospital admissions and deaths from covid-19 are simply too uncommon in the population being studied for an effective vaccine to demonstrate statistically significant differences in a trial of 30, 000 people. The same is true of its ability to save lives or prevent transmission: the trials are not designed to find out.”
As the trial researchers acknowledge, the existing data is not “sufficient to assess asymptomatic infection” (Moderna study),—or “address whether vaccination prevents asymptomatic infection” (Pfizer study)—with additional evaluations planned “to assess whether vaccination affects infectiousness.” The AstraZeneca study monitored asymptomatic transmission in one study arm but noted: “no testing plan for asymptomatic infections” in another arm of the study, which likewise didn’t allow conclusive findings on that point until more data emerge.
Summarizing these limitations, Tal Zaks, chief medical officer at Moderna acknowledged, “Our trial will not demonstrate prevention of transmission, because in order to do that you have to swab people twice a week for very long periods, and that becomes operationally untenable.” He also added, “Would I like to know that this prevents mortality? Sure, because I believe it does. I just don’t think it’s feasible within the timeframe [of the trial].”
2. Generalizability: Effective for more than healthy volunteers. As a general rule, it’s well known that formal study “efficacy” of a health intervention (as observed in controlled conditions) does not always predict real-world “effectiveness” in non-random, less ideal conditions. As an example of this, one scholar cited “the effectiveness of rotavirus vaccines in children in low-income and middle-income settings was lower than the efficacy observed in children in high-income countries.”
That being said, all clinical trials, including these ones, do their best to create representative samples. True representativeness, however, would involve so many variables that a clean comparison becomes difficult. So, like all controlled studies, these trials applied a set of filters to everyone who participated, in hopes of ruling out certain groups of people more naturally vulnerable to the disease. For instance, the Moderna trial had 14 different exclusion criteria—including people with a “known or suspected allergy” and those in an “immunosuppressive or immunodeficient state.” Pfizer researchers likewise noted, “key exclusion criteria included a medical history of Covid-19, treatment with immunosuppressive therapy, or diagnosis with an immunocompromising condition.” They added that “additional studies are planned to evaluate [the vaccine] in pregnant women, children younger than 12 years, and those in special risk groups, such as immunocompromised persons.”
The obvious question this raises is how well this kind of data generated with comparatively healthy volunteers will apply to those we are understandably seeking to prioritize in our COVID-19 response—namely, those who are immunocompromised, older, or otherwise more vulnerable in their health.
As Peter Doshi notes, “If frail elderly people…are not enrolled into vaccine trials in sufficient numbers to determine whether case numbers are reduced in this group, there can be little basis for assuming any benefit in terms of hospital admissions or mortality.” It’s true that study volunteers tended to be younger—with trials varying in how much they involved older participants in their samples. For instance, the AstraZeneca study notes that since participants older than 56 were added later in the study (starting in August), only 12% of participants were in that age bracket. Given that limitation, they note that “efficacy data in these [older] cohorts are currently limited by the small number of cases” and conclude that “vaccine efficacy in older age groups could not be assessed but will be determined, if sufficient data are available, in a future analysis after more cases have accrued.”
Other trials fared better in this regard. For example, Pfizer researchers report 21.4% of participants 65 years of age and older and 4.3% 75 years of age and older. And Moderna researchers report nearly 25% of participants 65 years of age or older—similarly acknowledging that “the relatively smaller numbers of cases that occurred in older adults and in participants from ethnic or racial minorities and the small number of previously infected persons who received the vaccine limit efficacy evaluations in these groups.”
Despite this acknowledged data limitation, most vaccination campaigns appear to be prioritizing older seniors, after health care workers, as early recipients. Indeed, at current projections, millions of Americans of all ages will be vaccinated by later this year. In the meanwhile, additional analyses such as Moderna’s “Real World Effectiveness Study” are starting this month, and continuing through December.
3. Safety: Effective without provoking other concerning effects that would outweigh benefits. The final issue is the one about which the public seems most interested: benefits aside, are there any inadvertent consequences that will negatively affect me in any way?
The level of careful monitoring across different phases of study should offer some assurance—each phase expanding the sample size and representativeness of the cohort. Most careful attention goes to the 7 days after the injection in these particular Phase III studies—with the most common immediate side effects (in this case, based on Pfizer data) being injection site pain (84%), fatigue (63%), headache (55%), muscle pain (38%), chills (32%), joint pain (24%), fever (14%), injection site swelling (10.5%), injection site redness (9.5%), nausea (1.1%), malaise (0.5%), and enlarged lymph nodes (lymphadenopathy) (0.3%).
These are the more common effects observed by researchers within the first 7 days, similar to what happens with other vaccines (and which, some experts are quick to point out, are most likely a good sign your immune response is reacting correctly, releasing cytokines, etc).
Moderna researchers note in an FDA briefing document that “severe solicited adverse reactions” are “more frequent after dose 2 than after dose 1.” While packaging inserts call this range of side-effects “mild to moderate” and “well-tolerated,” one physician representing the American Medical Association encouraged public health officials to help Americans realize “this is not going to be a walk in the park.”
It’s good that we’re focusing so carefully on the first week after the second vaccination shot. Yet, as one doctor put it candidly, “I’m not as worried about the short-term effects, the redness, the swelling, all those signs that show that the body has recognized the shot as foreign and is reacting to it. I’m much more concerned about potential long-term effects on brain inflammation or auto-immunity”—which, as we’ve noted earlier, usually take much longer than a few weeks to show up.
The effects that emerge after one week is something we are still learning about. In the studies, participants would self-report adverse effects after that first week if they notice something unusual. And we are starting now to have access to more reports from real-world administration across the initial millions of vaccinations. Last week, for instance, the CDC reported 4,393 reports of adverse reactions out of 2 million vaccine doses given through the federal government’s Vaccine Adverse Event Reporting System – including 29 cases of vaccine-related anaphylaxis.
Seven total cases of Bell’s palsy (facial paralysis) emerged in vaccinated participants in the Moderna and Pfizer trials, and two cases of transverse myelitis in the AstraZeneca trial. Researchers in each study raised the possibility these were “anecdotal finding[s]” and “chance event[s]” (Moderna) and “unlikely to be related to study interventions” in one case, with a “relationship remain[ing] possible” in the second case (AstraZeneca).
While acknowledging cases of intractable nausea and facial swelling as likely related to the vaccine, Moderna researchers note that in the case of a few other “serious adverse events of rheumatoid arthritis, peripheral edema (swelling of lower limbs or hands), dyspnea with exertion (sensation of running out of the air and of not being able to breathe fast), and autonomic dysfunction (damaged ANS nerves also called autonomic neuropathy), a possibility of vaccine contribution cannot be excluded.”
Beyond this, anything else concerning that may occur in months or years after is unknown at this point, although Phase IV trials will seek to monitor those better. Important questions, then, still remain.
Some justifiably wonder: aside from the understandable signs of the immune system working against the non-lethal pathogen, why are there any adverse effects at all? One theory of what prompts adverse effects centers on extra ingredients called “adjuvants” like polyethylene glycol (in both the Moderna and Pfizer vaccines)—a substance added to a vaccine in order to stimulate the immune system to make a strong immune response. Although common in vaccinations, these ingredients can sometimes trigger auto-immune responses and anaphylaxis in a subset of people.
While researchers are definitely monitoring for those effects, they openly acknowledge in their published reports how their study design limits some of this assessment. As the Moderna researchers write, “the ability to detect rare events is limited, given the trial sample size.” In a separate FDA briefing document under a section titled, Adverse reactions that are very uncommon or that require longer follow-up to be detected,” Moderna researchers state that “following authorization of the vaccine, use in large numbers of individuals may reveal additional, potentially less frequent and/or more serious adverse events not detected in the trial safety population.”
Pfizer studies likewise reported “with approximately 19,000 participants per group in the subset of participants with a median follow-up time of 2 months after the second dose, the study has more than 83% probability of detecting at least one adverse event, if the true incidence is 0.01%, but it is not large enough to detect less common adverse events reliably.” Dr. Susanne Hodgson and colleagues summarize the key point: that “clinical trials might not be sufficiently powered to detect … serious adverse events related to the vaccine, if they are uncommon.”
The one adverse effect researchers are watching out for most. Given the history of previous coronavirus vaccine efforts—including those for SARS, and MERS—there is one specific, longer-term concern that researchers in current trials are keeping an eye out for especially. Namely, the possibility that the vaccination could prompt a robust antibody response initially, while eliciting another kind of effect later on—something Dr. Peter Hotez, from the Baylor College of Medicine called the “unique potential safety problem of coronavirus vaccines” in testimony before Congress early in discussions of vaccine development. Referring to a decades-old mystery of why “some who got the [1960 RSV] vaccine actually did worse,” he explained: “What happens is you get immunized and then when you get exposed to the virus you get this paradoxical immune enhancement phenomenon.”
Referring to his own work attempting to develop a vaccine after the 2002 SARS epidemic, Dr. Hotez continued: “When we started developing coronavirus vaccines we noticed in laboratory animals that they started to show some of the same immune pathology that had resembled what happened 50 years earlier.” [The vaccinated animals had strong antibody response initially, followed by body-wide inflammation and terminal sickness after exposure to the real-life virus]. “So, we said, oh God this is going to be problematic.”
In short, “immunized animals had increased likelihood of infection or severe disease when subsequently challenged with the target pathogen,” as Dr. Susanne Hodgson and colleagues summarize—something they refer to as “vaccine-associated enhanced respiratory disease” (VAERD). Various other names that have been used for this phenomenon include “pathogenic priming,” “paradoxical immune enhancement” (PIE), and “antibody-dependent enhancement” (ADE).
This is one reason vaccines for SARS, MERS, and RSV never continued along the path to approval, as explained in a 2020 review called “Informed Consent Disclosure to Vaccine Trial Subjects of Risk of COVID-19 Vaccine Worsening Clinical Disease.” The authors underscore the possibility of this same risk in current efforts, noting that “COVID-19 vaccines designed to elicit neutralizing antibodies may sensitize vaccine recipients to more severe disease than if they were not vaccinated”—although, as Dr. Hodson points out, “it is unknown how it might manifest in humans.”
Thankfully, researchers are paying careful attention to this possibility, as evident in the clinical protocols. And in an FDA briefing document, Moderna researchers note that what they have found so far has been encouraging: “Available data do not indicate a risk of vaccine-enhanced disease” while adding: “however, risk of vaccine-enhanced disease over time, potentially associated with waning immunity, remains unknown and needs to be evaluated further in ongoing clinical trials.”
Such questions are not uncommon with new medical interventions, and many still believe their benefit far outweighs potential risks, while others naturally look for confirmations from additional data in the year ahead.
Interpreting emerging phenomena. It’s worth asking, what would happen if more serious adverse effects were to arise in the larger population 6 or 12 or 18 months down the line (that are not currently detectable in the study cohort)? Would they be attributed to the vaccine?
Not likely. If medical history is any indication, these concerning effects would likely be attributed to a new, novel pathogen—or to another condition, including coronavirus itself. And God forbid, were there to arise any adverse effects exacerbating COVID19 symptoms or making people more likely to succumb to the virus, it is also not hard to imagine these consequences being attributed primarily to those opting out of the vaccine, (rather than coming from the vaccination itself).
Given these limitations of existing protocols, the counsel from Oxford scholar Dr. Susanne Hodgson and colleagues seems especially wise:
The US FDA recommends that follow-up of study participants should continue for as long as is feasible, ideally for at least 1–2 years, to assess the duration of protection and potential for vaccine-associated enhanced respiratory disease as the immune response to the vaccine wanes. Given that COVID-19 vaccines might be deployed in the early post-marketing period to large populations over a short timeframe, it will be important that robust, ongoing pharmacovigilance is in place post licensure to identify safety signals that large-scale RCTs might not capture.
Other questions. Certainly, there are other questions worth exploring—especially when considering the scope of other vaccine candidates that could be reviewed (across eight manufacturers). For instance, many of these vaccines are drawing on new technology we’ve had less patient experience with historically, including the mRNA approach in the Moderna and Pfizer vaccines.
It’s also worth considering (and not overlooking) who sponsors the research. When scientific standards are met, clearly industry-sponsored research may provide valid and legitimate results. Yet selective dismissal of negative possibilities and over-emphasis on positives is still a painful memory from the recent opioid epidemic. And generally speaking, reviews of biomedical research have found that industry-funded studies were more likely to reach outcome conclusions favoring the sponsor’s product when compared with research independent of corporate interests. On this point, it’s worth noting that the trial conducted with the most academic independence—the third AstraZeneca trial set up and run by the scientists at Oxford who were determined to generate and publish the most comprehensive data—was the one that generated the 60% success finding (more in line with previous trials—but disappointing compared with the other trials).
As a final question, some continue to wonder why more attention and investment couldn’t also be given to other mass-scale preventive interventions that research suggests might make a difference in decreasing COVID-19 rates, such as systematically increasing Vitamin C or D intake. Dr. Angel Lybbert and Dr. Ben Satterfield from Mayo Clinic wrote about these kinds of possibilities in Public Square Magazine early in the pandemic (see Five Steps to More Confidence Against Coronavirus), but there’s been very little of this kind of commentary in the months since. Why not do more to collectively emphasize these kinds of adjustments as another way to boost immunity and strengthen our collective protection against the virus?
Summary—Let’s stay attentive and clear-eyed about limitations. The tendency everywhere these days in media journalism is to trumpet all these studies as “landmark findings” that prove “remarkable effectiveness” of COVID-19 vaccinations. For instance, the Senior Vice President at Pfizer said in a press release, “These pivotal data demonstrate that our COVID-19 vaccine candidate is highly effective in preventing COVID-19 disease and is generally well-tolerated.”
These are the kinds of statements that get passed along and channeled as a summary to the public, without reference to any of the nuanced issues above. That seems, in my mind, a disservice to a public that deserves a full risk/benefit profile so they can make an educated choice in the spirit of true “informed consent.” How are they to do this if presented with an overly glowing report of the benefits?
Although the pace of development has been heralded as a triumph of science, some continue to worry that this greatly accelerated approval process has sacrificed too much caution. Dr. Anthony Fauci himself has raised this kind of a concern months ago, sharing in one CBS interview his worry that UK regulators “just took the data from the Pfizer company and instead of scrutinizing it really, really carefully [t]hey said, ‘OK, let’s approve it. That’s it.’”
The key point to underscore here is the importance of humility, and recognizing that even our best medical trials are limited in what they can uncover – all of which means there is a potential for us to be mistaken and misled. As Dr. Cohen summarized several years ago, conventional research protocol focused on “initial, ‘selective’ action” can potentially distract and draw attention away from the “more extensive picture of complex, rippling [effects]” of any intervention.
Even with longer periods of time, these kinds of randomized, controlled trials simply aren’t designed to assess the full scope of an intervention’s effects. That’s why, as Dr. David Jacobs pointed out nearly two decades ago, there is so often a “disparity which exists between side effects established in randomized, placebo-controlled clinical trials” versus a “much broader range and severity of adverse drug reaction reports which emanate from non-RCT formats.”
That includes more nuanced emotional, mental, and psychological effects that may arise in the months or years following a health intervention, but which are simply not monitored in preliminary shorter-term studies. Consequently, Dr. Jacobs went on to note, it often appears that effects from medical interventions “in the realm of psychosocial functioning are ruled out in clinical trials … by virtue of non-investigation.”
As reflected above, some harbor concern that these trials are not set-up to assess long-term effects, more serious adverse effects, and older populations. Of course, if rigorous research evidence ultimately shows a vaccine to be effective over time, with acceptable side-effects, then such information should be broadly disseminated. Whether or not the existing evidence has reached that threshold is what every one of us must now decide.
