KP biostatisticians help monitor COVID-19 vaccine safety

Dr. Jennifer Nelson (center) helps lead nationwide efforts to watch for potential problems with vaccine safety.

Dr. Jennifer Nelson explains how Kaiser Permanente scientists are helping the CDC and FDA keep an eye out for rare adverse events

News that health care workers and high-risk populations are now getting COVID-19 vaccinations is a big source of pride here at Kaiser Permanente Washington Health Research Institute (KPWHRI), where our Vaccine and Treatment Evaluation Unit launched the world’s first COVID-19 vaccine trial in March 2020. By mid-December, their quick work and subsequent participation in a phase 3 trial helped lead the U.S. Food and Drug Administration (FDA) to approve that vaccine — the mRNA-1273 vaccine co-developed by Moderna Inc. and the National Institute of Allergy and Infectious Diseases (NIAID) — for broad distribution.

But our institute’s efforts to help stop the spread of COVID-19 are far from over. In addition to monitoring the health of the volunteers who enrolled in the trials, researchers here are working with the Centers for Disease Control and Prevention (CDC) and the FDA on a variety of projects to monitor the safety and effectiveness of the vaccines now being distributed to the general population, and to improve the care of patients with COVID-19.

To better understand this phase of the institute’s COVID-19 work, we spoke with Jennifer C. Nelson, PhD, director of KPWHRI’s Biostatistics Unit. An affiliate professor of biostatistics at the University of Washington (UW), Dr. Nelson was recently appointed to serve on Moderna’s External Safety Advisory Board for their COVID-19 vaccine program in 2021.

Q. Now that COVID-19 vaccine distribution is underway, we’ve heard that a few rare, but serious, reactions to the vaccines have occurred. Can you tell us how the CDC tracks such events — and how Kaiser Permanente researchers have been involved in such safety efforts?

Sure. The CDC and FDA have several ways to track adverse events following a vaccination. By “adverse events,” we mean problems such as life-threatening health issues or those that can result in hospitalization. These are not the kinds of common side effects we might expect to see with many vaccines — issues like a sore arm, a mild fever, or fatigue. One prime example of an adverse event is anaphylaxis, which is the rare but serious allergic reaction that must be immediately treated with a drug such as epinephrine. As of Dec. 30, the media had reported about a dozen cases of anaphylaxis related to the COVID-19 vaccines.

One of the mechanisms for tracking such events is the Vaccine Adverse Event Reporting System (VAERS), which collects reports from health care providers and the public as soon as they happen. The CDC and FDA track such incidents to determine if there are unexpected problems or unusual patterns occurring that might require further investigation and/or immediate action.

Current information about allergic reactions is available on the CDC website.

Q. What about the CDC’s Vaccine Safety Datalink (or VSD) — the system that researchers here helped launch back in 1990. How does that help with safety monitoring?

The VSD uses electronic health records from several large integrated health care systems — including Kaiser Permanente Washington — to proactively search de-identified medical records from a population of about 12 million people for signs of any connection between possible adverse events and vaccinations.

The VSD team, which includes researchers from all the participating organizations, can also do “rapid cycle analysis” using weekly data reports on specific health problems identified by VAERS, experience with the same vaccine in other countries, or prior experience with similar vaccines in the United States. Our colleagues at Kaiser Permanente Northern California’s Vaccine Study Center are leading these rapid analyses, which can determine if the rate of any adverse events among vaccinated people is higher than among comparison groups.

Q. Why is the VSD focused on integrated health systems like Kaiser Permanente? 

Many research organizations struggle to get records on vaccination because people can get their shots from a variety of providers. But organizations like Kaiser Permanente have access to complete health records on large, defined populations. This allows VSD to analyze data on patients’ vaccinations, prescription medications, lab results, and medical diagnoses such as stroke, heart attack, and so on. Also, our claims data can tell us if patients are getting vaccinations outside the system.

Plus, the Kaiser Permanente research institutes have the biostatistical expertise that’s needed to design complex studies using several different kinds of data from various organizations. Working on the VSD has given my colleague Andrea Cook and me the opportunity to lead the development of new statistical methods for actively monitoring electronic health records for rare adverse events across distributed data networks.

Q. I understand that KPWHRI’s biostatisticians and informaticists have done similar work for the FDA’s Sentinel Initiative. Can you describe that?

KPWHRI’s David Carrell and I, along with Patrick Heagerty, who is professor and past chair of biostatistics at UW, help lead the biostatistical and health informatics research for Sentinel, which monitors the safety of FDA-regulated medical products — including vaccines — using data from collaborators in academic medical centers, health care systems such as Kaiser Permanente, and health insurance companies.

Q. So how does Sentinel differ from the VSD?

I think of VSD and Sentinel as complementary systems. VSD studies a smaller population, with 12 million people versus the >100 million monitored through Sentinel. But VSD has much wider information because it includes both claims data and electronic health records. Sentinel primarily uses claims data only, although it’s working to expand. Also, VSD can access information much more rapidly and can conduct rapid chart review to confirm true adverse-event status.

Q. And will we be working on COVID-19 issues through Sentinel?

Yes. David and his colleagues at Vanderbilt University are working on scaling up natural language processing (NLP) and machine learning methods that extract information from physicians’ clinical text notes to improve researchers’ accuracy when identifying information about adverse events from electronic health records. Initially this work will focus on health outcomes for patients with COVID-19; later it will be translated to a broader range of health outcomes of interest to the FDA. Our long-term goal is to be able to leverage more data in clinical notes, ultimately uncovering more potential predictors of problems such as anaphylaxis related to vaccine or medication use. We’re also using more flexible statistical approaches such as machine learning to allow for more robust and assumption-free modeling of those associations. This lets the data drive what the model looks like. In addition to vaccine safety, we expect this NLP work may be extremely useful in research to improve treatment for COVID-19.

Q. What about studies of the COVID-19 vaccines’ actual effectiveness? How are KPWHRI researchers continuing to look into this?

Well, I think we can expect that our Vaccine and Treatment Evaluation Unit, led by Lisa Jackson, will continue to explore this question through the two phase-3 clinical trials currently underway, as well as future clinical trials. She’ll be looking at these issues, along with learning more about matters such as optimal dosage and frequency in various sub-populations. 

Meanwhile, Mike Jackson and his co-workers may help us learn more about the effectiveness of the COVID-19 vaccines through surveillance studies he’s conducting in Kaiser Permanente clinics. Mike is our lead investigator for a CDC-sponsored project that sends home-testing kits to patients who have symptoms of respiratory illness. The project was originally designed to determine the incidence of various flu strains in our population as well as the effectiveness of the annual flu vaccine. But once the COVID-19 vaccine is widely distributed, the project will also be able to determine how effectively that vaccine is working.

Q. Did you ever anticipate that you might be using your skills and knowledge as a biostatistician to help fight a worldwide pandemic? Do you have any insights about this that you’d like to share?

I am humbled that I can play a small part in the global response to this crisis. A big part of why I came to KPWHRI in the first place was the hope that I could make a positive impact in the world beyond what sometimes felt like purely academic contributions. I had no idea then that this hope would translate into such an enormous opportunity to contribute.

It’s been extremely rewarding to see so many people come together from different disciplines and types of organizations, including academia, government, and industry to rapidly share knowledge, interpret findings, and make decisions. The pace is incredible, and the generosity of health care and research communities to these efforts is awesome. I’m especially looking forward to serving as a member of Moderna’s External Safety Advisory Board for their COVID-19 vaccine program.