Predicting and preventing missed clinic visits

Biostatistician Yates Coley reports on new predictive analytics work that’s decreasing missed visits at KP Washington

 By Yates Coley, PhD, an assistant biostatistics investigator at Kaiser Permanente Washington Health Research Institute (KPWHRI) and predictive analytics lead for the institute’s Center for Accelerating Care Transformation (ACT Center)

Missed clinic visits limit appointment availability and result in lost opportunities for patients to receive care. Kaiser Permanente Washington recently partnered with the advanced analytics team at KPWHRI’s ACT Center on a learning health system project to predict and prevent missed visits. Our work involved using predictive analytics to validate a risk model — and then pairing it with a simple, evidence-based intervention: 2 reminder text messages instead of 1.

Based on results of our recent comparative effectiveness study in The Permanente Journal, the model and intervention led to a significant drop in missed visits identified as high risk: 7% in Primary Care and 11% in Mental Health and Wellness. Each month, that translates into about 126 fewer missed primary care visits and 106 fewer missed mental health visits at Kaiser Permanente Washington. It’s a big improvement from a relatively inexpensive and low-touch intervention — just an extra text message.

This innovation has already been scaled to more than a dozen other service lines across Kaiser Permanente Washington, including Cardiology, Oncology, and Physical/Occupational Therapy. And with ongoing support from the ACT Center, our work to improve the risk prediction model continues so that we can grow the positive impact on patients and care teams.

Using research capabilities to answer pressing health system questions

The work behind these promising results started back in 2018 and is a partnership between the ACT Center and Kaiser Permanente Washington’s Patient Access Business Operations (PABO) department. PABO is the team that manages appointment scheduling and reminders across our health system’s many service lines.

PABO first approached the ACT Center’s advanced analytics team because they were interested in using predictive analytics to help predict and prevent missed clinic visits. They knew that Epic — Kaiser Permanente’s electronic health record (EHR) — had a built-in, proprietary model for predicting missed visits. But they weren't sure about the model’s accuracy or what effective interventions they should consider for patients identified as high risk.

So that led us to some specific questions that ACT Center capabilities could help answer:

1. How well does Epic’s missed visits model perform for visits at Kaiser Permanente Washington? And would it be possible for our team to design a model that performed better?
2. What interventions are known to be effective in preventing missed visits
3. After selecting a model and intervention to prevent missed visits, what difference does it make?

Starting with good and aiming for better

We started by validating Epic’s missed visits prediction model among clinic visits across Kaiser Permanente Washington and found that the model was moderately accurate. Specifically, using Epic’s model among primary care visits scheduled 4 or more days in advance resulted in a 67% chance that a missed visit had a higher predicted risk score than a visit that wasn't missed. In analytics terminology, this is what we call “the area under the curve.” Among primary care visits in the top 40th percentile of risk, Epic’s model accurately identified 64% of eventual missed visits — which is referred to as the “sensitivity” of the risk score.

The ACT Center’s advanced analytics team knew we could likely develop a risk prediction model that performed better because of 2 key strengths in this area: our data and our methods.

First, our team is uniquely familiar with data that enables us to identify more and stronger predictors of the outcome of interest. To put it simply, better data leads to better predictions. In particular, my teammate Sharon Fuller was able to gather rich predictor information from visit and member characteristics available in our clinical and operational data sets. Sharon is a programmer whose work spans research at KPWHRI and business intelligence at Kaiser Permanente Washington, which gives her deep expertise in both types of data.

Second, our team had the opportunity to work with Ernesto Ulloa-Pérez, MS, who is the first author on our paper, and Adam Elder, MS. Ernesto and Adam are PhD students from the University of Washington's Biostatistics program whose expertise allowed us to implement more advanced statistical methods to use this massive data set to generate strong predictions. The result was a homegrown predictive model for missed visits that performed significantly better than Epic’s model — with an area under the curve of 78% and higher sensitivity.

Next, our team led the administrative process to purchase the Epic license needed to implement our own prediction models. And now we’re working with Kristine Rogers, KPWHRI’s Epic programmer, to build our new-and-improved model into Kaiser Permanente Washington’s EHR systems.

Learning from — and contributing to — evidence on what works to improve outcomes

While this work continues, members and care teams across Kaiser Permanente Washington are already benefitting from the use of Epic’s built-in model for predicting missed visits. But when it comes to improving outcomes, the model is only half the story. To make a difference, a risk model needs to be paired with an intervention that’s been proven effective for the population identified as high risk.

KPWHRI Collaborative Scientist Paula Blasi, MPH, led the ACT Center’s work to understand the evidence base for interventions to prevent missed visits — conducting a rapid literature review that suggested a second reminder message could be effective. This low-cost, low-touch intervention was of great interest to our partners in PABO. So, in early 2019, they implemented a second reminder text for patients in the 40th percentile of Epic’s risk model for missed visits in primary care and mental health. This led to scaling the model and intervention across services lines and to the promising results we recently published in The Permanente Journal.

One key strength of our study is that, over about 7 months in 2019, we were able to randomize tens of thousands of primary care and mental health visits to either 1 or 2 text message reminders. This speaks to the quality of our collaboration with the PABO team, who were open to learning what works best for patients and care teams. They were also committed to doing right by our patients, which meant they didn't want to charge them for missed visits. And they didn't want to overbook visits since that can be stressful for providers and patients alike.

Their openness to randomization made our comparative effectiveness work that much easier, and their partnership has benefitted our methods work. PABO has given us permission to use data from this project for statistical research to improve methods for developing and evaluating prediction models, which I really appreciate. Our ACT Center team will also be able to use data and insights from this project to inform how we develop and deploy other risk prediction models at Kaiser Permanente Washington — and how we evaluate their impact. It’s exciting to know that we’ve created a partnership and data set that will be valuable for years to come.

Dr. Coley's co-authors on this study include Ernesto Ulloa-Pérez, MS, from the Department of Biostatistics at the University of Washington School of Public Health, and the ACT Center's Paula Blasi, MPH; Emily Westbrook, MHA; Paula Lozano, MD, MPH; and Katie Coleman, MSPH.​​​

This was adapted from a story originally published by the ACT Center.