Behind the Curtain: Comparing Predictive Models Performance in 2 Publicly Insured Populations

Date

2024-11

Department

Program

Citation of Original Publication

Sun, Ruichen, Morgan Henderson, Leigh Goetschius, Fei Han, and Ian Stockwell. "Behind the Curtain: Comparing Predictive Models Performance in 2 Publicly Insured Populations." Medical Care 62, no. 11 (November 2024): 716. https://doi.org/10.1097/MLR.0000000000002050.

Rights

This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.

Abstract

Introduction: Predictive models have proliferated in the health system in recent years and have been used to predict both health services utilization and medical outcomes. Less is known, however, on how these models function and how they might adapt to different contexts. The purpose of the current study is to shed light on the inner workings of a large-scale predictive model deployed in 2 distinct populations, with a particular emphasis on adaptability issues. Methods: We compared the performance and functioning of a predictive model of avoidable hospitalization in 2 very different populations: Medicaid and Medicare enrollees in Maryland. Specifically, we assessed characteristics of the risk scores from March 2022 for the 2 populations, the predictive ability of the scores, and the driving risk factors behind the scores. In addition, we created and assessed the performance of an “unadapted” model by applying coefficients from the Medicare model to the Medicaid population. Results: The model adapted to, and performed well in, both populations, despite demographic differences in these 2 groups. However, the most salient risk factors and their relative weightings differed, sometimes dramatically, across the 2 populations. The unadapted Medicaid model displayed poor performance relative to the adapted model. Conclusions: Our findings speak to the need to “peek behind the curtain” of predictive models that may be applied to different populations, and we caution that risk prediction is not “one size fits all”: for optimal performance, models should be adapted to, and trained on, the target population.