Behind the Curtain: Comparing Predictive Models Performance in 2 Publicly Insured Populations

Abstract

Introduction: Predictive models have proliferated in the health system in recent years and have been used to predict both health services utilization and medical outcomes. Less is known, however, on how these models function and how they might adapt to different contexts. The purpose of the current study is to shed light on the inner workings of a large-scale predictive model deployed in 2 distinct populations, with a particular emphasis on adaptability issues.

Methods: We compared the performance and functioning of a predictive model of avoidable hospitalization in 2 very different populations: Medicaid and Medicare enrollees in Maryland. Specifically, we assessed characteristics of the risk scores from March 2022 for the 2 populations, the predictive ability of the scores, and the driving risk factors behind the scores. In addition, we created and assessed the performance of an “unadapted” model by applying coefficients from the Medicare model to the Medicaid population.

Results: The model adapted to, and performed well in, both populations, despite demographic differences in these 2 groups. However, the most salient risk factors and their relative weightings differed, sometimes dramatically, across the 2 populations. The unadapted Medicaid model displayed poor performance relative to the adapted model.

Conclusions: Our findings speak to the need to “peek behind the curtain” of predictive models that may be applied to different populations, and we caution that risk prediction is not “one size fits all”: for optimal performance, models should be adapted to, and trained on, the target population.