Statistical Methods for Detecting Anomalous Model Behavior with Unlabeled Data

Author/Creator ORCID

Date

2020-01-20

Department

Computer Science and Electrical Engineering

Program

Computer Science

Citation of Original Publication

Rights

Distribution Rights granted to UMBC by the author.
This item may be protected under Title 17 of the U.S. Copyright Law. It is made available by UMBC for non-commercial research and education. For permission to publish or reproduce, please see http://aok.lib.umbc.edu/specoll/repro.php or contact Special Collections at speccoll(at)umbc.edu

Abstract

As the applications and number of production-level machine learning models continue to increase, so too does the need for appropriate monitoring frameworks for these models. Models applied to ever-changing real world data will inevitably experience a shift in their distribution of incoming data referred to as concept drift. The quick and accurate detection of concept drift is critical to the efficient and effective use of these models. While previous approaches to solving this problem have required partially or fully labeled testing data or have focused on monitoring a single metric, we propose a model- and metric-independent approach which is able to detect concept drift in unlabeled data streams. We utilize symmetrized Kullback-Leibler divergence in combination with statistical randomization testing to provide an approach which is able to detect drift with tunable sensitivity. To demonstrate the utility of our approach, we apply to logistic regression models tested on a variety of problems using the reduced-resolution MNIST dataset from UCI [1], the National Weather Service's CF6 climate dataset [2], and Blitzer et al.'s multi-domain sentiment analysis dataset [3]. Our results show that our approach is able to reliably detect sudden drift as well as gradual drift using a sliding window approach.