Performance Evaluation of Minimum Average Deviance Estimation in High Dimensional Poisson Regression

Abstract

The second most expensive part of the 2010 Decennial Census was Address Canvassing (AdCan), a eld operation to prepare the Master Address File (MAF) for census day. The MAF is a database of households in the United States maintained by the US Census Bureau and is used as a basis for the census and household surveys that it conducts. Motivated by the importance of the MAF and the cost of a large scale AdCan operation, there is an interest to use statistical methodologies to explain MAF errors discovered during canvassing. Ideally, statistical models could be used to predict future errors and assist with updating of the MAF. A major challenge in constructing a MAF error model is that important predictor variables associated with MAF errors are not known. Some recent works at Census Bureau have carried out variable selection using a collection of data sources, treating counts of errors per census block as the outcome. It may be possible to use dimension reduction methodologies to obtain count models with much lower dimensional predictors. Adragni et al. [4] proposed a methodology called Minimum Average Deviance Estimation (MADE), which is based on the concept of local regression and embeds sufficient dimension reduction of the predictors. MADE assumes a forward regression with the response variable following an exponential family distribution, such as Poisson for counts. The goal of this project is to evaluate the performance of MADE on large data sets using simulations. We parallelized several snippets of the MADE source code to improve its performance and compare the speed up of these parallelized snippets with their serial alternatives. Simulated data sets with increasing dimensions are used to evaluate the run time. A limited stress test is performed to determine the extent of problem size that MADE can handle on maya, a high performance computing cluster at UMBC. These tests allow us to evaluate the capabilities of MADE to scale to large data sets, such as the AdCan modeling problem.