High dimensional, robust, unsupervised record linkage

Date

2020

Department

Program

Citation of Original Publication

Bera, Sabyasachi, and Snigdhansu Chatterjee. “High Dimensional, Robust, Unsupervised Record Linkage.” Statistics in Transition New Series 21, no. 4 (2020): 123–43. https://doi.org/10.21307/stattrans-2020-034.

Rights

Attribution-NonCommercial-NoDerivatives 4.0 International CC BY-NC-ND 4.0 Deed

Subjects

Abstract

We develop a technique for record linkage on high dimensional data, where the two datasets may not have any common variable, and there may be no training set available. Our methodology is based on sparse, high dimensional principal components. Since large and high dimensional datasets are often prone to outliers and aberrant observations, we propose a technique for estimating robust, high dimensional principal components. We present theoretical results validating the robust, high dimensional principal component estimation steps, and justifying their use for record linkage. Some numeric results and remarks are also presented.