A Scalable Data Science Workflow Approach for Big Data Bayesian Network Learning
Loading...
Links to Files
Author/Creator
Author/Creator ORCID
Date
2015-11-09
Type of Work
Department
Program
Citation of Original Publication
J. Wang, Y. Tang, M. Nguyen and I. Altintas, "A Scalable Data Science Workflow Approach for Big Data Bayesian Network Learning," 2014 IEEE/ACM International Symposium on Big Data Computing, London, UK, 2014, pp. 16-25, doi: 10.1109/BDC.2014.10.
Rights
© 2014 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale
or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Subjects
Abstract
In the Big Data era, machine learning has more potential to discover valuable insights from the data. As an important machine learning technique, Bayesian Network (BN) has been widely used to model probabilistic relationships among variables. To deal with the challenges of Big Data PN learning, we apply the techniques in distributed data-parallelism (DDP) and scientific workflow to the BN learning process. We first propose an intelligent Big Data pre-processing approach and a data quality score to measure and ensure the data quality and data faithfulness. Then, a new weight based ensemble algorithm is proposed to learn a BN structure from an ensemble of local results. To easily integrate the algorithm with DDP engines, such as Hadoop, we employ Kepler scientific workflow to build the whole learning process. We demonstrate how Kepler can facilitate building and running our Big Data BN learning application. Our experiments show good scalability and learning accuracy when running the application in real distributed environments.