Reproducible and Portable Big Data Analytics in the Cloud

Thumbnail Image

Files

Reproducible_and_portable_app_in_cloud-main.zip (1.2 MB)
Reproducible_and_Portable_Big_Data_Analytics_in_the_Cloud.pdf (3.47 MB)

Links to Files

https://ieeexplore.ieee.org/document/10045022

Permanent Link

http://hdl.handle.net/11603/24072
 https://doi.org/10.1109/TCC.2023.3245081

Collections

UMBC Information Systems Department
UMBC Computer Science and Electrical Engineering Department
UMBC Faculty Collection
UMBC Joint Center for Earth Systems Technology (JCET)
UMBC Student Collection

Author/Creator

Author/Creator ORCID

https://orcid.org/0000-0002-1665-8398
 https://orcid.org/0000-0002-7553-7932
 https://orcid.org/0000-0002-9933-1170

Date

02-15-2023

Type of Work

journal articles
Text

Department

Program

Citation of Original Publication

Wang, Xin, Pei Guo, Xingyan Li, Aryya Gangopadhyay, Carl E. Busart, Jade Freeman, and Jianwu Wang. “Reproducible and Portable Big Data Analytics in the Cloud.” IEEE Transactions on Cloud Computing 11, no. 3 (July 2023): 2966–82. https://doi.org/10.1109/TCC.2023.3245081.

Rights

This work was written as part of one of the author's official duties as an Employee of the United States Government and is therefore a work of the United States Government. In accordance with 17 U.S.C. 105, no copyright protection is available for such works under U.S. Law.
Public Domain Mark 1.0

Subjects

Abstract

Cloud computing has become a major approach to help reproduce computational experiments. Yet there are still two main difficulties in reproducing batch based Big Data analytics (including descriptive and predictive analytics) in the cloud. The first is how to automate end-to-end scalable execution of analytics including distributed environment provisioning, analytics pipeline description, parallel execution, and resource termination. The second is that an application developed for one cloud is difficult to be reproduced in another cloud, a.k.a. vendor lock-in problem. To tackle these problems, we leverage serverless computing and containerization techniques for automated scalable execution and reproducibility, and utilize the adapter design pattern to enable application portability and reproducibility across different clouds. We propose and develop an open-source toolkit that supports 1) fully automated end-to-end execution and reproduction via a single command, 2) automated data and configuration storage for each execution, 3) flexible client modes based on user preferences, 4) execution history query, and 5) simple reproduction of existing executions in the same environment or a different environment. We did extensive experiments on both AWS and Azure using four Big Data analytics applications that run on virtual CPU/GPU clusters. The experiments show our toolkit can achieve good execution performance, scalability, and efficient reproducibility for cloud-based Big Data analytics.