A Framework for Distributed Data-Parallel Execution in the Kepler Scientific Workflow System
Author/Creator
Author/Creator ORCID
Date
Department
Program
Citation of Original Publication
Wang, Jianwu, Daniel Crawl, and Ilkay Altintas. “A Framework for Distributed Data-Parallel Execution in the Kepler Scientific Workflow System.” Procedia Computer Science, Proceedings of the International Conference on Computational Science, ICCS 2012, 9 (January 1, 2012): 1620–29. https://doi.org/10.1016/j.procs.2012.04.178.
Rights
This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.
Attribution-NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0 DEED)
Attribution-NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0 DEED)
Subjects
Abstract
Distributed Data-Parallel (DDP) patterns such as MapReduce have become increasingly popular as solutions to facilitate data-intensive applications, resulting in a number of systems supporting DDP workflows. Yet, applications or workflows built using these patterns are usually tightly-coupled with the underlying DDP execution engine they select. We present a framework for distributed data-parallel execution in the Kepler scientific workflow system that enables users to easily switch between different DDP execution engines. We describe a set of DDP actors based on DDP patterns and directors for DDP workflow executions within the presented framework. We demonstrate how DDP workflows can be easily composed in the Kepler graphic user interface through the reuse of these DDP actors and directors and how the generated DDP workflows can be executed in different distributed environments. Via a bioinformatics usecase, we discuss the usability of the proposed framework and validate its execution scalability.