Provenance for MapReduce-based data-intensive workflows
| dc.contributor.author | Crawl, Daniel | |
| dc.contributor.author | Wang, Jianwu | |
| dc.contributor.author | Altintas, Ilkay | |
| dc.date.accessioned | 2024-02-19T15:02:52Z | |
| dc.date.available | 2024-02-19T15:02:52Z | |
| dc.date.issued | 2011-11-14 | |
| dc.description | C '11: International Conference for High Performance Computing, Networking, Storage and Analysis Seattle Washington USA 14 November 2011 | |
| dc.description.abstract | MapReduce has been widely adopted by many business and scientific applications for data-intensive processing of large datasets. There are increasing efforts for workflows and systems to work with the MapReduce programming model and the Hadoop environment including our work on a higher-level programming model for MapReduce within the Kepler Scientific Workflow System. However, to date, provenance of MapReduce-based workflows and its effects on workflow execution performance have not been studied in depth. In this paper, we present an extension to our earlier work on MapReduce in Kepler to record the provenance of MapReduce workflows created using the Kepler+Hadoop framework. In particular, we present: (i) a data model that is able to capture provenance inside a MapReduce job as well as the provenance for the workflow that submitted it; (ii) an extension to the Kepler+Hadoop architecture to record provenance using this data model on MySQL Cluster; (iii) a programming interface to query the collected information; and (iv) an evaluation of the scalability of collecting and querying this provenance information using two scenarios with different characteristics. | |
| dc.description.sponsorship | The authors would like to thank the rest of the Kepler team for their collaboration. This work was supported by NSF SDCI Award OCI-0722079 for Kepler/CORE and ABI Award DBI-1062565 for bioKepler, DOE SciDAC Award DE-FC02-07ER25811 for SDM Center, the UCGRID Project, and an SDSC Triton Research Opportunities grant. | |
| dc.description.uri | https://dl.acm.org/doi/10.1145/2110497.2110501 | |
| dc.format.extent | 9 pages | |
| dc.genre | conference papers and proceedings | |
| dc.genre | presentations (communicative events) | |
| dc.identifier | doi:10.13016/m2anpc-c4tr | |
| dc.identifier.citation | Crawl, Daniel, Jianwu Wang, and Ilkay Altintas. “Provenance for MapReduce-Based Data-Intensive Workflows.” In Proceedings of the 6th Workshop on Workflows in Support of Large-Scale Science, 21–30. WORKS ’11. New York, NY, USA: Association for Computing Machinery, 2011. https://doi.org/10.1145/2110497.2110501. | |
| dc.identifier.uri | https://doi.org/10.1145/2110497.2110501 | |
| dc.identifier.uri | http://hdl.handle.net/11603/31655 | |
| dc.language.iso | en_US | |
| dc.publisher | ACM | |
| dc.relation.isAvailableAt | The University of Maryland, Baltimore County (UMBC) | |
| dc.relation.ispartof | UMBC Information Systems Department Collection | |
| dc.relation.ispartof | UMBC Center for Accelerated Real Time Analysis | |
| dc.relation.ispartof | UMBC Data Science | |
| dc.relation.ispartof | UMBC Joint Center for Earth Systems Technology (JCET) | |
| dc.relation.ispartof | UMBC Center for Real-time Distributed Sensing and Autonomy | |
| dc.rights | This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author. | |
| dc.subject | MapReduce | |
| dc.subject | Provenance | |
| dc.subject | Scientific Workflows | |
| dc.subject | UMBC Big Data Analytics Lab | |
| dc.title | Provenance for MapReduce-based data-intensive workflows | |
| dc.type | Text | |
| dcterms.creator | https://orcid.org/0000-0002-9933-1170 |