A Comparison of Big Data Application Programming Approaches: A Travel Companion Case Study

Author/Creator ORCID

Date

2017

Department

Program

Citation of Original Publication

P. Guo, J. Wang and Z. Chen, "A comparison of big data application programming approaches: A travel companion case study," 2017 IEEE International Conference on Big Data (Big Data), Boston, MA, 2017, pp. 2869-2878. DOI: 10.1109/BigData.2017.8258255

Rights

This item may be protected under Title 17 of the U.S. Copyright Law. It is made available by UMBC for non-commercial research and education. For permission to publish or reproduce, please contact the author.

Abstract

With advances of big data technologies, there are many possible ways to program for each big data application. A challenge is to know the differences of the program approaches and decide which programming approach is the best for a particular big data application. In this paper, we use vehicle travel companion as a case study to explore four different programming approaches, including Spark RDD (with GroupBy or Join), Spark SQL with Hive and Hive on Hadoop, and tune the programmed big data applications. Our experiments show that the execution time of one programming approach could be more than 100-fold longer than that of another for the same application logic, which verifies that programming approach decision is important. We also explain the reasons for the differences. The findings could be applied to the selection of programming approach for other big data applications.