Task Allocation in Hybrid Big Data Analytics for Urban IoT Applications

Abstract

In urban Internet of Things (IoT) environments, data generated in real time could be processed by analytical applications in online or offline mode. In the management perspective of runtime environments, such modes can hardly be supported in a unified framework under multiple restrictions such as latency, utility, and QoS (quality of service). Meanwhile in the optimization perspective of specific applications, it is difficult for current infrastructure to efficiently allocate sufficient resources to tasks of an application, simultaneously considering multiple factors such as data size, velocity, and locality. In this article, two task allocation methods are proposed for batch and stream analytics to improve resource utility with auto-scaling guarantee when an analytical application is submitted or sudden workloads appear. Taking the highway domain as an example, the task allocation methods are implemented in a novel combined framework accordingly. Using both real-world and simulated data, extensive experiments show that our methods can improve utility efficiency with effective offload support.