Scalable Multivariate Causality Discovery From Large-Scale Global Spatiotemporal Climate Data

Abstract

The study of causality investigates cause-effect relationships among different variables of a system and has been widely researched in climatology. To discover causal relationships from time-series datasets, many data-driven causality discovery methods, e.g., Granger causality, PCMCI and Dynamic Bayesian Network, have been proposed. Most of the existing approaches face computing challenges when they are used to discover causality from the explosion of available data with increasing dimensionality. These causality discovery approaches mine time series data and generate a directed causality graph where each graph edge denotes a cause-effect relationship between the two connected graph nodes, yet their results differ from other algorithms in most cases. Furthermore, there is ever-increasing available climate data, which makes it more and more difficult to utilize existing causality discovery algorithms and technologies to generate causality results within reasonable time and budget. Three main challenges in discover causality from the large-scale and complex climate observation and simulation datasets are computing complexity, results uncertainty and reproducibility. To deal with computation complexity, we design and implement a new incremental parallel gradient boosting causality discovery method to address the challenge of learning non-linear and hybrid climate data with increasing data size and dimensionality. To deal with the challenge of uncertainty, which indicates the different results in various existing data-driven causality discovery methods, a hybrid model ensemble framework utilizing current existing data partitioning and ensemble techniques is proposed to generate more accurate and more stable results. Finally, to achieve reproducibility, we develop and deploy causality-as-a-service on AWS cloud for researchers to achieve user-friendly and budget-friendly when dealing with causality discovery on large-scale time-series data on cloud. Our experiments and results with both synthetic and real-world datasets show that the proposed methods are effective and efficient solutions to the challenges.