A Data Intensive Statistical Aggregation Engine: A Case Study for Gridded Climate Records

Thumbnail Image

Files

gre_cloudflow13.pdf (1.27 MB)

Links to Files

https://ieeexplore.ieee.org/document/6651122

Permanent Link

https://doi.org/10.1109/IPDPSW.2013.87
 http://hdl.handle.net/11603/30411

Collections

UMBC Computer Science and Electrical Engineering Department
UMBC Faculty Collection
UMBC Student Collection

Author/Creator

Author/Creator ORCID

https://orcid.org/0000-0002-4862-8396

Date

2013-10-31

Type of Work

conference papers and proceedings
preprints
Text

Department

Program

Citation of Original Publication

Chapman, David, Tyler A. Simon, Phuong Nguyen, and Milton Halem. “A Data Intensive Statistical Aggregation Engine: A Case Study for Gridded Climate Records.” In 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum, 2157–64, 2013. https://doi.org/10.1109/IPDPSW.2013.87.

Rights

© 2013 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Subjects

Abstract

Satellite derived climate instrument records are often highly structured and conform to the "Data-Cube" topology. However, data scales on the order of tens to hundreds of Terabytes make it more difficult to perform the rigorous statistical aggregation and analytics necessary to investigate how our climate is changing over time and space. It is especially cumbersome to supply the full derivation (provenance) of this analysis, as is increasingly required by scientific conferences and journals. In this paper, we address our approach toward the creation of a 55 Terabyte decadal record of Outgoing Long wave Spectrum (OLS) from the NASA Atmospheric Infrared Sounder (AIRS), and describe our open source data-intensive statistical aggregation engine "Gridderama" intended primarily for climate trend analysis, and may be applicable to other aggregation problems involving large structured datasets.