An effective online data monitoring and saving strategy for large-scale climate simulations

Citation

Xiaochen Xian, Rick Archibald, Benjamin Mayer, Kaibo Liu, Jian Li, "An effective online data monitoring and saving strategy for large-scale climate simulations", Quality Technology & Quantitative Management, 2018

Abstract

Large-scale climate simulation models have been developed and widely used to generate historical data and study future climate scenarios. These simulation models often have to run for a couple of months to understand the changes in the global climate over the course of decades. This long-duration simulation process creates a huge amount of data with both high temporal and spatial resolution information; however, how to effectively monitor and record the climate changes based on these large-scale simulation results that are continuously produced in real time still remains to be resolved. Due to the slow process of writing data to disk, the current practice is to save a snapshot of the simulation results at a constant, slow rate although the data generation process runs at a very high speed. This paper proposes an effective online data monitoring and saving strategy over the temporal and spatial domains with the consideration of practical storage and memory capacity constraints. Our proposed method is able to intelligently select and record the most informative extreme values in the raw data generated from real-time simulations in the context of better monitoring climate changes.

Read Publication