Publication

Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale

Citation

Logan, Jeremy, Mark Ainsworth, Chuck Atkins, Jieyang Chen, Jong Choi, Junmin Gu, James Kress et al. "Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale." Data Engineering: 35.

Abstract

The Adaptable I/O System (ADIOS) represents the culmination of substantial investment in Scientific Data Management, and it has demonstrated success for several important extreme-scale science cases. However, looking towards the exascale and beyond, we see the development of yet more stringent data management requirements that require new abstractions. Therefore, there is an opportunity to attempt to connect the traditional realms of HPC I/O optimization with the Database / Data Management community. In this paper, we offer some specific examples from our ongoing work in managing data structures, services, and performance at the extreme scale for scientific computing. Using the publish/subscribe model afforded by ADIOS, we demonstrate a set of services that connect data format, metadata, queries, data reduction, and high-performance delivery. The resulting publish/subscribe framework facilitates connection to on-line workflow systems to enable the dynamic capabilities that will be required for exascale science.

Read Publication Keywords Data Management Data storage systems In Situ and In-transit Workflows Metadata Management Storage and I/O Accelerated Performance Computing High performance computing Storage and I/O User-driven Software & Systems Adaptive Workflows Informatics Systems

Last Updated: May 28, 2020 - 4:00 pm