Publication

Querying Large Scientific Data Sets with Adaptable IO System ADIOS

Citation

Junmin Gu, Scott Klasky, Norbert Podhorszki, Ji Qiang, Kesheng Wu
Querying Large Scientific Data Sets with Adaptable IO System ADIOS, Asian Conference on Supercomputing Frontiers (SCFA) 2018: Supercomputing Frontiers pp 51-69

Abstract

In this paper, we design a query interface for ADIOS to allow arbitrary combinations of range conditions on known variables, implement a number of different mechanisms for resolving these selection conditions, and devise strategies to reduce the time needed to retrieve the scattered data records. In many cases, the query mechanism can retrieve the selected data records orders of magnitude faster than the brute-force approach. Our work relies heavily on the in situ data processing feature of ADIOS to allow user functions to be executed in the data transport pipeline. This feature allows us to build indexes for efficient query processing, and to perform other intricate analyses while the data is in memory.

Read Publication