Highlight

Speeding up Neutron Data Reduction Workflows for SNS and HFIR Instruments’ Users

Comparison of Mantid’s “LoadEventNexus” wall-clock times for Mantid v5.0 release and our proposed strategy on Mantid’s latest implementation. Results are shown for “hot” cached files (accessed several times) showing universal improvements across different ORNL SNS/HFIR instrument generated raw NeXus files (CG2 is GP-SANS, CG3 is BIO-SANS, NOM is NOMAD).
Comparison of Mantid’s “LoadEventNexus” wall-clock times for Mantid v5.0 release and our proposed strategy on Mantid’s latest implementation. Results are shown for “hot” cached files (accessed several times) showing universal improvements across different ORNL SNS/HFIR instrument generated raw NeXus files (CG2 is GP-SANS, CG3 is BIO-SANS, NOM is NOMAD).

Achievement

A multidisciplinary team of researchers from the Computational Science and Mathematics Division (CSMD) at Oak Ridge National Laboratory (ORNL) were able to provide significant speed ups to data analysis reduction workflows run by instrument scientists and users at the Spallation Neutron Source (SNS) and the High Flux Isotope Reactor (HFIR) using the Mantid framework. Mantid is a data analysis and visualization open-source framework used by several neutron facilities around the world. The performed research identifies memory and input/output (I/O) bottlenecks when loading the raw event-based instrument data stored using the standard NeXus file format, while introducing new data management and algorithmic strategies on Mantid. As a result, performance measurements in the small angle neutron scattering (SANS) instruments reduction workflows: GP-SANS, BIO-SANS, and EQ-SANS show consistent 30%, 19% and 11% speed ups, respectively. While several other instrument data, CORELLI, NOMAD show significant speed ups when loading individual raw event-based NeXus data files.

Significance and Impact

SNS and HFIR instrument scientists and the facilities large number of users experience a measurable decrease in time when analyzing their raw experimental data via reduction workflows that rely on the Mantid framework. Further impact is expected for all Mantid users at other neutron science facilities (e.g. ILL in France and ISIS in the UK) using the NeXus file format, as the impact of the universal changes scale up with the metadata richness and complexity of the instrument generated raw event-based data. Addressing I/O bottlenecks is fundamental as new instruments rely on data-driven applications (machine learning) on large volumes of data and enable users in their research findings from data processing. To share our work with the broader Computer Science community, the present effort spawned a paper at the 2020 Smoky Mountains Conference; and a paper and presentation at the 1st International Workshop of Big Data Reduction held at the 2020 IEEE Big Data Conference.

Research Details

  • Problem: SNS and HFIR data reduction workflows using Mantid suffer from I/O bottlenecks as reported by instrument scientists and users.
  • Initial research was done to profile different raw event-based data bottlenecks and understand the trade-offs of different proposed data and metadata management strategies, including modifying the Mantid internal architecture.
  • The solution introduced universal changes to the Mantid framework, and it’s available in the latests version: https://github.com/mantidproject/mantid/pull/28495
  • Measurements indicate universal improvements when loading NeXus using Mantid for a variety of instrument generated files at ORNL, while small angle neutron scattering (SANS) workflows wall-clock time was reduced between 10% and 30%. Speed ups impact several Mantid users at ORNL and other facilities in the world.  
  • Results from this work were published and presented at the 2020 IEEE International Conference on Big Data

Citation and DOI

Godoy, W.F., Peterson, P., Hahn, J., Billings, J.J.: Efficient data management in neutron scattering data reduction workflows at ORNL. In: International Workshop on Big Data Reduction held with 2020 IEEE International Conference on Big Data (accepted) https://iwbdr.github.io/iwbdr20/, pre-print available: https://arxiv.org/abs/2101.02591 
 

Overview

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities, SNS and HFIR, produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data.

Last Updated: January 21, 2021 - 1:33 pm