Oak Ridge OpenSHMEM Benchmark Suite

Oak Ridge OpenSHMEM Benchmark (OSB) Suite
Figure 1: Scaling example of multithreaded SSCA1 and GUPS benchmarks run on Linux cluster using ORNL developed OpenSHMEM-X.


The paper, titled “Oak Ridge OpenSHMEM Benchmark (OSB) Suite”, was presented at the 5th OpenSHMEM Workshop in Baltimore Maryland.

Significance and Impact

The paper was accepted and presented at the workshop, which described the current set of benchmarks and mini-applications/kernels available in the OpenSHMEM Benchmark (OSB) suite. The paper describes the intended use of the suite, discusses benchmark behavior, and provides details on enhancements for the OpenSHMEM specific version of the codes.  The paper also details on the three benchmarks that have a multithreaded variant (Graph500, SSCA1 and GUPS), which leverages the multithreaded support added in the latest OpenSHMEM v1.4 specification.

Research Details

  • A detailed description of every benchmark and kernel with special focus on how OpenSHMEM is used in them. The OSB suite currently includes: Graph500, SSCA1, NPB, GUPS and SHOMS.
  • Details are given regarding the enhancements made to extend the OpenSHMEM enabled benchmarks to provide multithreaded variants.
  • Usage information for the the benchmarks, and a description of the output from the benchmarks and how to interpret it.
  • Sample results using different OpenSHMEM implementations on a Linux development cluster (“Turing”) and a Cray XC30 supercomputer (“EOS”).


The assessment of application performance is a fundamental task in high-performance computing (HPC). The OpenSHMEM Benchmark (OSB) suite is a collection of micro-benchmarks and mini-applications/compute kernels that have been ported to use OpenSHMEM. Some, like the NPB OpenSHMEM benchmarks, have been published before while most others have been used for evaluations but never formally introduced or discussed. This suite puts them together and is useful for assessing the performance of different use cases of OpenSHMEM. This offers system implementers a useful means of measuring performance and assessing the effects of new features as well as implementation strategies.  The suite is also useful for application developers to assess the performance of the growing number of OpenSHMEM implementations that are emerging. In this paper, we describe the current set of codes available within the OSB suite, how they are intended to be used, and, where possible, a snapshot of their behavior on one of the OpenSHMEM implementations available to us.  We also include detailed descriptions of every benchmark and kernel, focusing on how OpenSHMEM was used. This includes details on the enhancements we made to the benchmarks to support multithreaded variants.  We encourage the OpenSHMEM community to use, review, and provide feedback on the benchmarks.

The OSB suite currently includes the following benchmarks:

  •  Graph500 (single & multithreaded variants) – Breadth-First Search (BFS) on large undirected graphs
  •  SSCA1 (single & multithreaded variants) – Sequence alignment algorithm with gap scoring that compares characters in text strings for matches (score)
  •  NPB (single threaded variant) – Mini-applications & scientific compute kernels (IS: Integer Sort, MG: Multi-Grid, BT: Block Tri-diagonal CFD solver, SP: Scalar Penta-diagonal CFD solver)
  •  GUPS (single & multithreaded variants) – Random memory access benchmark
  •  SHOMS (single threaded variant) – OpenSHMEM micro-benchmark testing suite.

The OSB suite is available at: https://github.com/ornl-languages/osb