Project Status: Active

Parallel Aggregate Persistent Storage

Papyrus is a programming system that provides features for scalable, aggregate, persistent memory. Papyrus provides a portable and scalable programming interface to access and manage parallel data structures on the distributed nonvolatile memory (NVM) storage. Papyrus consists of three components, virtual file system, C++ templates, and key-value store. Papyrus is portable across node-local NVM architecture and dedicated NVM architecture. Papyrus runs in a distributed fashion without any centralized control that may become a significant bottleneck in large-scale systems. Since Papyrus is implemented as a user-level library using MPI, and it does not require any type of system-level daemon or server, it runs on most existing HPC system.


  • Papyrus allows the programmers to exploit large aggregate NVM space in the system without handling complex communication, synchronization, replication, and consistency models.
  • Papyrus Virtual File System (VFS) provides a uniform aggregate NVM storage image across diverse NVM architectures. PapyrusVFS implements a subset of POSIX API related to file I/O.
  • Papyrus Template Container Library (TCL) provides a high-level container programming interface whose data elements can be distributed to multiple NVM nodes. PapyrusTCL provides three containers, including map, vector, and matrix, implemented as C++ templates.
  • Papyrus Key-Value Store (KV) stores keys and values in arbitrary byte arrays across multiple NVMs. PapyrusKV provides advanced features designed for HPC such as configuring consistency, zero-copy workflow, and asynchronous checkpoint/restart.



  • Jungwon Kim and Jeffrey S. Vetter. Implementing Efficient Data Compression and Encryption in a Persistent Key-Value Store for HPC. The International Journal of High Performance Computing Applications, 2019 (IJHPCA'19).
  • Jungwon Kim, Seyong Lee, and Jeffrey S. Vetter. PapyrusKV: A High-Performance Parallel Key-Value Store for Distributed NVM Architectures. The International Conference for High Performance Computing, Networking, Storage and Analysis, November 2017 (SC '17).
  • Jungwon Kim, Kittisak Sajjapongse, Seyong Lee, and Jeffrey S. Vetter. Design and Implementation of Papyrus: Parallel Aggregate Persistent Storage. IEEE International Parallel and Distributed Processing Symposium, May 2017 (IPDPS '17)

Last Updated: February 25, 2021 - 3:17 pm