Project Status: Active
Node-local burst buffers are becoming an indispensable hardware resource on large-scale supercomputers to buffer the bursty I/O from scientific applications. However, there is a lack of software support for burst buffers to be efficiently shared by applications within a batch-submitted job and recycled across different batch jobs. In addition, burst buffers need to cope with a variety of challenging I/O patterns from data-intensive scientific applications.
UnifyFS is a user-level burst buffer file system under active development. UnifyFS supports scalable and efficient aggregation of I/O bandwidth from node-local burst buffers while having the same life cycle as a batch-submitted job. UnifyFS supports both N-N and N-1 write/read using a new distributed consistency model called lamination. It efficiently accelerates scientific I/O based on scalable metadata indexing, co-located I/O delegation, and server-side read clustering and pipelining.
For more information, or the current software and documentation, visit https://github.com/LLNL/UnifyFS
Last Updated: March 11, 2021 - 4:00 pm