Improving scalability and reliability of MPI-agnostic transparent checkpointing for production workloads at NERSC

Authors: Prashant Singh Chouhan, Harsh Khetawat, Neil Resnik, Jain Twinkle, Rohan Garg, Gene Cooperman, Rebecca Hartman-Baker and Zhengji Zhao

Session: Talk Session II

Time: Thursday, February 4, 2021 11:10am - 11:40am PST

Abstract - Checkpoint/restart (C/R) provides fault-tolerant computing capability, enables long running applications, and provides scheduling flexibility for computing centers to support diverse workloads with different priority. It is therefore vital to get transparent C/R capability working at NERSC. MANA, a transparent checkpointing tool, has been selected due to its MPI-agnostic and network-agnostic approach. However, originally written as a proof-of-concept code, MANA was not ready to use with NERSC’s diverse production workloads, which are dominated by MPI and hybrid MPI+OpenMP applications. In this talk, we present ongoing work at NERSC to enable MANA for NERSC’s production workloads, including fixing bugs that were exposed by the top applications at NERSC, adding new features to address system changes, evaluating C/R overhead at scale, etc. The lessons learned from making MANA production-ready for HPC applications will be useful for C/R tool developers, supercomputing centers and HPC end users alike.

Index Terms—transparent checkpointing, MANA, DMTCP, split-process, production workloads, supercomputing

Download PDF - will be available at arXive.org

Authors: