Modern workflow managers make it easier to automate, reproduce, and scale scientific analyses on high-performance computing (HPC) platforms. In this Primer*, we'll introduce management software like Snakemake and Nextflow and discuss their role in HPC environments. We will then demonstrate how to use persistence1, a dedicated CURC node for running workflow management software that addresses a common challenge: the need to run managers for weeks or even months outside of HPC job schedulers and their associated wall clock limits.
*What is a Primer? A Primer is a session that provides an introduction to concepts, systems, and tools. You can expect to leave a Primer with an introductory-level understanding of the topic.