Birds of a Feather

Interactive and Urgent HPC

Authors: Albert Reuther (Massachusetts Institute of Technology (MIT), Lincoln Laboratory), Nick Brown (Edinburgh Parallel Computing Centre (EPCC)), William Arndt (Lawrence Berkeley National Laboratory (LBNL))

Abstract: Many HPC systems are managed using batch queues; however, not all HPC applications and workflows are best served by batch queue systems. Interactive prototyping, urgent streaming data analysis, application steering, and in-situ visualization are among the workflows that require interactive and urgent capabilities to be effective. After three successful SC BoFs and seven successful workshops at SC and ISC, the interactive and urgent HPC community is writing a position paper during the summer of 2023 to document progress and cast future research foci. In this BoF, we will present the state of the draft paper and solicit discussion and feedback.

Long Description: The traditional scientific computing workflow is driven by external constraints, particularly as time available on a shared resource, rather than by the requirements of the scientists’ agendas. Interactivity enables users to exploit HPC resources in new and revolutionary ways. Interactive prototyping, urgent streaming data analysis, application steering, and in-situ visualization are among the workflows that require interactive and urgent capabilities to be effective.

A community of domain scientists, tool developers, and supercomputer operators interested in the interactive and urgent use of supercomputing systems has been growing through a series of three successful SC BoFs (2015, 2017, 2018) and seven successful workshops at SC (2019, 2020, 2022) and ISC (2018, 2019, 2022, 2023). In these workshops, a variety of topics have been covered including high productivity languages for rapid prototyping; scheduling policies; workflow integration and optimization; user education; and many impactful case studies.

Much progress has been made in bringing interactive and urgent HPC capabilities online at many supercomputing centers across the world. At this point in the progress, the Interactive and Urgent HPC community has decided to write a position paper to document the progress that has been made so far and to identify research areas and questions that need to be addressed going forward. The contributors to this paper are from a variety of supercomputing centers from across the world, and they bring to the effort many interesting backgrounds, experiences, and ideas. A draft of the paper is expected to be complete several weeks before SC23, and this BoF will be our primary avenue for getting feedback from the greater interactive and urgent HPC community. That feedback will be incorporated in the position paper, which will subsequently be published.

Website: https://www.interactivehpc.com/