Birds of a Feather

Software Testing for Scientific Computing in HPC

Authors: Keita Teranishi (Oak Ridge National Laboratory (ORNL)), Jeffrey Young (Georgia Institute of Technology), Alessandro Orso (Georgia Institute of Technology)

Abstract: Effective software testing plays a critical role in guaranteeing the performance, correctness, and reproducibility of applications and software. When it comes to testing high-performance computing (HPC) software and applications, unique requirements arise due to factors such as massive parallelism, concurrency and heterogeneity, the scale of target platforms, lack of oracles, and application-specific verification and validation techniques. In this BoF session, we aim to foster insightful discussions among a panel of expert speakers and the audience, focusing on methodologies and challenges in HPC software testing, and deepen our understanding in this crucial part of HPC software development.

Long Description: This Birds of a Feather session is focused on discussing and engaging a wider audience in topics related to software engineering with a specific focus on testing strategies for high-performance computing software and applications.

Software testing plays a crucial role in ensuring the performance, correctness, and reproducibility of software and applications. Robustness of software and applications not only improves productivity, but also improves the trust of outcomes in simulations and analyses. While significant progress has been made in methodologies, techniques, and software tools for testing in various areas of software engineering, the adoption of state-of-the-art testing practices in high-performance scientific computing lags behind. This is due to unique requirements such as floating-point arithmetic, massive parallelism, and concurrency in numerical algorithms and applications, as well as heterogeneity in system architecture, and performance portability across different node architectures and accelerators. Additionally, challenges arise from the lack of oracles, indeterminism in program execution, and emerging floating point data types driven by artificial intelligence and machine learning applications, making conventional testing methodologies less feasible. The rapid evolution of HPC systems, with advancements in system size and complexity, has outpaced the development of testing methodologies in the HPC domain. Consequently, many test suites for HPC software and application programs currently lack comprehensive test coverage and a systematic approach to verification and validation in a scalable manner.

To this end, this BoF will engage with these principles and several other key topics related to testing for HPC applications by discussing the state-of-the-art as well as common pitfalls or challenges that currently limit the usefulness of our general testing strategy. An ideal outcome of this BoF would be the discussion of key issues and some agreement by the panel and audience on which issues would be good next steps to promote in efforts like classes, tutorials and workshops, or even new research programs run by government agencies. Explicitly, we are looking to get attendees’ feedback on what techniques might be most useful to increase their engagement and usage of software testing best practices and strategies..

The BoF will be organized as an expert panel session with pre-seeded questions to start discussion and extensive general Q&A with and from the attendees. We have reached out to Sunita Chandrasekaran of University of Delaware, Manish Motwani of Georgia Tech, Shahzeb Siddiqui of NERSC, Mark Gates of University of Tennessee, and Francesco Rizzi from NexGen Analytics, and they have agreed to be panelists for this BoF. The proposed panelists and organizers are all experienced practitioners of scientific software and software engineering principles at scale. Dr. Young and Dr. Orso are also the associate director and director of Georgia Tech’s scientific software engineering center at Georgia tech funded via Schmidt Futures’s VISS Program.

We plan to incorporate online polling such as Mentimeter to try and gauge attendee interest in the key software testing concepts and challenges. Outcomes from the BoF will be shared online and through future workshops and events being planned by the organizers.

Website: https://ssecenter.cc.gatech.edu/sc-23-bof