14,295 Attendees

438 Exhibitors

Notably, this year drew a record-breaking 14,295 attendees, reflecting the world’s ever-expanding interest in HPC. The exhibit floor was also packed with 438 exhibitors, also an SC record, of which 90 were entirely new to the conference.

This convergence of all things computing was exemplified by the SC23 theme, “I Am HPC,” which served to remind everyone that they are an equal and welcome contributor to this thriving HPC community. As the SC23 General Chair, Dorian Arnold put it, the theme “aims to humanize the field of High Performance Computing and ensure it’s accessible to all.”

Post-Conference Resources

Speaker Highlights 

An Inspirational Keynote

Hakeem Oluseyi’s keynote speech was especially relevant to this year’s theme of “I Am HPC.” Oluseyi’s life story provided a shining example of how intelligence and hard work can overcome adversity. Moreover, Oluseyi posed the notion that while receiving resources necessary to succeed can be needlessly difficult for so many people, it often only takes one outstretched hand to make a difference.

Opening Plenary Brilliance

The opening plenary panel was hosted by Valerie Taylor of Argonne National Laboratory. Taylor moderated a discussion, I Am HPC: The Impact and Future Directions of HPC, featuring Kathy Yelick of the University of California Berkeley, Mateo Valero of the Barcelona Supercomputing Center, Enobong (Anna) Branch of Rutgers University, and Frank Indiviglio of the National Oceanic and Atmospheric Administration.

While technology was up for discussion, the group also addressed issues concerning the STEM pipeline and its reach within underrepresented sectors of society and the daily impacts HPC has, including in areas such as biomedicine and drug design, threat assessment, and climate modeling.

SCinet Delivers

WHen Powerful Matters

SCinet is created annually for use by exhibitors and attendees at the SC conference. Since this network must be able to perform extraordinary feats, such as the 400G connectivity required by AmLight for a research experiment, SCinet is the fastest network in the world for the week of the SC conference.

SCinet by the numbers

This year, SCinet required 66,814 feet of fiber optic cable, around two times Mount Everest’s height. The network was capable of 6.71 terabits per second, allowing someone to download Netflix’s entire content library in about an hour. Every piece of equipment in the network is donated, and this year’s total retail price for the donated equipment used to drive SC23 was approximately $40 million.

Awards Spotlight

Society Awards

ACM Gordon Bell Prize

“Large-scale Materials Modeling at Quantum Accuracy: Ab Initio Simulations of Quasicrystals and Interacting Extended Defects in Metallic Alloys” (University of Michigan, Indian Institute of Science, ORNL)

ACM Gordon Bell Prize for Climate Modelling

“The Simple Cloud-Resolving E3SM Atmosphere Model Running on the Frontier Exascale System” (SNL, LLNL, LBNL, ANL, ORNL, HPE, PNNL)

ACM Student Research Competition (Undergraduate)

Ronak Singh Monga (Indiana University & LBNL)

ACM Student Research Competition (Graduate)

Jiajun Huang (University of California Riverside)

ACM SIGHPC Outstanding Doctoral Dissertation Award

Keren Zhou (Rice University)

ACM SIGHPC Emerging Woman Leader in Technical Computing Award

Amanda Randles (Duke University)

ACM SIGHPC Fellowships in Computational and Data Science

Jeremy Williams (KTH Royal Institute of Technology), Alyssa Smolensky (University of Missouri), Van Truong (University of Pennsylvania), Lauren Ostrowski (University of California San Diego), Michelle Garcia (Dartmouth College), Beauty Chabuka (Florida State University)

ACM/IEEE-CS George Michael Memorial HPC Fellowships

J. Gregory Pauloski (University of Chicago), Rohan Basu Roy (Northeastern University)

ACM/IEEE-CS Ken Kennedy Award

Keshav Pingali (University of Texas at Austin)

IEEE-CS Sidney Fernbach Award

Manish Parashar (Scientific Computing and Imaging (SCI) Institute and University of Utah)

IEEE-CS TCHPC Award for Excellence for Early Career Researchers in High Performance Computing

Johannes Doerfert (Lawrence Livermore National Laboratory), Wengian Dong (Florida International University), Prashant Pandey (University of Utah)

SC Awards

Test of Time Award

Shaw DE, RO Dror, JK Salmon, et al. 2009. “Millisecond-scale molecular dynamics simulations on Anton”

Best Paper Award

“PanguLU: A Scalable Regular Two-Dimensional Block-Cyclic Sparse Direct Solver on Distributed Heterogeneous Systems”, Xu Fu, Bingbin Zhang, Tengcheng Wang, Wenhao Li, Yuechen Lu, Enxin Yi, Jianqi Zhao, XiaohanGeng, Fangying Li, Jingwen Zhang, Zhou Jin, Weifeng Liu (SSSLab, China University of Petroleum-Beijing, China)

Best Student Paper Award

“HEAR: Homomorphically Encrypted Allreduce”, Marcin Charpek, Mikhail Khalilov (ETH Zürich), Torsten Hoefloer (ETH Zürick, Microsoft)

Best Reproducibility Advancement Award

“HEAR: Homomorphically Encrypted Allreduce”, Marcin Charpek, Mikhail Khalilov (ETH Zürich), Torsten Hoefloer (ETH Zürick, Microsoft)

Best Research Poster Award

“Evaluating Performance Portability of GPU Programming Models”, Joshua H. Davis, Pranav Sivaraman, Isaac Minn, Abhinav Bhatele (University of Maryland)

Best Visualization Award

“ExaWind at NREL: Upping the Ante”, Nicholas Brunhardt-Lupo, Ashsh Sharma (NREL)

Student Cluster Competition (Overall Winner)

RACKlette (ETH Zürich, Switzerland)

Sophia Herrmann, Alexander Sotoudeh, Hannes Eberhard, Samuel Karsko, Yi Zhu, Alexander Pietak (Team), Torsten Hoefler (Advisor)

Student Cluster Competition (Highest Linpack Benchmark)

Radiance of Weiming (Peking University, China)

Shengyu Liu, Yuanhang Sun, Tianle Xu, Zhuofeng Wang, Jiameng Huang, Ruicheng Wang (Team), Chun Fan (Advisor)

Onward to the Next SC

SC24 in Atlanta, GA

As always, Denver was a gracious and hospitable host for SC23, thanks partly to the excellent infrastructure provided by the Colorado Convention Center in Denver. Next year, SC24 will take place in Atlanta at the Georgia World Conference Center. The SC24 conference theme is “HPC Creates” and will focus on the enormous generative power that supercomputing affords the modern world.

Early SC24 Applications

Presently, Women in IT Networking at SC (WINS) is seeking qualified female candidates in their early- to mid-career to join the SCinet volunteer workforce at SC24. WINS is heavily focused on diverse representation and will provide full travel support and mentoring by experienced engineering experts before, during, and after the conference.

Additionally, Lead Student Volunteer applications have opened. This is a fantastic opportunity for former student volunteers interested in elevating their SC experience.

Dr. Arnold boasts an accomplished career, that includes being recognized as an ACM Distinguished Speaker and an IEEE Senior member, with the honor of receiving two R&D 100 Awards. Dr. Arnold’s commitment to diversity and inclusion is evident through his roles as the General Chair for the 2017 ACM Richard Tapia Celebration of Diversity and the co-founding of “The HPC Pipeline Workshop: Diversifying the HPC Workforce.”He is a researcher, that addresses the challenges of making extremely large computing systems accessible to experts from various domains. Dr. Arnold’s contributions, both academically and in advocacy, mark him as a leader in the field, pushing the boundaries of high-performance computing while ensuring that it remains inclusive and relatable to all. In his role as SC23 General Chair, he has been pivotal in making the Supercomputing Conference (SC) more relatable and inclusive. Drawing inspiration from the “I am Tiger Woods” campaign, which conveyed the message that anyone, regardless of their background, could aspire to greatness just as Tiger Woods did in golf, the “I am HPC” tagline aims to humanize the field of High Performance Computing and ensure it’s accessible to all.”

SC23 General Chair

Now, let’s delve deeper into the world of Dr. Dorian C. Arnold and learn more about his journey, perspectives, and contributions in the realm of High Performance Computing as he answers insightful questions about his career, experiences, and the future of the HPC community.

Dorian C. Arnold

Associate Professor of Computer Science, Emory University

Dorian on LinkedIN

Q: What single event most made you realize you wanted a career in HPC/computing?

Arnold: While studying for my master’s degree at the University of Tennessee, I took a course from Dr. James Plank; his engaging, fun, and effective teaching style sparked my curiosity in computer systems. Dr. Plank later became my M.S. research advisor. While at UTK during a department-wide event, Dr. Jack Dongarra who was already renown at the time, introduced himself to me, a lowly student, and asked me about my interests. I was struck by his humble, welcoming demeanor. My data points were computer systems are cool, HPC is cool, and the people are cool; what more could I ask for!

Q: What do you consider your biggest contribution to the HPC/computing community?

Arnold: I don’t know if it’s my biggest contribution, but perhaps the one that has had the biggest practical impact: during my PhD, I wrote the code for two software systems: MRNet, a highly-configurable software overlay network for large scale data communication and aggregation, and STAT a debugging tool that is built on top of MRNet. Both these codes have been used and still are in use in production mode on supercomputers across the world.

Q: In the past 35 years, what is the most significant overlooked breakthrough that has impacted the field in your eyes?

Arnold: Overlooked? Perhaps high-level programming languages and environments, coupled with compilers, that have helped to make HPC admissible and accessible to a broad range of scientists and engineers, who no longer have to understand all the low-level nuances of these complex systems to develop correct and efficient codes.

Q: What would you like to see change about, within, or among the HPC/computing community?

Arnold: More access, more diversity, more equitable opportunities, more inclusivity — for all definitions of those terms.

Attendee Resources

Personalize Your Schedule

Program Highlights

Expansive Exhibits

Students@SC

Engaging Activities

SCinet Technology

SCinet by the Numbers

Let the Festivities Begin

Inclusivity Matters

All of these programs and events make SC23 a dynamic and enriching experience, offering unique insights, networking opportunities, leading-edge technology, breakthrough research, and a chance to celebrate the groundbreaking work and people within the HPC community.

At the same time, if you were to look at who was on the floor 35 years ago versus today, you would see a much more diverse picture of genders, ethnicities, and nationalities. As Roscoe Giles, the first African American SC Conference Chair in 2002, and Dorian C. Arnold, this year’s General Chair, can attest, this transformation was no accident. It took concerted and, at times, controversial efforts by SC’s senior leadership and volunteers, including the establishment of an inclusivity committee around 2015.

We sat down with Giles and Arnold for a far-reaching conversation about the conference’s evolution. Arnold and Giles shared insights from their experiences as General Chairs and attendees of SC. They also shared their thoughts on fostering diversity at SC and within the HPC community in general and the significance of this year’s theme: I Am HPC.

Beginning at the End

Roscoe Giles

SC2002 General Chair

Both Arnold and Giles appreciate the provocative nature of this year’s theme, “I Am HPC,” which challenges attendees to contemplate their roles within the HPC sphere. Arnold explains: “One of my personal ambitions is creating a more inclusive and diverse world. As the General Chair who has the honor of choosing the conference theme, I wanted to understand what the conference meant to my executive committee to capture the essence of our collective experience. Though technical aspects were discussed, it was the human-centric theme that most captivated us. We landed on ‘I Am HPC’ because we felt it pays homage to all who contribute to the industry—from technical pioneers to the global community benefiting from HPC advancements. After all, the main reason for HPC is to learn more about the world so we can make it a better place for the sake of the people living in it.”

Giles believes the theme is essentially the embodiment of inclusivity. “It underscores the human dimension of HPC. It invites all attendees to see themselves as part of the overarching HPC narrative. Regardless of our backgrounds—be they technical, ethnic, or personal—we are united in our quest to advance HPC, and every story leading us here is valid. The theme prompts us to take ownership and responsibility for HPC’s direction in technology, applications, and the future. I envision someone new or from a non-traditional background initially feeling disconnected from ‘I Am HPC’ but ultimately embracing the idea, realizing that they too belong and contribute to the field,” he says.

Reflecting on his first SC conference in 1991, Giles underscores the importance of community building through interpersonal connections. “I remember a moment where I was on the up escalator, and another black attendee was passing by on the down escalator, and we both saw each other and locked eyes,” explains Giles. “It was striking, and I was curious about what was happening in his world. That moment stuck with me, and I know as we look to broaden diversity in the field, every connection and every shared experience adds a brick to a stronger and more inclusive foundation.”

dorian C. Arnold

SC23 General Chair

Steps in the Right Direction

Both Giles and Arnold are SC veterans, with Arnold participating consistently for over two decades and Giles attending most years since 1991. They concur that the most prominent change over time has been the conference’s exponential growth. Arnold attributes part of this growth to the diversification of its programs and exhibits, now including a robust academic component.

While both are happy to see increased diversity at SC, both acknowledge that inclusivity isn’t without its hurdles. Giles says that the nature of the conference itself makes the path to inclusivity nonlinear. “The SC conference’s structure, where each year is a fresh start, helps ensure its flexibility, adaptability, and ongoing growth. This same structure also makes it challenging to embed lasting programs. For example, some diversity programs from my tenure and around that time no longer exist today. But having said that, I do think the overall diversity within the conference has increased considerably. Early on, around the 2002 timeframe, we had some specialized programs in the education framework, along with collaborations with minority-serving institutions, that made a difference. Today, the commitment to holistic diversity and broadened participation is more systematic, spanning beyond a singular year’s agenda,” he notes.

Roscoe Giles with the SC2002 Committee.

Dorian Arnold with past and future General Chairs.

Arnold applauds the senior leadership for the progress, noting, “Transitioning from a mindset where diversity initiatives were considered ‘broader engagement’ to the establishment of an inclusivity committee around 2015 marked a pivotal shift. I believe it really paved the way for more comprehensive and effective inclusivity efforts. While there is still much work to be done, I think it really provides a fabric for making not just the conference but the entire HPC community, more diverse and inclusive. Over the coming years, we hope to see increasingly bigger and more impactful changes until we don’t need to have these conversations anymore.”

Navigating Enduring Barriers

Arnold identifies a lack of awareness on a couple of levels as one of the ongoing barriers to entry for marginalized groups. “Raising awareness about the challenges for these groups remains important,” notes Arnold, “but at the same time, we also need to increase awareness of the opportunities linked to diversity and inclusion. That’s easier said than done since we’re all only human and often default to understanding and evaluating experiences based on gender identity, race, nationality, socio-economic conditions, and so on. This might lead to unintentional or intentional myopia, where we may be unaware or dismissive of others’ experiences. So, raising awareness is crucial because progress will always be limited without that awareness. But awareness is only the start.  After gaining this awareness, we need to take action and ask, ‘How can I make a difference?’ And the more resources, seminars, books, and tools that we have to foster understanding and guide actions, the better.”

“Over time, we’ve seen more diverse communities emerge within HPC, including women and people of color. These communities support and reinforce each other, which is crucial for fostering diversity.”

Giles, meanwhile, underscores the importance of community building: “Rather than solely focusing on individual entry into the field, it’s vital to create and nurture groups of people, building communities that empower and reinforce each other,” says Giles. “Over time, we’ve seen more diverse communities emerge within HPC, including women and people of color. These communities support and reinforce each other, which is crucial for fostering diversity.”

To encourage participation from underserved communities and institutions at SC23, Arnold notes that he and his team tried to always put people at the core of the conference’s decisions and actions in support of the ‘I Am HPC’ theme. The initial efforts began by urging committee leaders to broaden their inclusivity scope and consider integrating new groups and perspectives. Arnold’s team has also actively sought to expand the conference’s technical reach, delving into new and impactful areas within HPC. Two of the team’s initiatives will be featured prominently at SC23. “This year, we’re launching the Illuminations Pavilion—a space for researchers who, due to financial limitations, couldn’t attend otherwise,” says Arnold. “The conference offers everything from exhibit space to travel assistance for accepted applicants. For the 35th anniversary celebration, we’re also focusing on the people behind pivotal technical advances in HPC. We’re particularly looking to highlight contributions from ‘hidden groups’ that have impacted the HPC realm over the years.”

A Wonderful Flash

Reflecting on his role as General Chair, Arnold admits a newfound respect for the intricacies of organizing such a large-scale event. “The aim is to make everything seem effortless to the attendees, but that belies the monumental effort of an army of volunteers that goes on behind the scenes for the better part of three years.”

Looking ahead to SC23 itself, both Giles and Arnold are most excited about interacting with the incredible community. “Ultimately, the seeing and sharing is what I’m really looking forward to. And I hope that some of the elements like the ‘I Am HPC’ theme and the celebration of people we’re trying to infuse into the conference are as visible as possible and impactful to the attendees,” says Arnold. “I really want everyone to walk away feeling included and charged up about what’s possible far beyond the conference itself.”

Roscoe in front of his SC2002 history panel.

Dorian greeting a prospective SC23 attendee.

Giles said that after helping to coordinate this year’s speakers, he’s excited to see the program actually happen. “Personally, coming from an applications background as a theoretical physicist, I always look forward to the research exhibits and opportunities to interact with researchers using HPC to advance science and engineering across many interesting disciplines. One of my great recollections from when I was chair is how fast the conference goes by compared to the amount of time you spend planning it. It’s sort of over in a flash, but it’s a “wonderful flash.”

About Dorian C. Arnold

Dorian Arnold is a tenured associate professor of computer science at Emory University. He specializes in research on distributed systems, fault tolerance, and HPC with an emphasis on performance and reliability in large-scale environments. With over 60 peer-reviewed publications amassing more than 1800 citations, Arnold’s contributions have earned him two R&D 100 awards. Beyond academia, Arnold has been a leading voice in the HPC community, undertaking steering committee roles for SC, General Chair for SC23, and Associate Editor of IEEE Transactions on Parallel and Distributed Systems. A passionate advocate for diversity, he has chaired events such as the 2017 ACM Richard Tapia Celebration of Diversity and the 2016 CRA HPC Pipeline Workshop. Arnold holds a doctorate from the University of Wisconsin, an M.S. from the University of Tennessee, and undergraduate degrees from Regis University and St. John’s College.

About Roscoe Giles

Roscoe Giles is a professor in the Department of Electrical and Computer Engineering at Boston University, where his research focuses on the application of high-performance and parallel computing to physics and materials problems. Over his distinguished career, he has held numerous leadership roles in the scientific community, including chairing the Department of Energy’s Advanced Scientific Computing Advisory Committee, Boston University Faculty Council, and SC02. A dedicated advocate for inclusivity in the tech realm, Giles was honored with the A. Nico Habermann award in 2000 for his efforts to increase minority participation in computer and computational science. He earned his Ph.D. in physics from Stanford University in 1975, marking a historic milestone as one of the first African American doctorates in the field at Stanford. He also holds a Bachelor of Arts in physics from the University of Chicago.

Decades of Expertise & Insight

Throughout his career, Grossman has authored or co-authored more than 200 publications and pioneered or contributed to an array of advancements in data management and analysis in computing, so he has many interesting stories about using HPC to advance data science.

As much of Grossman’s work focuses on health-care-related subject matter, and August is National Wellness Month, it seemed an ideal time to sit down and discuss some highlights from his career, including gleaning some insights about the fast-evolving use of HPC in healthcare. And, for those who want to dig a little deeper, Grossman provided links to several papers that capture noteworthy moments of impact in data science and beyond.

Robert Grossman, Phd

Frederick H. Rawson, Distinguished Service Professor of Medicine and Computer Science, Jim and Karen Frank Director of the Center for Translational Data Science, Chief of the Section of the Biomedical Data Science (Department of Medicine), University of Chicago; Chair, Open Commons Consortium

Robert on LinkedIN

Q: How did you get started using HPC systems in your work?

Grossman: Beginning in the mid to late 1980s, I worked on the problem of how you can manage and analyze large amounts of scientific data. Back then, this was not a popular topic, and HPC was focused on using large machines for simulations and other specialized tasks.  

I was interested in what types of questions you could answer and problems that you could solve by creating a database of scientific objects and then querying the database to make discoveries or to answer basic questions. This was a variant of the basic trade-off in algorithms of trading space for time. For example, if you wanted to quickly answer a question about a flow of a differential equation or control system, perhaps you could simply answer the question with a sequence of simple queries to a database that stored trajectory segments of the system. [1] [2]

A few years later, software infrastructure planning for the Superconducting Super Collider (SSC) was beginning. One of the challenges for the SSC was how to manage and analyze all the data for it. The SSC was designed to produce petabytes of data, which was million times larger than the data that was routinely analyzed at that time. Working with Drew Baden from the University of Maryland, we proposed what was then a radical approach—create a distributed database of all the events produced by the SSC. [3]

This led to several computer science challenges: 1) how do you create very large databases of distributed scientific data; 2) how do you transfer large scientific datasets over wide area networks; and 3) what are the algorithms to process, explore, analyze, and share the data in these databases?

These three questions are now familiar SC topics, and I ended up working on various aspects of these topics for much of the past 30 years.

Q: What are some of the key projects you have worked on through the years?

Grossman: I have worked on a number of projects over the past 30 years, so I’ll just give you some highlights about a few of them that seem to have notable impacts, including the Petabyte Access and Storage Solution (PASS) project, National Scalable Cluster Project (NSCP), and a startup I founded called Magnify. 

The PASS project, which the DOE [Department of Energy] funded, was born out of an effort to develop what was effectively big data technology for SSC with the hope that the technology could be used more generally. Collaborating with several institutions, we tackled the “foreseen data access problems of the next generation of scientific experiments…characterized by a large sample of complex event data (~10¹⁵ bytes), a dilute signal, and a large (~1000) and geographically distributed user community.” [4] Keep in mind that in the early 1990s, 1 TB of disk cost roughly $750,000. So, at the time, 1 PB of disk would have been approximately $750 million. 

The PASS project led us to prototype three different technology approaches: scaling relational databases, scaling object-oriented databases, and developing object stores for managing large collections of data objects. My team eventually focused on the latter, developing an open-source lightweight manager for large data object collections called PTool. Although its design sparked controversy, it was efficient, scalable, and enabled access to large amounts of data (it scaled 10x-100x larger than the first two prototype options). It was also open source. The SSC was canceled, but our core software design became generally accepted and was mostly picked up by the Large Hadron Collider (LHC) that was built and is operated by CERN.

“Our core software design became generally accepted and was mostly picked up by the Large Hadron Collider (LHC) that was built and is operated by CERN.”

After the SSC’s cancellation, we applied for an NSF [National Science Foundation] grant to develop a PASS-like software architecture for broad scientific use. The result was the NSCP, an early example of what later became known as “the grid.” [5] The NSCP-1 Meta-Cluster was completed in 1996 and interoperated three geographically distributed clusters. The first NSCP Meta-Cluster contained approximately 100 nodes and 3 TB of disk geographically distributed among the participating sites and was connected by laboratory, campus, and national ATM [asynchronous transfer mode] networks. 

At the time, NSCP used PTool for data management to create what would now be called a data warehouse (at the time, we called it a lightweight object manager). Around the same time, I developed an open-source software tool called PSockets that used parallel TCP [Transmission Control Protocol] connections to increase bandwidth and move PTool-managed data over wide area networks. [6] PSockets and its successors ended up winning several bandwidth challenges at SC conferences. 

In 1996, I founded a startup called Magnify to take the lessons learned from the PASS Project and apply them to financial services and online advertising. We used commodity clusters of workstations along with PTool and similar techniques to manage out of memory data and build and “glue together” ensembles of machine learning models in a simple, elegant way. We also developed specialized software applications for deploying these models into operational environments. We came across the idea of using ensembles on our own and met quite a bit of resistance from statisticians and others who objected to the approach. Although others had also begun to use ensembles in machine learning and other areas, they were not well known at that time. Looking back, the idea of an ensemble is a simple and obvious one, which dates back in some form to Condorcet’s jury theorem from the 18th century.

Q: Let’s shift gears and talk about your current work health- and wellness-related work with the Genomic Data Commons (GDC). What is the GDC, and why has it been so effective?

Grossman: We were awarded a contract from the National Cancer Institute in 2014 to develop the Genomic Data Commons, which was launched in 2016 by then Vice President Biden as part of the Cancer Moonshot. Today, the GDC is used by over 60,000 researchers each month and, on average, over 2 PB of data are accessed or downloaded each month.[7]

Robert Grossman speaking to then Vice President Joe Biden at the launch of the Genomic Data Commons on June 6, 2016.

Importantly, we curate and harmonize all the data submitted to the GDC by running a common set of bioinformatics pipelines. [8] The importance of curating and harmonizing the data to build a successful data platform cannot be overemphasized. [9]  

Building effective data platforms has always been about choosing the right set of questions to optimize, so users have efficient access to the data they need and a simple intuitive experience using the data platform. Sometimes, we are more successful with this approach than other times, but it is always one of the criteria in mind when developing new data systems. Jim Gray eloquently summarized this approach with his advice: Give me your 20 most important questions you would like to ask of your data system, and I will design the system for you. [10]

Given the value of the GDC to researchers, we have continued iterating the technology for wider use with the goal of making it better and easier to replicate. Our Gen3 version is open source, and today there are over 20 Gen3 data commons built by ourselves and others supporting research in cardiovascular disease, COVID, infectious diseases, irritable bowel disorder, opioid use disorder and pain management, and other diseases—in addition to cancer.

Our goal is to reach 200 Gen3 data commons by working with partners and by developing a Gen3 as a Service.

Q: Where do you see the future of HPC in the next decade, particularly in relation to health care and wellness?

Grossman: The role of HPC will continue to be important in simulations that are the basis for understanding molecular interactions; identifying candidate drug targets; building agent-based simulations; creating digital twins of cells, tissues, organs and living systems; and a variety of other biomedical applications.

On the other hand, for many important problems in biology, medicine, and health care, we are not compute-limited, but rather data-limited. The HPC challenge is to build the data platforms that can manage, explore, analyze, and share biomedical data at the scale needed and with the governance, security, and compliance required so we can tease out interesting small effects.

“The HPC challenge is to build the data platforms that can manage, explore, analyze, and share biomedical data at the scale needed and with the governance, security, and compliance required so we can tease out interesting small effects.”

Q: What are some important lessons you’ve learned throughout your career in HPC that you believe would benefit those just starting out in the field?

Grossman: Just as there is no substitute for actually programming, there is no substitute for actually exploring data, understanding data, building multiple models over the data, and understanding the advantages and disadvantages of the different models. Today, it is so easy to use software to build a model that you can fall into the trap of building a model that has serious problems because you don’t understand the data. In other words, the wonderful power of software today to build statistical, machine learning, and other models with such little effort can fool us into believing we understand the data when, in practice, this often requires a lot of work and effort.

It’s also easy to forget the wonderful flexibility researchers have at universities. They can tackle important, ambitious problems and fail. All that is required is that you succeed from time to time and then talk and write about your successes. There are very few other careers where you have this level of flexibility.

Learn More:

[1] R. Grossman, “Querying databases of trajectories of differential equations: Data structures for trajectories,” NAS 1.26:185040, Jun. 1989. Accessed: Jun. 21, 2023. [Online]. Available: https://ntrs.nasa.gov/citations/19890016401.

[2] R. Grossman, “Querying databases of trajectories of differential equations. I. Data structures for trajectories,” in Twenty-Third Annual Hawaii International Conference on System Sciences, IEEE Computer Society, 1990, pp. 18–24. doi: 10.1109/HICSS.1990.205171.

[3] D. Baden and R. Grossman, “A model for computing at the SSC (Superconducting Super Collider),” Superconducting Super Collider Lab., Dallas, TX (United States), SSCL-288, Jun. 1990. doi: 10.2172/6515278.

[4] D. R. Quarrie, C. T. Day, and S. Loken, The PASS project: A progress report. 1994. doi: 10.2172/10172158.

[5] I. Foster and C. Kesselman, The Grid 2: Blueprint for a new computing infrastructure. Elsevier, 2003. https://doi.org/10.1016/B978-1-55860-933-4.X5000-7. 

[6] H. Sivakumar, S. Bailey, and R. L. Grossman, “PSockets: The case for application-level network striping for data intensive applications using high speed wide area networks,” in SC’00: Proceedings of the 2000 ACM/IEEE Conference on Supercomputing, IEEE, 2000, pp. 38–38. doi: 10.1109/SC.2000.10040.

[7] A. P. Heath et al., “The NCI Genomic Data Commons,” Nat. Genet., pp. 1–6, Feb. 2021, doi: 10.1038/s41588-021-00791-5.

[8] Z. Zhang et al., “Uniform genomic data analysis in the NCI Genomic Data Commons,” Nat. Commun., vol. 12, no. 1, Art. no. 1, Feb. 2021, doi: 10.1038/s41467-021-21254-9.

[9] R. L. Grossman, “Ten lessons for data sharing with a data commons,” Sci. Data, vol. 10, no. 1, Art. no. 1, Mar. 2023, doi: 10.1038/s41597-023-02029-x.

[10] A. S. Szalay, “Jim Gray, astronomer,” Commun. ACM, vol. 51, no. 11, pp. 58–65, 2008. doi: 10.1145/1400214.1400231.

Discover the Future

6–10 August 2023

Join SIGGRAPH in person 6–10 August in Los Angeles, or view live-streamed sessions virtually, and connect with your community of professionals from around the world. If you’re passionate about discovering the bold future of art, animation, research, gaming, production, and more, SIGGRAPH 2023 is the place to be.

Papers Sneak Peek

View the SIGGRAPH 2023 Technical Papers trailer and understand the reason why a past conference attendee said they “were just mindblowingly amazing.”

Why Attend?

Hear from past SIGGRAPH conference participants themselves about why they attend year after year.

SIGGRAPH Has It All

Explore the full program

For our art and design creators, the Art Papers and Art Gallery programs are sure to inspire artists’ own new ideas. Our gaming and interactive techniques aficionados can check out the Real-Time Live!, VR Theater, and Immersive Pavilion programs to expand their horizons on what’s next in games and new realities.

For production and animation connoisseurs, the Electronic Theater, VR Theater, and Production Sessions have what they are looking for at the forefront of storytelling. Dive into research and new technology in the Courses, Birds of a Feather, Educator’s Forum, Frontiers, Panels, Posters, Technical Papers, and Talks programs. Explore the full program to learn more:

“This year, we celebrate this magnificent milestone in computer graphics and interactive techniques as we embark on the next 50 years of innovation. We invite you to contribute your audacious ideas in computer graphics, digital art, animation, visual effects, new realities, artificial intelligence, research, and more. Let’s celebrate the Age of SIGGRAPH as a living era, one that will reach generations in 2023 and beyond. I look forward to imagining and inventing our bold and bright future, together!”

— ERIK BRUNVAND, SIGGRAPH 2023 CONFERENCE CHAIR

Expand Your Horizons

Early Formative Experiences

Williams’ introduction to business started in her early years when she passed time after school and late into the evening at her family’s music store. Her family later started a small computer business, and Williams headed off to college to study business in 1990. By 1992, the family business was growing, and Williams’ mom needed help, so Amy returned home to offer a hand. “That was a pivotal moment for me, making the decision to leave school and trusting the experience itself could get me where I needed to go,” says Williams. “I did finish my bachelor’s degree, but that was my introduction to the high-tech world.”

Amy williams

Owner and CEO, PIER Group

pier group

Back at home, Williams’ business education began in earnest where she quickly learned most aspects of the family computer business. “I started in purchasing and helped with shipping and receiving, then got involved in finance, and even spent time in the sales side of the business building relationships with customers and manufacturers,” Williams explains. Over time, she built a division of the family company that focused strictly on higher education and larger research institutions, which led her to HPC. Eventually, Williams, her husband and a small team decided to buy out the research and education division from her family and operate entirely on their own as the Partners In Education & Research (PIER) Group.

Following Heart & Soul

Williams says she was particularly interested in the education and research space for a few reasons. “One reason research is so near and dear to me is that our daughter was diagnosed with leukemia when she was eight. At that time, we went to St. Jude Children’s Hospital and learned about how hospitals were sharing data and how important it was to finding cures,” notes Williams. “My small team also recognized the unique nature of the education and research space in terms of the need for close working relationships with stakeholders, ranging from students to staff to faculty in any institution. Our team loved building those relationships and the computing environments that are helping solve interesting and important scientific challenges.” Today, PIER Group supports research and education facilities across the country. As an extension of its commitment to research organizations, PIER Group has contributed volunteers and hardware that facilitated the SCinet wireless network for three years running.

“Our team loved building those relationships and the computing environments that are helping solve interesting and important scientific challenges.”

Leading by Example

Notably, Williams’ early experiences in the family business had a profound impact on who she is today and how she leads. “While I didn’t have a career path in mind when I was younger, I learned about the value of working hard from a very young age and treating everyone as equals and as part of the team. I think if you interviewed our team, they would say I lead by example. And while I have the title of CEO today and own the business, I really do try to make sure everyone has a voice, and we all recognize one another as peers. It doesn’t matter if someone is in the back office or an engineer, helping each other and keeping our clients happy is what matters most and it’s important to me that everyone feels like they’re not just part of a team, but really part of a family. We’ve intentionally tried to stay lean and nimble so we don’t lose that.”

Matthew Magill, principal consultant, and Becky Schneider, vice president of programs and office management at PIER Group, both offered testament about Williams’ empowering and energizing leadership style. “PIER Group is a relatively small company, and Amy’s outward focus on employees and customers really does make it feel like a family. It’s a great change of pace compared to bigger corporate environments I’ve worked in and makes the challenges and problem solving in bigger projects more fun and rewarding,” explains Magill. Schneider echoed Magill’s sentiments, stating that William’s generosity and warmth are inspiring: “Amy really does want to see employees succeed, so she pushes us and challenges us and also never misses the opportunity to celebrate our wins and progress.”

Amy enjoys the outdoors and staying active in her free time.

Amy working with the PIER Group team picking up trash during one of their community outreach programs.

Working Beyond Barriers

Williams says she got to where she is today through a combination of hard work, serendipity, and curiosity. “It just kind of happened organically,” she says. But she also recognizes that the technology field still can be intimidating for women. Williams explains: “Last week, we were in Las Vegas for a big conference, and after a meeting in one of the conference rooms, we took a picture. It wasn’t until I left the room and looked at the picture that I thought about the fact that I had been the only woman in a room full of 16 men. I don’t know why that doesn’t give me pause or make me nervous, but I can see why it might make other women uncomfortable. There are areas in tech like engineering where you typically just don’t see as many women as men in many organizations.” 

To improve the balance in engineering, Williams thinks it’s important to continue encouraging young girls to pursue STEM programs. “I didn’t think about that as much in the past, but with a grandchild on the way, it’s something that’s got my attention,” says Williams. 

PIER Group itself is a certified Women’s Business Enterprise (WBE) and home to a diverse team. When asked if she has any advice for women looking to get into the HPC industry, Williams encourages women to focus on their interests and not perceived limitations. “My advice is to go into any endeavor or opportunity focused on working hard and doing the right thing. I didn’t intend to be a CEO, but I always cared about my relationships. I think the combination of my hard work and focus on relationships helped me get where I am today. So, don’t overthink it when starting out. Just find an organization you like, work hard, be yourself and know that success isn’t going to come without hard work.”

“My advice is to go into any endeavor or opportunity focused on working hard and doing the right thing.”

Roy’s Path to Strategic Lead

If there’s one thing Dr. Roy Campbell doesn’t ask when he goes to work every morning, it’s “why?” After a career spanning nearly 30 years with the DoD, he has no doubt that the work he’s doing is meaningful, so he’s excited to go to work and take on big challenges related to modernizing the DoD’s HPC program every day, even if it often leads to compressed schedules at home.

Dr. Roy Campbell

Chief Strategist, Department of Defense High Performance Computing Modernization Program (DoD HPCMP)

Roy at AIAA

Before his career arc tilted in a strategic direction, Campbell spent his early years in HPC working on networking and data storage solutions for the DoD Supercomputing Resource Center located in Vicksburg, Miss. “In 2002, I moved to the HPCMP’s center in Aberdeen, Md., where I learned to benchmark supercomputers,” explained Campbell. “And then in 2008, I moved to the HPCMP management office in D.C., where I served as the lead [Defense Research and Engineering (DREN) Program Manager] for the DoD’s research, development, test, and evaluation network. I also served as the HPCMP’s Deputy Director, Chief Technology Officer, and Chief Scientist.”

Today, as the HPCMP Chief Strategist, his main responsibility is formulating a strategy or vision for U.S. defense supercomputing. To do that, Campbell examines a wide range of organizational, financial, technological, geopolitical, scientific, and modernization trends. The end goal is to provide senior officials with an actionable blueprint for solving very difficult problems with supercomputers. “Our program was initially focused on defense grand challenges that contribute to scientific knowledge. Today, however, our goal is much broader. We also work to reduce the cost, schedule, and risk for a wide variety of military platforms,” he said. 

Reducing Risks & Costs

One of HPCMP’s missions is to provide a virtual space to support the design and testing of jets, helicopters, ships, submarines, tanks, and antennas without building expensive prototypes or putting the public at risk. “Removing bad designs and refining good ones early through virtual testing saves an extraordinary amount of time and money,” said Campbell. For example, he noted that troubleshooting for the Sikorsky CH-53K King Stallion heavy-lift cargo helicopter helped isolate why the copter engines were highly inefficient whenever they flew near the ground. Simulations created by software developers at the Naval Research Lab revealed that when the helicopters were at low altitudes, engine exhaust was looping back into the engine and straining them. “Minimally, that probably saved more than $100 million in testing and other sundries,” said Campbell.

HPC simulations have also been integral to improving and streamlining training for the fighter jet refueling process with the Boeing KC-46 Pegasus, leading to both significant cost savings and improved safety. Campbell also noted that Congress is now even pushing the DoD to explore how it could use supercomputing modeling and simulations to reduce or eliminate the need for live fire testing near costly assets, such as Ford Class Aircraft Carriers.

CH-53K Helicopter Simulation

Jet Refueling with Boeing KCC-46 Pegasus

For Campbell, some of the most exciting work his team is pursuing is around melding hard and soft sciences into single calculations: “Our biggest challenge right now is supporting advanced scenario analyses that incorporate both hard and soft sciences in a single calculation to produce actionable advice for campaign planners. Hard sciences are generally based on physics. Soft sciences include the analysis of social media feeds, economics, population growth, and sentiment.”

Navigating Technology Shifts

When asked about the key hardware and software technologies being considered for the next-generation HPC systems within the DoD, Campbell noted that trends related to quantum and neuromorphic computing aren’t as relevant to defense supercomputing at the moment and likely won’t be for some time. “Today, as we face the end of Moore’s Law, the big shifts we are seeing include a transition away from monolithic general-purpose chips toward general-purpose chiplets and special-purpose monolithic chips,” explained Campbell. “The former affects defense supercomputing the most and may present future opportunities for designing chiplets specifically for defense use cases.”

Collaboration & Community Building

Campbell, who has three degrees in electrical engineering with an emphasis on communications systems and information theory, was hooked on supercomputing once he understood how the low-level components of a supercomputer impacted science and engineering disciplines. He notes that without a mentor, it would have been nearly impossible to gain traction in his early career. “By far, Dr. Larry Davis had the greatest influence over my career. I met him in 2003 in Washington D.C. He gave me an opportunity as a junior member of his supercomputer benchmarking team,” said Campbell. Given his experience, he laments that the push for greater efficiency has driven many organizations to remove mentoring opportunities: “Our community suffers from the aftermath of this short-sighted metric.”

Although he is the HPCMP Chief Strategist, Campbell said that collaboration remains key to his growth, and he is standing on the shoulders of giants. “Over the years, Ms. Thuc Hoang [the Director of DOE NNSA’s ASC Program] has been a great collaborator, and I learn a lot from her on a regular basis. The scale, breadth, and complexity of her program is impressive. And, the foresight of Hon. John J. Young, Jr., the 2007-2009 Undersecretary for Acquisition, Logistics, and Technology is critical to my role. He championed a significant investment in supercomputer-based design software for jets, helicopters, ships, submarines, tanks, and antennas that allows the DoD HPCMP to provide significant value to the DoD and the U.S.” 

An Industry Pioneer

Given his history in HPC publishing, Tom has a unique perspective on industry developments. Tom was kind enough to sit down with us to share highlights from his more than 30-year journey in HPC. In this Q&A, we covered a broad range of topics, including how he got into HPC publishing, highlights from his career, thoughts on industry progress with diversity and inclusion, and what he’s most excited about today in HPC.

Thomas B. Tabor

CEO, Tabor Communications

Q: This article will be published during Asian American and Pacific Islander Heritage month, and you were born in Guam. I’ll bet it wasn’t a direct route to go from being an islander growing up in Guam to being a publisher covering the leading edge of computing. How did you get into HPC-related news and publishing?

Tabor: I’m bi-racial, Chamorro (Guam) and Caucasian. I grew up in the Chamorro culture, and I celebrate it every day. I was born and raised on Guam, with my mother being an island girl and my father a white World War II Navy guy. Growing up, Guam was a small island focused on family and community, which resonates with me personally, professionally, and culturally. The island’s population was around 50,000 back then, but today it has grown to around 170,000.

Guam’s matriarchal culture had a big influence on me, and I seek out powerful, assertive women who are doers, like my wife, sisters, and mother. This has shaped how I approach life and business, and I’m a strong advocate for gender, ethnicity, and lifestyle diversity in the workplace.

My introduction to the HPC space was random luck. I started my career in publishing and later spent some time working in manufacturing before returning to publishing. When I decided to re-enter the publishing field, I answered an ad for Supercomputing Review magazine, which is no longer around. At the time, I couldn’t even turn on a PC and had no familiarity with technology. Still, they gave me a job, and I quickly became fascinated by the topics we covered and the way technology was transforming and impacting society. Today, it’s not lost on me that I went from an island that is about 30 miles long and 4-8 miles wide to running a publication about the fastest computers in the world. It’s been quite a journey.

Q: You were an early mover in HPC-related publishing. Take us back to the late 80s. How did you come to be involved with HPC publications, and what were you up against?

Tabor: When we were publishing Supercomputing Review, the federal government was the primary customer for HPC. The first Gulf War led to a sudden freeze in HPC spending, which affected our advertising revenue. We lost 75% of our advertising almost overnight, significantly impacting our operating budget. Despite trying to sustain it, the magazine was eventually sold to a Texas publishing group.

Meanwhile, we were developing an Internet-based publication called SuperNET, which I acquired and renamed HPCwire. We called it “wire” to create an association with electronic news services since most people were new to the Internet but generally familiar with wire services versus an electronic news service.

In the very early days, we had only a few employees and ~$500 in the bank, so I burned the midnight oil to build a media kit, and we started creating content. Then one day, I received a call from Joe Grisillo, a VP at Fujitsu, who expressed interest in our work with SuperNET and was interested in supporting us.

After setting up a meeting with him, I put our savings into dressing up our office so we looked as established as possible and gave him a demo using a Mac Classic PC. When I was done, Joe looked at me and said something I didn’t expect: “This looks really cool. How can I help you?” I was expecting him to ask for a proposal, so on the fly, I suggested he could sponsor us. Surprisingly, he immediately agreed to sponsor us for $10,000, which helped us make payroll and attend an SC conference in Minnesota. There, we secured half a million dollars in advertising, and that’s how our journey began. (Editor’s note: SC92 was held at the Minneapolis Convention Center in Minneapolis, and was keynoted by Larry Smarr, then Director of the National Center for Supercomputing Applications (NCSA). He spoke on the topic, “Grand Challenges! Voyages of Discovery in the 1990s.” Tom and Larry both live in La Jolla, California, and are neighbors and still great friends.)

Tom Tabor at Supercomputing Review Magazine in the Early ’90s

Q: What are you most proud of with respect to Tabor Communications and your history in HPC publishing?

Tabor: One thing I’m most proud of at Tabor Communications is our staff diversity and low employee turnover rate. Many staff members have been with us for over a decade – some over two decades, with the shortest tenure being two to three years. Building a healthy culture contributes to this longevity, and despite ups and downs over the past 30 years, our team’s dedication is what I’m most proud of.

We’ve also had a lot of cool publishing firsts. For example, early on, pre web browser, we built a Bulletin Board System (BBS) that functioned like a webpage. Our editors uploaded content daily, and it was hard to keep up. To address this, I asked our manager to create a weekly synopsis of the content via email. This eventually evolved into an email newsletter with each issue resembling a table of contents, including an article title and abstract, and advertisements.

Next to each article, we added a small box created with brackets. Readers could reply to the email with an ‘X’ in the box to request the full article. Using a simple Unix grep command, we would grab and send the requested article back to the reader. This innovation led to the creation of the world’s first paid subscription, paid advertising publication on the Internet.

Q: Throughout your career, you’ve served as the publisher for a host of publications and events, so you’ve had a front-row seat to HPC industry evolution. What industry developments were most surprising when they happened, and what trends or developments are you keeping a close eye on today? 

Tabor: Some of the most surprising and interesting industry developments include the incredible trajectory of HPC technology. When we first started, we dealt with millions of floating point operations per second (MegaFlops), and today we’ve reached a billion billion operations per second (ExaFlops.) Each breakthrough in processing power has increased the ability of HPC to impact society and improve quality of life.

The most disruptive developments are probably the Internet and the web browser, which came out of the HPC field and have generated hundreds of trillions of dollars for the economy. The web browser was co-developed by Marc Andreessen, now General Partner of Andreessen Horowitz, at the NCSA under the leadership and guidance of Larry Smarr.

Today, AI is clearly the next major disruptive technology. About five years ago, I remember talking to Horst Simon, then-deputy director of Lawrence Berkeley National Lab, and asking him when we would start seeing real progress in AI. At the time, he said we were a ways away and that it would likely take exaflop capabilities. Well, we just broke the exaflop barrier last year, and then quickly saw a release of a host of groundbreaking AI tools, including ChatGPT, bringing us all the more closer to Artificial General Intelligence (AGI).

I think these developments will be disruptive in positive ways. HPC, and tech in general, has been built to offload mundane tasks and help humanity focus on more profound endeavors. As technology advances, human cognitive capabilities will advance and adapt. While the rapid pace of change can be intimidating, many people will use these new tools without fully realizing their impact, much like how we use smartphones today.

Tom at the Frontier Supercomputer at Oak Ridge National Laboratory

Q: Since diversity inclusion is so important to you, where do you think the industry has made the most strides, and where is significant work yet to be done?

Tabor: The HPC community is cool and unique in that it’s global but also pretty small, so the efforts to improve diversity are easily identified and their results recognizable. Anecdotally, you can see the progress on the floor at SC, where there are more women and people of color than there ever were very early on. And, it’s great to see outreach efforts to historically black colleges and other diverse institutions.

I think some of the greatest strides have been made through Women in HPC, which is a large and very active community. The Women in HPC mentorship program engages female senior executives from industry to provide guidance to women in their HPC career paths starting in grad school. Access to mentors who can show women the ropes and help them navigate their careers has been an important leap forward for diversity and inclusion in the industry.

Overall, I’m really encouraged by what we’re doing. And, certainly, there’s always more room to grow.

Q: When you look at trends in HPC, what are you keeping a close eye on, and what gets you most excited and why?

What an exciting time to be in HPC! Cloud has democratized access to HPC capability, and we’re on the cusp of Quantum Computing (QC) and Artificial General Intelligence…it’s not a matter of “if,” but “when.” I’m not done because we have so much to explore. In fact, at our Wall Street event last year, we assembled a panel of three leading QC analysts and a futurist to discuss the realities and future of quantum computing. Our goal was to provide the audience with valuable insights and a narrative to share with their leadership, helping them understand what they should be monitoring and striving for in the quantum realm.

It is very cool and a big eye-opener thinking about HPC-related disruptions that are yet to come. Quantum and AI are going to be very disruptive to the point where you can imagine a world where mundane tasks for humans are handed off to systems, and we can focus on benevolence, our quality of life, and equality. I’m an optimist, so I’m really excited about the possibilities.

The E3SM project, which began around 10 years ago, is a high-resolution climate modeling initiative aimed at developing state-of-the-art climate system models for use on exascale computers, as envisioned by the Department of Energy (DOE). The main objectives of the E3SM project are to advance climate modeling capabilities to address diverse applications in support of DOE mission requirements. The project includes over 100 scientists and software engineers at multiple DOE laboratories as well as several universities.

In February 2023, the E3SM project had an early opportunity to run its Simple Cloud Resolving E3SM Atmosphere Model version 1 (SCREAMv1) on Frontier, the first U.S. exascale computer, making it the first cloud-resolving model to run on an exascale computer on GPUs. The team achieved unprecedented results, with the atmospheric component operating at a rate exceeding 1 simulated-year-per-day (SYPD). It accomplished this remarkable feat using 8192 Frontier nodes, each equipped with 4 AMD MI250 GPUs.

Note that this Q&A interview with David and Mark has been edited for clarity and length.

Sharing E3SM Insights

David C. Bader

David leads the climate research program for Lawrence Livermore National Laboratory, a Department of Energy research institution. He has been an active researcher and research manager in meteorology and climate science, focusing on the intersection of science with high-performance computational/information technologies for 35 years. Bader currently is the lead Principal Investigator for the Energy Exascale Earth System Model, a collaboration of 8 DOE National Laboratories and multiple universities.

David on Linkedin

Mark A. Taylor

Mark is a mathematician who specializes in numerical methods for parallel computing and geophysical flows. He currently serves as Chief Computational Scientist for the DOE’s Energy Exascale Earth System Model (E3SM) project. He developed the Hamiltonian structure preserving formulation of the spectral element method used in E3SM’s atmospheric component model.

Mark at Sandia

Q: How does E3SM simulate Earth’s water cycle, biogeochemistry, and cryosphere?

David: The original model has four main components: an atmospheric general circulation model (akin to weather prediction models), an ocean circulation model, a sea ice model for icy regions, and a land model, which includes sub-models for elements like vegetation. Sometimes, rivers are considered a separate component, connecting land and ocean. For climate change analysis, it’s essential to have a land ice component to account for melting glaciers in places like Greenland and Antarctica that contribute to rising sea levels.

What sets E3SM apart is its coupling with a multi-sector dynamics model, previously known as an integrated assessment model. This model connects climate change to economic and energy systems, integrating a human component into the analysis. It enables us to evaluate climate change’s impact on energy consumption, energy production source choices, and more. We’re currently in the early stages of implementing this integrated approach.

Mark: I think it’s also worth mentioning the flux coupler. This component connects all the models by facilitating communication between them. For instance, when precipitation occurs in the atmosphere, the coupler transfers this data to the ocean model, ensuring that each component accurately reflects the current state of the climate system.

I would also just add that our model, like most climate models, simulates Earth’s atmosphere using differential equations. When we know the differential equations governing aspects such as atmospheric motion, the model solves those equations. For other aspects where the equations are unknown, we approximate them using parameterizations.

Q: What simulations does E3SM conduct to address key Earth system science questions?

David: Like most modeling groups, establishing credibility is crucial. Every few years, the Coupled Model Intercomparison Project (CMIP) conducts a series of climate change simulations that are featured in IPCC [Intragovernmental Panel on Climate Change] reports. These simulations involve the input of various scenarios, including future emissions of carbon dioxide, aerosols, and other atmospheric constituents, to project the future climate. Our model, along with other major modeling groups, participates in these simulations.

One feature of our model that’s not unique but also not common is the use of regionally refined areas, particularly over North America. This allows us to obtain a more detailed view of climate change in specific regions of interest within the US or elsewhere in the world. Most models run with a horizontal grid spacing of around 100 kilometers in both the atmosphere and ocean. However, our model can refine this grid spacing down to 10 kilometers in both the atmosphere and ocean, without having to apply high refinement everywhere. This offers a more focused approach to studying regional climate change impacts.

With the latest exascale model, we’ve even achieved a resolution of three kilometers, which is capable of resolving individual storms. The simulations generated by this model resemble actual weather events, providing a more detailed and accurate representation of the Earth’s climate system. Previously, high-resolution models typically had a horizontal grid spacing of around 25 kilometers; three kilometers is a resolution better than most of today’s weather prediction models.

Mark: A three-kilometer resolution is a goal for many climate modeling centers, as it enables storm and cloud-resolving scales. However, achieving this level of detail requires significant computational resources.

Q: How does E3SM incorporate human influences on Earth’s systems?

David: The most significant and obvious human impact on climate change is the emission of greenhouse gasses, primarily carbon dioxide and methane, resulting from the consumption of fossil fuels for our global energy needs. Human activities also produce atmospheric aerosols, which can have a temporary cooling effect but ultimately mask the actual warming that is occurring. So as we clean up air pollution, we may inadvertently exacerbate climate change.

Another way humans affect the climate is by altering land use. Converting forests and rainforests into farmland, rerouting rivers, and constructing dams can change the way land interacts with the atmosphere, influencing long-term climate patterns. Understanding past, present, and future land use changes can help us predict their impact on regional and global climate.

Mark: The CMIP, which the IPCC relies on for many of its reports, includes scenarios for things like future CO2 concentrations. For example, the highest expected levels, or scenarios where the world makes a heroic effort to reduce CO2. So there are often different CO2 levels prescribed in, say, a forest model for instance. For some of the other things Dave mentioned, we try to model and predict rather than prescribe. 

Q: How are exascale computing architectures impacting atmospheric modeling?

Mark: The DOE recently delivered the world’s first exascale computer, Frontier, and we finally got our code for a Simple Cloud Resolving Atmosphere Model running on Frontier’s advanced architecture. The exascale computing capacity enables us to run very high-resolution cloud-resolving models, allowing us to simulate the motions in the atmosphere responsible for cloud formation, rather than approximating them. 

The DOE has been working on building the US exascale computers for some time. The hardware they chose for both Frontier and the upcoming Aurora systems is GPU-based, although climate models have run on CPUs for decades. So adapting our codes—and it’s a huge amount of code—to run on GPUs represents a significant change and it took roughly five years to accomplish. 

In general, atmospheric modeling is divided into two components: the dynamical core [how air moves and the atmosphere behaves] and the physical parameterizations. We began by focusing on the dynamical core, spending about two years exploring various approaches, programming models, and parallelization strategies. GPUs have numerous computing cores, necessitating the exposure of a lot of parallelism. Once we figured that out, we switched to a more sophisticated dynamical core suitable for high resolution. The process of porting that dynamical core went much faster, as our programming model and parallelization approach had been established.

At that point, the team was well established and had a good understanding of how to work with GPUs, so we turned our focus to the second component, the physical parameterizations, often referred to as physics. The team spent about a year rewriting this code to run on GPUs.

The final stage involved developing the driver, which integrates all the components, handles I/O, and communicates with the land, ice, and ocean components through the coupler.

Something that’s unique to our project is that the code is written in C++… Climate models are usually developed in Fortran, and while there are approaches to get Fortran to use GPUs, we didn’t think they were mature enough so we ended up rewriting the code in C++. And then we use a performance portability layer called Kokkos that allows us to support both GPUs and CPUs. 

We’re quite satisfied with the performance and deployability of our models, as we have maintained CPU performance on various CPU systems, including those with ARM chips and more traditional Intel and AMD-based CPUs. At the same time, our code is running on GPUs from two different vendors, AMD and Nvidia, each with their distinct programming models. We also hope to be running on Intel GPUs soon.

We hope that our approach is robust enough to accommodate new architectures down the road, but since it is difficult to predict upcoming developments, we can’t guarantee that.

David: Another thing worth mentioning is that the ability to manage the data from our new models is not advancing at the same rate the computational hardware is. One significant challenge that many simulation groups, including ours, face is managing the vast amount of output generated by these models. We need to find ways to make the volume of data more manageable. One approach we’d like to implement is running ensembles of individual simulations all at once and then collecting statistics from that group of simulations to create more compact output.

Q: What are some of the big challenges faced by the E3SM project as you dive deeper into exascale computing?

David: Finding individuals who understand both the science and computational systems required to write code for simulation models can be challenging. We often need to train them ourselves because it’s not as simple as assigning a programmer to create an object. Each team member needs to have a solid understanding of the science and a deep understanding of the machine.

Mark leads our computational science group, which consists of a small, tight-knit group of experts totaling around 20 members. It has taken us 10 years to assemble this team, and we would find it difficult to replace any one of them because each possesses a unique combination of aptitudes, experience, and skills that are not easily replaceable.

Q: That’s a great segue into the “I Am HPC” theme for SC23. What advice would you give to students or other researchers interested in getting involved with the E3SM program?

Mark: For E3SM model development, Dave is right that scientists with domain expertise who also enjoy computing and software development are the best fit for this type of work. The level of computer and computational science knowledge required to work with these models today is significantly greater than when I started, so I would recommend both a background in mathematical modeling and the software engineering skills that make it easier to work with large complex codes.

Q: How do you see the findings and results of the E3SM project benefiting society, and what are some practical applications of its research?

David: Climate modeling as a whole, not just our models specifically, is incredibly important right now. Although these projections—and it’s crucial to note that they are projections rather than predictions—are imperfect, the output from various models will help policymakers make more informed decisions in the face of uncertainty. 

The fact that the climate is changing is widely recognized, and there are all kinds of people who need to make decisions based on timeframes of 10 to 30 years. For example, they need to decide where to build infrastructure or what size of heating and cooling systems to install in buildings. To be more useful, models must improve and become more location-specific and time-specific. That’s the direction we’re heading towards.

Mark: In a model development project, you try and make the best model you can, and you have your hands full just trying to do that. Fortunately, there’s a highly skilled community of model developers building models used by climate scientists for hypothesis testing, so we are making important progress.

Learn More

Lori Diachin Asks All the Important Questions

Dan, Verónica, and Dustin Listening Intently

Each volunteer body meets regularly according to their team’s objectives, with most meetings conducted remotely. However, building on the notion that some ideas are better hatched in person, there typically are six face-to-face meetings spread throughout the two years prior to the current year’s conference. As such, 54 volunteers and contracted professionals met in Indianapolis for the third scheduled in-person meeting in advance of SC23. During the two-day meeting the teams shared their progress, discussed needs, solved problems, and assessed their long-term objectives. This meet-up allowed everyone to align their goals and create synergies via impromptu collaboration. The volunteers worked hard during the day and enjoyed informal team-building opportunities at select Indianapolis venues in the evenings, such as dinner at The Garage Food Hall.

SCinet Members Rocking Game Night

Dorian Arnold Speeding Towards November

SC23 Planning Committee