BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Denver
X-LIC-LOCATION:America/Denver
BEGIN:DAYLIGHT
TZOFFSETFROM:-0700
TZOFFSETTO:-0600
TZNAME:MDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0600
TZOFFSETTO:-0700
TZNAME:MST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260422T000711Z
LOCATION:710
DTSTART;TZID=America/Denver:20231112T170000
DTEND;TZID=America/Denver:20231112T171000
UID:submissions.supercomputing.org_SC23_sess427_misc326@linklings.com
SUMMARY:Lightning Talk:  Inherent Checkpointing Properties of Nested Paral
 lelism
DESCRIPTION:Stanislav Bratanov (Intel Corporation)\n\nOur discovery was a 
 complete accident. We worked on nested parallelism in OpenMP for easier wo
 rk decomposition and better work balancing:\n\n#pragma omp parallel num_th
 reads(N)\n{\n    #pragma omp parallel num_threads(M)\n    {\n        #prag
 ma omp parallel num_threads(K)\n        {\n        }\n    }\n}\n\nCreating
  thousands of threads (for real-life N, M, K values) was out of question, 
 and queuing thousands of tasks at each nesting level had a significant seq
 uential overhead, so we came up with a concept of task groups.\n\nA task g
 roup is a descriptor that has information on the entire data range for all
  tasks and on the number of tasks expected to process that data range (N, 
 M, or K, for each nesting level, respectively). \n\nThe task group descrip
 tor can be queued by the main thread, and worker threads will create indiv
 idual tasks from that descriptor, in parallel. Each individual task is res
 ponsible for finding its portion of the data range based on the task’s ind
 ex within the group.\n\nThat sounded like how GPU kernels typically worked
 . Hence, we converted nested parallel task groups into GPU kernels at a pr
 e-compilation stage, to let our runtime offload those kernels (or task gro
 ups) to GPUs. \n\nNow, when we already have individual kernels that can lo
 cate their data, why not compile those kernels into self-contained librari
 es that can be executed on remote systems?  That’s how each OpenMP-like ne
 sted parallel region became a task group, a kernel, and a library. \n\nAnd
  then we ran into all sorts of issues once we tested the runtime on loosel
 y coupled nodes on a local network (some GPUs were not super-reliable eith
 er).\n\nMore thinking brought understanding that our runtime actually “kno
 ws” whether the execution on a remote node or a GPU was unsuccessful, so i
 t can report that status back to the main program.\n\nWe were able to craf
 t a runtime that enables nested parallel execution on remote systems and G
 PUs (and GPUs on remote systems) and reports the execution status for each
  nesting level of parallel work.\n\nWhat a programmer needs to do now, is 
 simply check for error after the return from a parallel region and restart
  execution (given the inputs are separate from outputs and have not been o
 verwritten):\n\n#pragma omp parallel num_threads(N)\n{\n    while(1)\n    
 {\n        int error=0;\n        #pragma omp parallel num_threads(M) statu
 s(&error)\n        {\n            #pragma omp parallel num_threads(K)\n   
          {\n            }\n        }\n        if(!error)\n        {\n     
        break;\n        }\n    }\n}\n\nWe implemented N-body simulation (mi
 llions of bodies) and ran it on two systems: one significantly more powerf
 ul than the other, to ensure active work-stealing over network. The experi
 ment took roughly 30 minutes, which gave us plenty of time for disrupting 
 it by deleting all intermediate files generated by our runtime and switchi
 ng network connections on and off. After comparing the outputs with those 
 from an undisturbed run, we found the results to be identical (within the 
 GPU floating-point rounding error). \n\nThat is why we now advocate for ne
 sted parallelism as a promising solution for remote parallel execution, wi
 th “natural” checkpoints and simple restarting capabilities at each nestin
 g level.\n\nTag: Fault Handling and Tolerance\n\nRegistration Category: Wo
 rkshop Reg Pass\n\nSession Chairs: Gene Cooperman (Northeastern University
 ); Donglai Dai (Advanced Micro Devices (AMD)); Rebecca Hartman-Baker (Nati
 onal Energy Research Scientific Computing Center (NERSC), Lawrence Berkele
 y National Laboratory (LBNL)); and Bogdan Nicolae (Argonne National Labora
 tory (ANL), Illinois Institute of Technology)\n\n
END:VEVENT
END:VCALENDAR