Feasibility Studies in Multi-GPU Target Offloading

Many of the largest supercomputers are based on heterogeneous architectures with multiple general-purpose graphics processing units (GPGPUs) per compute node. While many APIs for GPU programming are vendor-specific, OpenMP offers a portable alternative. Therefore OpenMP target offloading is advantageous in terms of long-term code sustainability.

Further, many applications have already been parallelized with OpenMP. Hence the amount of work needed to port the code to GPUs may be limited. However, the support for the OpenMP 5.x specification is not equally mature across different compilers. Additionally, the multi-GPU support in the OpenMP 5.x specification is limited.

We explore what is possible with the Nvidia NVC compiler. We present a case study of solving the Poisson equation on multiple GPGPUs to outline which approaches for multi-target offloading give good results. We find that a task-based multi-GPU implementation leads to better performance than generating deferrable tasks with the clause.

We demonstrate that data transfers and computations can be fully overlapped by using only the subset of the OpenMP specifications, which is supported in the 22.3 release of the Nvidia NVC compiler. For compute nodes with multiple Nvidia A100 or V100, we obtain close to ideal strong scaling when increasing the number of accelerators.