ASTROPHYSICS

The choice of SYCL presented several advantages that facilitated the task: unlike proprietary programming models such as NVIDIA’s CUDA, SYCL is a crossplatform layer, not limited to a single vendor’s hardware: this alleviates the need for redundant development, maintenance and coordination of multiple GPU ports. At the same time, SYCL allows for more fine-granular control than other, more general programming methods such as OpenMP offloading.

In particular, the most notable features of DPEcho are:

• most methods and algorithms of ECHO 3D-HPC (the most notable exception being the UCT divergence-cleaning);
• CMake integration for structured compilation;
• simplified device selection with SYCL device aspects;
• the use of Unified Shared Memory (USM) for a more familiar and efficient programming mode for the ease of science users;
• portability to most common CPUs and GPUs with no code change, including good performance on heterogeneous hardware.

The latter can easily be achieved thanks to the Intel LLVM free compilers, or the free-to-use plugins developed by Codeplay for use with Intel’s oneAPI toolkit which enable the generation of code for NVIDIA’s or AMD’s GPUs without the need to compile LLVM oneself.

Thus, after the kick-off phase on SuperMUC-NG, it was possible to execute DPEcho using hardware by all major vendors, on the Intel Data Center GPU Max Series (formerly Ponte Vecchio), the NVIDIA H100 and the NVIDIA A100, the AMD Instinct MI100, and on a number of Intel CPUs.

                                                            ____________________________________

                                                              ____________________________________

For the evaluation, we setup a representative GR-MHD test – the propagation of plane-parallel Alfvén waves (magnetic waves powered by magnetic field tension) along the 3D diagonal of a cubic simulation box (Figure 1), with periodic boundary conditions along all directions. Alfvén waves are dispersionless, thus especially suitable for testing the accuracy of numerical schemes. We aim at efficiently filling the GPU memory and execution units, in order to minimise the relative SYCL initialisation and scheduling overhead, and test production-ready configurations. We therefore perform a scaling test over the problem size (at different resolutions) from 64 to 512 grid elements per side. Given the relatively high resolution we achieve, we make the test more demanding by fitting multiple wave creases along each side. Performance measurements are presented in Figure 2. For tile-able GPUs, we show both single-tile (ST in the figure) and implicit scaling (i.e. using the full hardware, IS in the figure) results. Besides a predictable variability due to this fact, all the tested hardware brings considerable speedup over the pure CPU runs. Despite the minimal data transfer granted by the USM and malloc device, all GPUs reach a plateau of performance only for larger boxes (>192ˆ3), due to scheduling overhead and device utilisation.

                                                          ____________________________________

                                                               ____________________________________

In a further test, we evaluated the weak scaling behavior of DPEcho on multiple GPUs. We ran configurations ranging from one to 192 nodes of the upcoming cluster SuperMUC-NG Phase 2, a cluster comprising 236 nodes with four Intel Data Center GPU 1550 accelerators each. The results, given in Figure 3, show that DPEcho can scale effortlessly to enable larger-scale simulations.