Staff from the High-Performance Computing Center Stuttgart (HLRS), Jülich Supercomputing Centre (JSC), and Leibniz Supercomputing Centre (LRZ)—all members of the Gauss Centre for Supercomputing (GCS)—recently came together to participate in the SCAsia (SCA20) Data Mover Challenge.
The second annual competition invites supercomputing centres from around the world to innovate and use novel approaches to move large amounts of data around the world as quickly and efficiently as possible.
Due to the lingering threat surrounding the novel coronavirus (COVID-19), the physical conference was cancelled, but with the competition already complete, the awards committee was still able to name winners in different categories. The GCS team won for the “best systemic approach.”
“We were honored to come away from the competition with the award for best systemic approach to data transfer,” said Jochen Buchholz, staffer at HLRS and leader of the GCS team. “As datasets continue to become larger and more complex, sharing best practices with other leading HPC organizations has grown increasingly valuable in ensuring that we are helping researchers spend less time moving data and more time analyzing it.”
For the competition, each team had 3 days to test their approach, one for running the challenge itself, and one for an interview and questions from the judging committee. For the competition itself, the team had to send 1 terabyte worth of data across the globe, with file sizes ranging from a few kilobytes to several gigabytes. Essentially, the committee wanted to not just test the teams’ abilities to transfer a large amount of data, but also see how they innovated when it came to a variety of file sizes.
For the competition, the team used the UNICORE File Transfer (UFTP) tool, developed at JSC, in order to show its potential not only in transferring data between the three GCS centres, but also globally. However, the team hit stumbling blocks at the beginning of the competition when it encountered network issues. In order to address this issue, the team began manually debugging alongside the competition committee, eventually helping solve the issues limiting bandwidth for the competitors and improving the team’s performance in the process. It was later revealed that all teams had similar issues with the underlying network. Despite these challenges, the team managed to utilize the given 100 Gb/s bandwidth temporarily and optimize transfers for large numbers of small files. For this effort, the team was awarded the best systemic approach.
In addition to the GCS team, the National Institute of Informatics (Japan) won the most innovative award, iCAIR /Starlight (USA) won the best speed and science innovation award, and the Japan Aerospace Exploration Agency won the experimental excellence award.
For more information on the Data Mover Challenge, click here.