ENGINEERING AND CFD

Figure 2 illustrates the generation of sound by a fluid in motion with a jet from a nozzle. The noise-generating fluid motion usually can be found in a comparably small area around the structure in the ExaFSA setting. Thus, a spatial separation can be naturally found for these scenarios. In Figure 2, the jet with turbulent fluid motion can be clearly separated from the area where pressure waves are transported. The idea in the coupled simulation is now to gradually simplify the equations to solve them as soon as possible. Starting with the full compressible Navier-Stokes equations, including friction, the first simplification that can be done is neglecting the viscous terms when there are no more shear flows. This results in nonlinear inviscous Euler equations. Finally, when the variation in the state gets sufficiently small, the nonlinearities can be neglected and only the linearized Euler equations need to be solved.

For the surface coupling between the individual domains, we use the general library preCICE [2]. This tool allows the coupling of various solvers via a generic interface description. This allows us to combine dedicated solvers for each physical domain. For the current investigation, we first utilize the same solver for both the fluid and the acoustic domain, albeit with different discretizations. Our solver, Ateles [3], is a Discontinuous-Galerkin solver with a modal basis. This is especially well-suited for linear problems like acoustic wave propagation. A high spatial scheme order can be utilized with this scheme, which reduces the required memory to represent the solution accurately.

Nonlinear equations, however, drastically increase the computational effort in this scheme for high orders. Nevertheless, we employ the solver for this project also for the fluid domain, though with a low spatial scheme order. Looking forward, we plan to replace the solver in the fluid domain.

During this project we computed several jet configurations and improved the scalability of the approach. The communication algorithm in preCICE was changed and some load-balancing was introduced to account for increased computational efforts for the interpolation in the high-order scheme.

A large part of the project was concerned with the generation of a reference solution with a large fluid domain without coupling. We then investigated the feasibility and accuracy of the coupled simulation. This showed good agreement with the produced reference and after the mentioned improvements also a good runtime. We use more than 20 million core hours on the various investigations, with around a third of it spent on the production of the reference for comparison.

In most runs, we used 16,384 cores per simulation, as this proved to be the most feasible queueing option. The resulting data is stored in one file per point in time and domain. Typically, 100 points in time were used for each simulation resulting in around 200 files where each contains the complete solution in space and is also suitable to restart the simulation. Tracking specific probes in the domain with a higher time resolution produces some other, smaller files. But these are only few and small. In the fluid domain, the solution requires around 10 GB of disk space for a mesh with around 30 million elements. In the acoustic domain with a high scheme order, less degrees of freedom are required and the solution data fits into files of less than half that size. In total, each simulation produced roughly 1.5 TB of data. In the course of the project, we utilized more than 10 TB of disk space for the various investigations.

On-going Research / Outlook

The just finalized phase 1 of the ExaFSA project focused on setting up the framework and gaining experience with the general quality of the coupling algorithms, especially on the data mapping between the processes. Future work focuses on the simulation of real-world fluid-structure-acoustic interaction to bring new insights into such applications and realize computational optimization of, for example, the sound design of aircraft or wind turbines.

We can now build on the foundation and improvements that we were able to establish during this initial project. The detailed simulation of the problem, including also the fluid-structure interaction, further increases the computational costs. The follow-up work on these large-scale investigations have already started in a new project.

References and Links

[1] ipvs.informatik.uni-stuttgart.de/SGS/EXAFSA/
[2] H.-J. Bungartz, F. Lindner, B. Gatzhammer, M. Mehl, K. Scheufele, A. Shukaev and B. Uekermann. preCICE -- A Fully Parallel Library for Multi-Physics Surface Coupling. In Computers and Fluids. Elsevier, 2015.
[3] Jens Zudrop. 2015. Efficient Numerical Methods for Fluid- and Electro-dynamics on Massively Parallel Systems, Ph.D. Dissertation, RWTH Aachen, Germany

Scientific Team:

Principal Investigator: Sabine Roller¹
Researchers: Verena Krupp¹, Harald Klimach²
Project partners: Benjamin Uekermann²

¹ Institute of Simulation Techniques and Scientific Computing, University of Siegen
² Department of Scientific Computing (Informatics), TU München

Scientific Contact:

Prof. Dr.-Ing. Sabine Roller
Simulationstechnik und Wissenschaftliches Rechnen (STS)
Universität Siegen
Adolf-Reichwein-Straße 2, D-57068 Siegen (Germany) 
e-mail: sabine.roller[at]uni-siegen.de