ENGINEERING AND CFD
EXASTEEL - Bridging Scales For Multiphase Steels
Principal Investigator:
1) Axel Klawonn, 2) Oliver Rheinbach
Affiliation:
1) Mathematical Institute of the University of Cologne (Germany), 2) Institute of Numerical Analysis and Optimization, Technische Universität Bergakademie Freiberg (Germany)
Local Project ID:
hfg01
HPC Platform used:
JUQUEEN of JSC
Date published:
The project EXASTEEL is concerned with parallel implicit solvers for multiscale problems in structural mechanics discretized using finite elements. It is focussed on modern high strength steel materials. The higher strength and better ductility of these materials largely stems from the carefully engineered grain structure at the microscale. The computational simulations used therefore take into account the microstructure, but without resorting to a brute force discretization (which will be out of reach for the foreseeable future). The researchers' approach combines a computational multiscale approach well known in engineering (FE) with state-of-the-art parallel scalable iterative implicit solvers developed in mathematics.
The EXASTEEL Project
In the FE2 (Finite Element2) algorithm, no material law is known at the macroscopic level. Instead, in each Gauß integration point of the macroscopic problem, a representative volume element (RVE) is attached which carries the microstructure of the material. For the microstructure, a material law is assumed to be known, e.g., from experiments. In each of these macroscopic Gauß integration points, a boundary value problem for the microscopic RVE is solved. The resulting stresses are averaged and, using a consistent tangent, given back to the macroscopic level. The researchers' simulation code FE2TI implements this algorithm and is permanently extended and improved within a strong co-design using new results from algorithm development, improved material modeling, and performance engineering [1]. Based on simulations using the complete JUQUEEN system hosted at the Jülich Supercomputing Centre, the code FE2TI has been awarded the High-Q-Club membership1 in 2015. The PIs of the EXASTEEL research consortium come from the fields of computational mathematics (A. Klawonn, O. Rheinbach), material science/engineering (D. Balzani, J. Schröder), and computer science (G. Wellein, performance engineering; O. Schenk (EXASTEEL-2), PARDISO sparse solver).
Table 1: FE production runs using the FE2TI software; see also [1].
Copyright: Technische Universität Bergakademie FreibergInexact FETI-DP
To solve sparse linear systems, the scientists have developed a new inexact FETI-DP (Finite Element Tearing and Interconnecting - Dual Primal) method without sparse direct solvers for nonlinear problems [2]. This method is designed to be more memory efficient compared with classical FETI-DP methods, where the memory consumption is dominated by LU- or Cholesky-factorizations of sparse matrices.
Figure 1: Scaling up a linear elasticity problem [2].
Copyright: Technische Universität Bergakademie FreibergNonlinear FETI-DP
Current nonlinear FETI-DP developments are based on a “decomposition before linearization” paradigm and thus, since many localized nonlinear problems occur, have the potential to increase the local work and concurrency [3,4]. The researchers developed different nonlinear variants for use in the FE2TI package; see Fig. 2.
Figure 2: Comparison of different nonlinear FETI-DP methods (NL1 - NL4) with Newton-Krylov-FETI-DP (NK).
Copyright: Technische Universität Bergakademie FreibergAMG for elasticity
In all of the reserachers' inexact FETI-DP implementations, an Algebraic Multigrid (AMG) preconditioner for the FETI-DP coarse problem was used. But the parallel AMG solver is also highly scalable and can efficiently make use of more than half a million parallel processes on its own. The scientists have considered recent AMG variants tailored for systems of PDEs and adapted especially to elasticity problems [5]. In this project, we have cooperated with the authors of BoomerAMG (Lawrence Livermore National Laboratory). Here, in H-AMG-LN, the BoomerAMG preconditioner uses a special interpolation which exactly interpolates the rigid body modes. In Fig. 3, different approaches for systems of PDEs are compared: U-AMG, H-AMG, H-AMG-LN. The most recent H-AMG-LN approach, tailored for linear elasticity, clearly performs best. For details, see [5]. It is therefore now used for elasticity problems where possible.
Figure 3: Comparison of different AMG methods for systems of PDEs provided in BoomerAMG; see [2] and [6].
Copyright: Technische Universität Bergakademie Freiberg1 http://www.fz-juelich.de/ias/jsc/EN/Expertise/High-Q-Club/FE2TI/_node.html
Acknowledgements
The computing time allocation on HPC system JUQUEEN installed at JSC currently is the main source of supercomputing time for the interdisciplinary research project EXASTEEL within the DFG priority program Software for Exascale Computing (SPPEXA, www.sppexa.de).
References
[1] D. Balzani, A. Gandhi, A. Klawonn, M. Lanser, O. Rheinbach, and J. Schröder, One-way and fully-coupled FE methods for heterogeneous elasticity and plasticity problems: Parallel scalability and an application to thermo-elastoplasticity of dual-phase steels, 2015, Accepted to Lect. Notes Comput. Sci. Eng., TUBAF Preprint: 2015-13, http://tu-freiberg.de/fakult1/forschung/preprints.
[2] Axel Klawonn, Martin Lanser, and Oliver Rheinbach, A highly scalable implementation of inexact Nonlinear FETI-DP without sparse direct solvers, vol. 112, 2016, TUBAF Preprint 2015-17 at http://tu-freiberg.de/fakult1/forschung/preprints.
[3] A. Klawonn, M. Lanser, and O. Rheinbach, Nonlinear FETI-DP and BDDC methods, SIAM J. Sci. Comput., 36, no. 2, A737–A765, 2014.
[4] Axel Klawonn, Martin Lanser, and Oliver Rheinbach, Toward extremely scalable nonlinear domain decomposition methods for elliptic partial differential equations, SIAM J. Sci. Comput., 37, no. 6, C667–C696, 2015.
[5] A.H. Baker, A. Klawonn, T. Kolev, M. Lanser, O. Rheinbach, and U.M. Yang, Scalability of classical algebraic multigrid for elasticity to half a million parallel tasks, Lecture Notes in Computational Science and Engineering, 113, 113–140, 2016, TUBAF Preprint 2015-14 at http://tu-freiberg.de/fakult1/forschung/preprints.
[6] Axel Klawonn, Martin Lanser, and Oliver Rheinbach, “Exasteel - From micro to macro properties”, in: InSiDE - Innovative HPC in Germany, Autumn 2016, A. Bode, Th. Lippert, and M.M. Resch, (Eds.), vol. 14 (2), pp. 80–85. 2016, http://inside.hlrs.de/editions/16autumn.html#exasteel-from-micro-to-macro-properties
Principal investigator:
1. Prof. Dr. Axel Klawonn - Mathematisches Institut, Universtität zu Köln - e-mail: axel.klawonn [at] uni-koeln.de
2. Prof. Dr. Oliver Rheinbach - Institut für Numerische Mathematik und Optimierung, Technische Universität Bergakademie Freiberg - e-mail: oliver.rheinbach [at] mathematik.tu-freiberg.de
Project contributors:
1. Dr. Martin Lanser - Mathematisches Institut, Universität zu Köln - e-mail: martin.lanser [at] uni-koeln.de
2. Prof. Dr.-Ing. Jörg Schröder - Institut für Mechanik, Abt. Bauwissenschaften, Fakultät für Ingenieurwissenschaften,
Universität Duisburg-Essen - e-mail: j.schroeder [at] uni-due.de
3. Prof. Dr.-Ing. Daniel Balzani - Institut für Mechanik und Flächentragwerke, Fakultät Bauingenieurwesen, TU Dresden -
e-mail: daniel.balzani [at] tu-dresden.de
4. Prof. Dr. Gerhard Wellein - Department Informatik, Technische Fakultät, Universität Erlangen-Nürnberg -
e-mail: gerhard.wellein [at] rrze.uni-erlangen.de