PSC Simulation Support for Novel Accelerator Concepts
Principal Investigator:
Hartmut Ruhl, Karl-Ulrich Bamberg
Affiliation:
Ludwig-Maximilians-Universität München, Faculty of Physics, Chair for Computational and Plasma Physics
Local Project ID:
pr74si
HPC Platform used:
SuperMUC and SuperMUC-NG of LRZ
Date published:
Since the moment ultra-short high-power lasers became available, their potential use for accelerators drew great interest, as the charge separation in plasmas can induce enormous electromagnetic field strengths on a sub-micrometer scale. Accurate modeling of the plasma dynamics is essential for understanding of how the desired acceleration properties can be obtained. Considerable research efforts, both on a theoretical and experimental level, are still needed to achieve ambitious goals, such as medical applications for accelerated protons via laser interaction with mass-limited targets (MLT). These MLTs, such as micro-foils, nano-clusters, needles and wires are potential sources of fast particles and high-energy photons used for purposes such as imaging or treatment planning. The high-energy photons generated in non-linear laser interaction with ultra-thin-foils are of interest e.g. in the context of ultra-short attosecond X-ray pulses (AXP) that are required for the imaging of biological processes like protein folding or the behavior of Rhodopsin in the human retina.
Inspired by the results of larger full kinetic PSC simulations, we also have found a new ion acceleration regime, called Ion Wave Breaking Acceleration (IWBA) [2], where collimated and mono-energetic 200-400 MeV ion beams could be produced with available experimental parameters.
While these approaches use lasers to accelerate electrons which for their part accelerate protons/ions, the AWAKE project in contrast uses highly energetic protons for a new linear lepton accelerator concept for multi GeV electrons on some tens of meters instead of kilometers with the help of wake fields.
In previous projects, the technology necessary to run 10m box size simulations with micrometer resolution was established, such as enhanced memory management, better parallelism and increasing the I/O speed of checkpoints to an average of 105 GB/s to be able to run the simulation for full four weeks of pure wall-clock time on 32.768 cores. In the course of this project, the basic baseline case of AWAKE was simulated, producing dozens of TB with each output step. However, it turned out that instead of simulations with increased density, the experimentalists required several observables/output quantities with much higher temporal resolution. As a consequence, the major effort of the new project was put into increasing global communication efficiency to enable heavy “on-the-fly” data processing and analysis on the scale of tens of thousands of cores. This was a major challenge, but solving some serious issues dramatically increased the performance, so a lot of core hours could be invested in some new complex simulations involving quantum electrodynamical (QED) effects that were originally planned for a later project.
The Plasma Simulation Code (PSC) [1] is a general-purpose framework to solve the extended Maxwell-Vlasov-Boltzmann system of equations via the PIC approach. The original FORTRAN version evolved to a modern modularized C simulation framework supporting bindings to FORTRAN as well as C/CUDA and features selectable field and particle pushers. The PIC approach is well-known for its good scaling capability via configuration space parallelization. A Hilbert-Peano space-filling curve is used for efficient, dynamic and adaptive load and memory balancing allowing for complex and dynamic geometries.
Ultra-Thin Foils (UTFs): Generation of high energy photons by the "slingshot" effect
Researchers: Karl-Ulrich Bamberg, Hartmut Ruhl
A very strong and very short (only a few cycles) – but experimentally available – circularly polarized laser pulse can “press” a major fraction of the electrons out of a 10 nm thick carbon-like foil. The generated electric field strongly accelerates the electrons back to the ions, and relativistic effects create an even shorter AXP useful to “film” protein folding which happens on the time scale of 10e-18 seconds.
Together with AWAKE (see below), the nano-foil project is one of our biggest simulations requiring half a trillion grid cells. Due to this only few output steps are possible. The video below (Fig. 0) shows a 3D-Volume rendering of such a scenario (reduced resolution). During this project, the focus was placed on frequency analysis, which therefore also requires information from every time step. One goal consisted in the extension of the special output routines for better frequency sampling. In addition, we could reduce memory footprint of the simulation from 16 islands of SuperMUC Phase 1 (requiring “Block Operation”) to 8 islands, making the ultra-thin foil simulations much more feasible.
Fig. 0/mp4 video: Volume-rendering of a 3D simulation of a 10nm diamond-like-carbon foil irradiated with an ultra-short circularly polarized laser pulse as a sketch for the nano-foil project. Green-yellow is the electron cloud, clearly rotating in the circular electric field. Black-white is the expanding ion background and blue-red is the reflected light. For the sake of clarity, the incoming laser pulse is not shown. © Ludwig-Maximilians-Universität München, Faculty of Physics
AWAKE: Linear lepton acceleration on the scale of meters instead of kilometers
Researchers: Nils Moschüring, Karl-Ulrich Bamberg, Hartmut Ruhl
The AWAKE project studies the interaction of a 450 GeV proton beam of the SPS pre-accelerator at CERN with a 10 m long plasma. Moving window technology allows for reducing the active memory footprint and the costs by a factor of 30 to about 3% of a full simulation. Nevertheless, every single output dump still takes 3 TB and checkpoints may take even up to 12 TB. For collaboration with experimentalists, these data were still not sufficient, as time-averaged observables requiring information from every single time step were also demanded. Therefore, instead of studying the beam filamentation at higher plasma densities, we redesigned the simulation and implemented heavy inline data processing and analysis in consultation with Dr. Konstantin Lotov (Head of Simulation Efforts of AWAKE).
Fig. 1: Visualization of the 450 GeV AWAKE ion beam (green), generated by the CERN SPS, inside the plasma (2D slice) and the accelerated electron witness beam (red/yellow). After 5m, micro-bunching and a strong wakefield (black/white pits) can clearly be recognized. © Ludwig-Maximilians-Universität München, Faculty of Physics
We also ran a set of simulations with lower plasma density in the linear regime as benchmark with quite convincing results. However, especially the inline data processing on large scales led to a significantly increased amount of collective communication. This revealed an MPI issue most likely related to the IBM-MPI behavior on SuperMUC Phase 1 (Fig. 2).
Fig. 2: The purple line represents the time for the MPI_reduce (sum) routine used by an important inline analysis. The same analysis was done between the timespan 6 to 8 and 10 to 12. The green line corresponds to the total wall-clock time necessary for a time-step. There is a strong correlation showing that most of the time is spent in this MPI-reduction, blocking the simulation. © Ludwig-Maximilians-Universität München, Faculty of Physics
Exhaustive investigations carried out together with experts from the LRZ Astrolab and engineers from IBM found leads that the problem could be related to the interaction of the collective and point-to-point communication patterns in the PSC with intra-node RDMA transfers. After their elimination the problems did not reappear.
Another big challenge consisted in the fact that the available IBM implementation, “pami_tune” provided only one algorithm for “MPI_GatherV”, which apparently used only “MPI-Root” as receiver. As INTEL did not scale to that many SuperMUC islands (at that time), a custom-tuned tree-like algorithm was written for the PSC, decreasing the wall-clock time from several hundred seconds (on 4-8 islands) to usually less than 40 ms, which is a speed-up factor of more than 2000.
After having resolved these issues, we were able to provide the experimentalists with all the information they needed (Fig. 3) to reach the initial goal of confirming that the reduced codes (fluid-based and 2D cylindrically symmetric) can accurately represent the relevant physics as well as fully kinetic 3D simulations.
The Head of Simulation Efforts of AWAKE confirmed that the respective results showed high concordance, and by now also coincide with initial experimental results, e.g. the PSC simulations correctly showed that there would be no hosing, a result that can only be predicted with fully kinetic 3D simulations. This represents an unprecedented benchmark at this extreme scale for Particle-in-Cell codes. This work has been published as “First Fully Kinetic 3D Simulation of the AWAKE Baseline Scenario” by N. Moschüring, K. Lotov, K. Bamberg, F. Deutschmann, H. Ruhl (2019 [7]).
Fig. 3: The maximum acceleration field of three runs, differently resolved, demonstrating clearly the need for high resolutions. The red curve corresponding to the required resolution for the AWAKE baseline case could only be obtained after resolving the mentioned MPI issues. The blue and green lines correspond to reference results, obtained by using reduced models (∇Φ as the derivative of the potential denotes sort of an averaged Ez). They show very good agreement [6]. © Ludwig-Maximilians-Universität München, Faculty of Physics
Mass-Limited Targets (MLTs): Ion acceleration for medical purposes
Researchers: Viktoria Pauw, Karl-Ulrich Bamberg, Hartmut Ruhl
The approach consists in the long/strong acceleration of the electrons and the use of the thereby generated electric field to “pull” and therefore accelerate the protons. This is the direct contrast to AWAKE, where protons accelerate leptons.
Substantial progress was made in finding a parameter range that offers a higher ratio for the conversion of laser pulse energy into fast ion energy when using MLTs levitated in a Paul trap [5] to produce fast ions. In this project we tested different pulse shapes, field strengths and target geometries and densities.
By altering these parameters, the dynamic of the acceleration process can be shifted between different regimes like target normal sheath acceleration (TNSA), Coulomb Explosion (CE) and Radiation Pressure Acceleration (RPA). We studied the transition between these different dynamics and the properties of their fast ion spectrum and found that the maximum proton energies rise approximately linearly with pulse field strength for the near solid targets that were used in the experiments that were run in parallel to the simulation efforts.
Due to the necessary resolution, a typical simulation requires about 20 billion grid cells and runs approximately 12 hours on one island on SuperMUC Phase 1. Larger ones require up to 4 islands.
The simulations revealed that the absorption of laser energy can be enhanced by lowering the density of the target at primary pulse interaction. Experimentally, this can be achieved by pre-expanding the target with a minor pulse before the main interaction. With a target at roughly critical density (nc), the laser can penetrate the target completely and we have an enhanced RPA effect in addition to the coulomb explosion observed so far in experiments and simulations.
Additionally, the length of the pulse is adjusted to the expansion time of the exploding target. This leads to an increase of maximum proton energy from 100 MeV for solid targets to 400 MeV for less dense targets with a pulse energy of 100 J. If these results can be reproduced in experiment, it will be a significant improvement as compared to the older results, as ion energies of a few 100 MeV are then within reach at much smaller pulse energies that can be delivered by relatively common laser facilities. This set-up also concentrates the accelerated protons much more in the forward direction (Fig. 4/mp4-Video) compared with the Coulomb exploding situation. The work was presented at the GSI-PHEDM conference Hirschegg in January 2020 as "PIC simulation of laser irradiated micro-plasma with varying density" and won the "Laser and Particle Beams" Young Scientist Award. The corresponding paper will soon be published.
Fig. 4/mp4-Video: A laser pulse concentrated to a size of 1/10th of a hair’s breadth turns a microscopic plastic sphere into hot plasma of many billion degrees, accelerating the electrons in the material close to the speed of light. / The ultra-fast ions generated in this process could be used to treat cancer or start fusion reactions. © Ludwig-Maximilians-Universität München, Faculty of Physics
Ion Wave Breaking Acceleration (IWBA): Experimentally easy available laser driven ion acceleration in gases and foam
Researchers: Bin Liu, Karl-Ulrich Bamberg, Hartmut Ruhl
Compared to the results of previous IWBA projects, the goal was to simplify the setup for experimentalists by finding laser parameter sets that should be commonly available for experimental laser physicists. The pulse form is now a pure Gaussian, in time as well as in space, and the laser intensity is lower than 1021 W/cm2 . The difficulty to overcome is then the reduced parameter window for the IWBA regime. For example, with an intensity of 6 ∙ 1020 W/cm2, the optimal initial plasma density has to be in the small range of 6.5nc to 8.5nc . Up to now, there is no good theory to analytically describe these phenomena. So it is necessary to scan the parameters very carefully to find the small window. Furthermore, the ion trapping happens in a very small region of space, therefore requiring high resolutions of at least 50 cells per micron [2, 3]. Larger runs (Fig. 5) usually require 2000-2560 cores taking 24-48 hours resulting in about 100 kilo-core-hours per run. The parameter scan consisted of several hundred runs (also smaller ones) on SuperMUC and also on its “little sister” Hydra (for “throughput”). The typical output is about two Terabyte per run being reduced to some Gigabytes during post-processing.
Fig. 5: 3D-iso-surface plot for a 3D simulation with a 4 Joule circularly polarized laser pulse with a peak intensity about 6 ∙ 1020 W/cm2. The pure Gaussian profile is common for experimental setups. The marked ions are accelerated to energies in the range of 60-80MeV. © Ludwig-Maximilians-Universität München, Faculty of Physics
Some of the core-hours saved due to the change in the AWAKE simulation campaign and the increased communication efficiency were invested in the former mentioned projects (IWBA, MLT and UTFs) with the wonderful results described above and in the mentioned publications.
However, another part of the freed core-hours could be used to address some QED projects that were originally postponed and not part of the grant, due to their complexity but nevertheless succeeded to produce already worthwhile results.
QED Vacuum break down in counter propagating circularly polarized laser pulses: Matter-antimatter production
Researchers: Fabian Deutschmann, Constantin Klier, Karl-UIrich Bamberg, Hartmut Ruhl
For very strong laser fields, effects like radiation reaction have to be accounted for. For the PSC a “photon” system was implemented for two reasons: On the one hand, the required resolution to resolve the radiation reaction is way too high even for moderate laser fields. On the other hand, the classic Maxwell equations are insufficient for stronger fields (a0 ~ 1000). The photons are generated by event generators for both cases, based on Monte-Carlo methods described in the theses of Constantin Klier and Fabian Deutschmann. Furthermore, adaptive time sub-cycle steps were necessary to implement correct dynamics and rates in very strong acceleration cases.
Because of the high memory requirements due to the dynamic production and rebalancing of particles, these simulations needed a complete island of SuperMUC Phase 1. The need for good statistics and the arithmetically complex equations to be solved required more than 120 wall-clock hours runtime, only for the results shown here.
Fig. 6: Left a) shows the field strength (Gaussian in space) while the center b) shows the plasma charge density in a rotating electric field (a0 = 1000). In the center a patch is clearly masking the field. The patch of clearly over critical density was formed by pair production and holding the particles in the focus, while initial conditions were 10 times under critical. On the right c) the temporal evolution of the density (radius versus time) is plotted, showing that the increasing electric field (Gaussian in time) pushes plasma away into the outer areas, but in the center, where the intensity is high enough, it holds the particles and seeds pair production. © Ludwig-Maximilians-Universität München, Faculty of Physics
Karl-Ulrich Bamberg, Dr. Patrick Böhl, Fabian Deutschmann, Constantin Klier, Dr. Bin Liu, Nils Moschüring, Viktoria Pauw, Prof. Dr. Hartmut Ruhl (PI), all: Ludwig-Maximilians-Universität München, Faculty of Physics, Chair for Computational and Plasma Physics
AWAKE Collaboration (CERN), Max Planck Institute of Quantum Optics (MPQ)
[1] www.plasma-simulation-code.net
[2] Liu, B., J. Meyer-ter-Vehn, and H. Ruhl. "Scaling of ion trapping in laser-driven relativistically transparent plasma." arXiv preprint arXiv:1803.06358 (2018).
[3] Liu, B., Meyer-ter-Vehn, J., Ruhl, H. and Bamberg, K.-U. “Laser Acceleration of Electrons, Protons and Ions”, SPIE conference proceedings (2017) Volume 10240, IV.
[4] Ostermayr, T. M., et al. "Proton acceleration by irradiation of isolated spheres with an intense laser pulse." Physical Review E 94.3 (2016): 033208.
[5] Ostermayr, T. M., et al. "A transportable Paul-trap for levitation and accurate positioning of micron-scale particles in vacuum for laser-plasma experiments." Review of Scientific Instruments 89.1 (2018): 013302.
[6] Caldwell, A., et al. "Path to AWAKE: Evolution of the concept." Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 829 (2016): 3-16.
[7] N. Moschuering et al. “First Fully Kinetic 3D Simulation of the AWAKE Baseline Scenario”, Plasma Phys. Control. Fusion 61 (2019): 104004
Karl-Ulrich Bamberg
Ludwig-Maximilians-Universität München
Arnold-Sommerfeld-Center (ASC)
Computational & Plasma Physics
Theresienstr. 37, D-80333 München (Germany)
e-mail: Karl-Ulrich.Bamberg [at] physik.uni-muenchen.de
http://www.theorie.physik.uni-muenchen.de/lsruhl/index.html
LRZ project ID: pr74si
September 2020