PSC Simulation Support for Novel Accelerator Concepts
Ludwig-Maximilians-Universität München, Faculty of Physics, Chair for Computational and Plasma Physics
Local Project ID:
HPC Platform used:
SuperMUC of LRZ
Since the moment ultra-short high-power lasers became available, their potential use for accelerators is of great interest as the charge separation in plasmas can induce enormous electromagnetic field strengths on a sub-micrometer scale. Accurate modeling of the plasma dynamics is essential for the understanding of how the desired acceleration properties can be obtained. Considerable research efforts, both on a theoretical and experimental level, are still needed to achieve ambitious goals, such as medical applications for accelerated protons via laser interaction with mass-limited targets (MLT). These MLTs, such as micro-foils, nano-clusters, needles and wires are potential sources of fast particles and high-energy photons used for purposes such as imaging or treatment planning. The high-energy photons generated in non-linear laser interaction are also of interest in the context of ultra-short attosecond X-ray pulses (AXP) that are required for the imaging of biological processes like protein folding or the behavior of Rhodopsin in the human retina.
Inspired by the results of larger full kinetic PSC simulations, we also have found a new ion acceleration regime, called Ion Wave Breaking Acceleration (IWBA) , where collimated and mono-energetic 200-400 MeV ion beams could be produced with available experimental parameters.
While these approaches use lasers to accelerate electrons which for their part accelerate protons/ions, the AWAKE project in contrast uses highly energetic protons for a new linear lepton accelerator concept for multi GeV electrons on some tens of meters instead of kilometers with the help of wake fields.
In previous projects, the technology necessary to run 10m box size simulations with micrometer resolution was established, such as enhanced memory management, better parallelism and increasing the I/O speed of checkpoints to an average of 105 GB/s to be able to run the simulation for full four weeks of pure wall-clock time on 32.768 cores. In the course of this project, the basic baseline case of AWAKE was simulated, producing dozens of TB with each output step. However, it turned out that instead of simulations with increased density, the experimentalists required several observables/output quantities with much higher temporal resolution. As a consequence, the major effort of the new project was put into increasing global communication efficiency to enable heavy “on-the-fly” data processing and analysis on the scale of tens of thousands of cores. This was a major challenge, but solving some serious issues dramatically increased the performance, so a lot of core hours could be invested in some new complex simulations involving quantum electrodynamical (QED) effects that were originally planned for a later project.
The Plasma Simulation Code (PSC)  is a general-purpose framework to solve the extended Maxwell-Vlasov-Boltzmann system of equations via the PIC approach. The original FORTRAN version evolved to a modern modularized C simulation framework supporting bindings to FORTRAN as well as C/CUDA and features selectable field and particle pushers. The PIC approach is well-known for its good scaling capability via configuration space parallelization. A Hilbert-Peano space-filling curve is used for efficient, dynamic and adaptive load and memory balancing allowing for complex and dynamic geometries.
The AWAKE project studies the interaction of a 450 GeV proton beam of the SPS pre-accelerator at CERN with a 10 m long plasma. Moving window technology allows for reducing the active memory footprint and the costs by a factor of 30 to about 3% of a full simulation. Nevertheless, every single output dump still takes 3 TB and checkpoints may take even up to 12 TB. For collaboration with experimentalists, these data were still not sufficient, as time-averaged observables requiring information from every single time step were also demanded. Therefore, instead of studying the beam filamentation at higher plasma densities, we redesigned the simulation and implemented heavy inline data processing and analysis in consultation with Dr. Konstantin Lotov (Head of Simulation Efforts of AWAKE).
We also ran a set of simulations with lower plasma density in the linear regime as benchmark with quite convincing results. However, especially the inline data processing on large scales led to a significantly increased amount of collective communication. This revealed an MPI issue most likely related to the IBM-MPI behavior on SuperMUC Phase 1 (Fig. 2).
Exhaustive investigations carried out together with experts from the LRZ Astrolab and engineers of IBM found leads that the problem could be related to the interaction of the collective and point-to-point communication patterns in the PSC with intra-node RDMA transfers. After having these disabled the problems did not reappear.
Another big challenge consisted in the fact that the available IBM implementation, “pami_tune” provided only one algorithm for “MPI-GatherV”, which apparently used only “MPI-Root” as receiver. As INTEL did not scale to that many SuperMUC islands (at that time), a custom-tuned tree-like algorithm was written for the PSC, decreasing the wall-clock time from several hundred seconds (on 4-8 islands) to usually less than 40 ms, which is a speed-up factor of more than 2000.
We are currently summarizing the challenges inherent in the execution of this extreme scale simulation in a paper entitled “Enabling the First Fully Kinetic 3D Simulation of the AWAKE Baseline Scenario”, K. Bamberg, N. Moschüring, P. Böhl, K. Lotov, H. Ruhl (2018).
After having resolved these issues, we were able to provide the experimentalists with all the information they needed (Fig. 3) to reach the initial goal of confirming that the reduced codes (fluid-based and 2D cylindrically symmetric) can accurately represent the relevant physics as well as fully kinetic 3D simulations.
The Head of Simulation Efforts of AWAKE confirmed that the respective results showed high concordance, and by now also coincide with initial experimental results, e.g. the PSC simulations correctly showed that there would be no hosing, a result that can only be predicted with fully kinetic 3D simulations. This represents an unprecedented benchmark at this extreme scale for Particle-in-Cell codes. This work is currently in the process of publication as “First Fully Kinetic 3D Simulation of the AWAKE Baseline Scenario” by N. Moschüring, K. Lotov, K. Bamberg, F. Deutschmann, H. Ruhl (2018).
Ultra-thin foils (UTFs)
While AWAKE uses 450 GeV protons to accelerate leptons, the other subprojects use much more commonly available lasers to accelerate electrons. For instance, a short circularly polarized laser pulse can “press” a major fraction of the electrons out of a 10 nm thick carbon-like foil. The generated electric field strongly accelerates the electrons back to the ions, and relativistic effects create an even shorter AXP useful to “film” protein folding which happens on the time scale of 10e-18 seconds.
Together with AWAKE, the nano-foil project is one of our biggest simulations requiring half a trillion grid cells. And likewise, also only few output steps are possible. The video below shows a 3D-Volume rendering of such a snapshot. During this project, the focus was placed on frequency analysis, which therefore also requires information from every time step. One goal consisted in the extension of the special output routines for better frequency sampling. In addition, we could reduce memory footprint of the simulation from 16 islands of SuperMUC Phase 1 (requiring “Block Operation”) to 8 islands, making the ultra-thin foil simulations much more feasible.
Mass-limited targets (MLTs)
Another approach consists in the longer/stronger acceleration of the electrons and the use of the thereby generated electric field to “pull” and therefore accelerate the protons. This is the direct contrast to AWAKE, where protons accelerate leptons.
Substantial progress was made in finding a parameter range that offers a higher ratio for the conversion of laser pulse energy into fast ion energy when using MLTs levitated in a Paul trap  to produce fast ions. In this project we tested different pulse shapes, field strengths and target geometries and densities.
By altering these parameters, the dynamic of the acceleration process can be shifted between different regimes like target normal sheath acceleration (TNSA), Coulomb Explosion (CE) and Radiation Pressure Acceleration (RPA). We studied the transition between these different dynamics and the properties of their fast ion spectrum and found that the maximum proton energies rise approximately linearly with pulse field strength for the near solid targets that were used in the experiments that were ran in parallel to the simulation efforts.
Due to the necessary resolution, a typical simulation requires about 20 billion grid cells and runs approximately 12 hours on one island on SuperMUC Phase 1. Larger ones require up to 4 islands.
The simulations revealed that the absorption of laser energy can be enhanced by lowering the density of the target at primary pulse interaction. Experimentally, this can be achieved by pre-expanding the target with a minor pulse before the main interaction. With a target at roughly critical density (nc), the laser can penetrate the target completely and we have an enhanced RPA effect in addition to the coulomb explosion observed so far in experiments and simulations.
Additionally, the length of the pulse is adjusted to the expansion time of the exploding target. This leads to a tenfold increase of maximum proton energy from roughly 25 MeV achieved in previous work  to 250 MeV, while keeping the pulse energy (2 J) and the laser peak intensity (8 ∙ 1020 w/cm2) roughly the same as before. If these results can be reproduced in experiment, it will be a significant improvement as compared to the older results, as ion energies of a few 100 MeV are then within reach at much smaller pulse energies that can be delivered by relatively common laser facilities. This set-up also concentrates the accelerated protons much more in the forward direction (Fig. 4/mp4-Video) compared with the Coulomb exploding situation. These results will be published as “New Target Parameters for Improved Ion Acceleration in Laser Irradiated MLT" by V. Pauw, P. Hilz, K.-U. Bamberg and H. Ruhl (2018).
Ion Wave Breaking Acceleration (IWBA)
Compared to the results of previous projects, the goal was to simplify the setup for experimentalists by finding laser parameter sets that should be commonly available for experimental laser physicists. The pulse form is now a pure Gaussian, in time as well as in space, and the laser intensity is lower than 1021 w/cm2 . The difficulty to overcome is then the reduced parameter window for the IWBA regime. For example, with an intensity of 6 ∙ 1020 w/cm2, the optimal initial plasma density has to be in the small range of 6.5nc to 8.5nc . Up to now, there is no good theory to analytically describe these phenomena. So it is necessary to scan the parameters very carefully to find the small window. Furthermore, the ion trapping happens in a very small region of space, therefore requiring high resolutions of at least 50 cells per micron [2, 3]. Larger runs (Fig. 5) usually require 2000-2560 cores taking 24-48 hours resulting in about 100 kilo-core-hours per run. The parameter scan consisted of several hundred runs (also smaller ones) on SuperMUC and also on its “little sister” Hydra (for “throughput”). The typical output is about two Terabyte per run being reduced to some Gigabytes during post-processing.
Some of the core-hours saved due to the change in the AWAKE simulation campaign and the increased communication efficiency were invested in the former mentioned projects (IWBA, MLT and UTFs) with the wonderful results described above and in the mentioned publications. However, another part of the freed core-hours could be used to address some Q.E.D projects that were originally postponed and not part of the grant, due to their complexity but nevertheless succeeded to produce already worthwhile results.
Q.E.D. Vacuum break down in counter propagating circularly polarized laser pulses
For very strong laser fields, effects like radiation reaction have to be accounted for. For the PSC a “photon” system was implemented for two reasons: On the one hand, the required resolution to resolve the radiation reaction is way too high even for moderate laser fields. On the other hand, the classic Maxwell equations are insufficient for stronger fields (a0 ~ 1000). The photons are generated by event generators for both cases, based on Monte-Carlo methods described in the theses of Constantin Klier and Fabian Deutschmann. Furthermore, adaptive time sub-cycle steps were necessary to implement correct dynamics and rates in very strong acceleration cases.
Because of the high memory requirements due to the dynamic production and rebalancing of particles, these simulations needed a complete island of SuperMUC Phase 1. The need for good statistics and the arithmetically complex equations to be solved required more than 120 wall-clock hours runtime, only for the results shown here.
The promising simulation results for nano targets shall be supported with experimental data in 2018. If the results hold up, further investigation into optimization of the experimental set-up and the parameters of pulse and target is of interest.
Given the success of the AWAKE simulations, the small discrepancy between PSC, L-Code and experiments might originate from minimally deviating initial conditions which might be researched by many shorter runs scanning different initial parameters. But the results suggest that for most of the realistic scenarios it is fortunately unnecessary to rely on full kinetic simulations.
The challenge of the IWBA project is that in order to explore more details near the wave-breaking point as described in , much higher resolution is required.
Karl-Ulrich Bamberg, Dr. Bin Liu, Nils Moschüring, Viktoria Pauw, Prof. Dr. Hartmut Ruhl (PI), all: Ludwig-Maximilians-Universität München, Faculty of Physics, Chair for Computational and Plasma Physics
AWAKE Collaboration (CERN), Max Planck Institute of Quantum Optics (MPQ)
 Liu, B., J. Meyer-ter-Vehn, and H. Ruhl. "Scaling of ion trapping in laser-driven relativistically transparent plasma." arXiv preprint arXiv:1803.06358 (2018).
 Liu, B., Meyer-ter-Vehn, J., Ruhl, H. and Bamberg, K.-U. “Laser Acceleration of Electrons, Protons and Ions”, SPIE conference proceedings (2017) Volume 10240, IV.
 Ostermayr, T. M., et al. "Proton acceleration by irradiation of isolated spheres with an intense laser pulse." Physical Review E 94.3 (2016): 033208.
 Ostermayr, T. M., et al. "A transportable Paul-trap for levitation and accurate positioning of micron-scale particles in vacuum for laser-plasma experiments." Review of Scientific Instruments 89.1 (2018): 013302.
 Caldwell, A., et al. "Path to AWAKE: Evolution of the concept." Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 829 (2016): 3-16.
Computational & Plasma Physics
Theresienstr. 37, D-80333 München (Germany)
e-mail: Karl-Ulrich.Bamberg [at] physik.uni-muenchen.de
LRZ project ID: pr74si