Flow Simulation at the Exascale

Tony Chan, President of KAUST

Tony Chan, President (former), King Abdullah University of Science and Technology

Mar 28, 08:40 - 09:00

KAUST Library

President's speech to open the conference. Brief Biography Professor Chan assumed his role as the third president of KAUST in September 2018 after nearly a decade as president of The Hong Kong University of Science and Technology. He led a strategic planning process to lead KAUST into its second decade, including growing the faculty and students by up to 50%. He has expanded KAUST's research emphasis from energy, water, food and environment to include digital and health, launching new initiatives in Artificial Intelligence, Smart Health, Cyber Security and Circular Carbon. He is increasing

Scaling up Hidden Fluid Mechanics

George Em Karniadakis, Professor, Applied Mathematics, Brown University, USA

Mar 28, 09:00 - 10:00

KAUST Library

We will review physics-informed neural network and summarize available extensions for applications in fluid dynamics. We will also introduce new NNs that learn functionals and nonlinear operators from functions and corresponding responses for system identification. The universal approximation theorem of operators is suggestive of the potential of NNs in learning from scattered data any continuous operator or complex system. We first generalize the theorem to deep neural networks, and subsequently we apply it to design a new composite NN with small generalization error, the deep operator network (DeepONet), consisting of a NN for encoding the discrete input function space (branch net) and another NN for encoding the domain of the output functions (trunk net). We demonstrate that DeepONet can learn multiscale operators spanning across many scales and trained by diverse sources of data simultaneously; we wil include examples from hyoersonics and bubble dynamics.

Non-ideal compressible-fluid dynamics: a challenge for theory and numerical methods

Alberto Guardone, Professor of Fluid Mechanics, Politecnico di Milano, Italy

Mar 28, 10:30 - 11:00

KAUST Library

In the close proximity of the liquid-vapour saturation curve and critical point, well-known thermodynamic phenomena including large compressibility and critical point effects results in very unusual fluid dynamics features, including non-ideal or rarefaction shock waves, mixed and split waves. This unconventional behaviour, which cannot occur in the ideal flow of dilute gases, is referred to as Non-Ideal Compressible-Fluid Dynamics or NICFD. The focus of this contribution is to review the theoretical background of NICFD and to discuss the impact of highly non-ideal conditions on the design and properties of numerical schemes for compressible flows. Exemplary flow fields will be presented and compared to available experimental data from the Test-Rig for Organic VApours (TROVA) of Politecnico di Milano, a unique facility in which supersonic flows in non-ideal conditions can be measured and observed. The quantification of the uncertainty of operating conditions and of fluid properties is discussed.

Industry-relevant implicit LES via spectral/hp element methods

Gianmarco Mengaldo, Assistant Professor of Fluid Mechanics, Department of Mechanical Engineering, National University of Singapore

Mar 28, 11:00 - 11:30

KAUST Library

We present the successful deployment of high-fidelity Large-Eddy Simulation (LES) technologies based on spectral/hp element methods to a real automotive car, namely the Elemental Rp1 model . The simulation presents the common challenges of an industry-relevant simulation, namely high Reynolds number and complex geometry. To the best of the authors' knowledge, this simulation represents the first fifth-order accurate transient LES of an entire real car geometry. Moreover, this constitutes a key milestone towards considerably expanding the computational design envelope currently allowed in industry, where steady-state modelling remains the standard. In this talk, we highlight the key developments that were required to achieve the simulation, from mesh generation to improvements in solver and numerical technology.

Double enriched finite volume spaces for the DNS of fluid-particle interaction - Research Conference

Gabriel Wittum, Professor, Applied Mathematics and Computational Sciences

Mar 28, 11:00 - Mar 30, 11:30

KAUST Library

We present double enriched finite volume spaces for the simulation of free particles in a fluid. This involves forces beeing exchanged between the particles and the fluid at the interface. In an earlier work we derived a monotithic scheme which includes the interaction forces into the Navier-Stokes equations by direct coupling. In multiphase flows oscillations and spurious velocities are a common issue. The surface force term yields a jump in the pressure and therefore the oscillations are usually resolved by extending the spaces on cut elements in order to resolve the discontinuity. For the construction of the enriched spaces proposed in this paper we exploit the Petrov-Galerkin formulation of the vertex-centered finite volume method (PG-FVM). From the perspective of the finite volume scheme we argue that wrong discrete normal directions at the interface are the origin of the oscillations. The new perspective of normal vectors suggests to look at gradients rather than values of the enriching shape functions. The crucial parameter of the enrichement functions therefore is the gradient of the shape functions and especially the one of the test space. The distinguishing feature of our construction therefore is an enrichment that is based on the choice of shape functions with consistent gradients. These derivations finally yield a fitted scheme for the immersed interface. Numerical tests were conducted using the PG-FVM. We demonstrate that the enriched spaces are able to eliminate the oscillations. We apply the method to the parallel computation of the “Drafting, kissing and tumbling” benchmark.

Simulation-enabled discoveries in wall-bounded turbulent flows

Mar 28, 11:30 - 12:00

KAUST Library

Turbulence remains the last frontier in nonlinear classical mechanics. We highlight how large-scale computing on supercomputers (specifically the CRAY XC40 Shaheen II at KAUST) has contributed to discoveries in turbulence. We discuss the limitations of brute force simulations and how employing turbulence models have enabled us to reach realistic flow Reynolds number and led to new insights in complex multi-scale nonlinear turbulent flows. Specifically, we will present two case studies. One is about the phenomenon of the drag crisis (wherein at below a specific critical Reynolds number there is a catastrophic increase in drag) and how subtle changes at the boundary translate to large scale effects. The second case study is on the generation and sustenance of electric fields in sandstorms, and the validation of our hypothesis that turbulence is the key driving force leading to large scale electric fields measured in sandstorms.

An embedded strategy for large scale incompressible flow simulations in moving domains

Ramon Codina Professor, Universitat Politècnica de Catalunya, Spain

Mar 28, 14:00 - 14:30

KAUST Library

In this work we describe a methodology to approximate the incompressible Navier-Stokes equations in time dependent domains. To deal with the motion of the domain, we employ a fixed-mesh method that we call fixed-mesh ALE. It consists of writing the equations in a moving ALE reference system but then projecting them onto a fixed background mesh. This implies that the boundaries of the elements do not necessarily coincide with the physical boundaries, and thus there is the possibility of badly cut elements. We use a Nitsche's type formulation to prescribe the boundary conditions and stabilise the bad cuts introducing a term that penalises the gradient of the unknown orthogonal to the finite element space in a patch that contains the badly cut element. The flow formulation is a stabilised finite element method that allows one to treat convection dominated flows and to use equal velocity pressure interpolation. Furthermore, this formulation can be shown to behave as an implicit large eddy simulation approach. A key issue is that the sub-grid scales on which the formulation depends are allowed to be time dependent; this fact has proved to be crucial for the robustness of the approach. Finally, the calculation of the velocity and the pressure is segregated by using a fractional step scheme designed at the pure algebraic level. The strategy described is applied to real life problems, and in particular to the simulation of the air flow generated by a train moving inside a tunnel.

How to generate solutions to technology and industry inspired by a scientific use of computational mechanics

Norberto Marcelo Nigro, Professor of Computational Mechanics, Professor of Numerical Methods in Transport Phenomena, National University of Littoral, Argentina

Mar 28, 14:30 - 15:00

KAUST Library

It is well known how humanity has contributed to technological progress based, long ago, on empirical results and currently seeking to use the advances that technology itself has to offer. This synergistic effect has accelerated in recent years and one of the factors that has led to this exponential growth is the advent of increasingly powerful computers that allow us to recreate the physical phenomena behind all innovation in virtual form. But it's not just about assigning the success of this synergy to the hardware. There is also a share of responsibility to the contribution that from the theory of computer science, from mathematics and from physics and chemistry has produced today the appearance of a discipline that we can summarize in computational mechanics. This ability to anticipate problems that may arise in the development of a product or process positions us more quickly and safely in an advantageous situation. It is not only anticipating, it is also better understanding what this kind of virtual radiography offers us when simulating a problem. In this work we intend to show some examples that, arising from industrial problems or from technology itself, allowed us to deepen with a scientific method in order to offer more and better solutions than those of good engineering practices developed long ago. In addition to being inside a land where a Technology Park operates (www.ptlc.org.ar), our Institute helps the growth of spin-offs, many of them inspired by lines of research supported by computational mechanics, which makes the minimum viable product more accessible for its delivery to the market.

Paradigms for a 21st Century University

David Keyes, Senior Associate to the President, King Abdullah University of Science and Technology

Mar 28, 19:20 - 20:20

KAUST Library

The King Abdullah University of Science and Technology (KAUST), founded in 2009 in a 45-sq km academic village on the shores of the Red Sea, was created to be a 21st century “House of Wisdom” in the tradition of the 9th century Bayt al Hikmah that gave the world much of its modern mathematics, physics, chemistry, and medicine. KAUST is singular, or nearly so, among universities worldwide. The environment is one in which both long-term high-risk curiosity-driven research and goal-oriented research thrive. It is a post-graduate-only (primarily doctoral) institution, structured organizationally to lower barriers to interdisciplinary work. It offers awesome research facilities. It hosts on its own campus research labs of major multinational corporations and it incubates start-ups. Its residential community has a global composition of more than 120 countries with no majority or “group think.” It is hosted by a dynamically awakening country full of opportunities for creative young talents. Any one of these highly desirable features can make a university stand out. KAUST has them all! But to understand what KAUST offers to researchers, students, collaborators, and the world of stakeholders one should know a bit more. We summarize KAUST’s research paradigms and pillars, its strategies, and its mission. We also assess KAUST after its first 12 years of operations.

Omar Knio, Dean of CEMSE, Opening remarks

Omar Knio, Professor, Applied Mathematics and Computational Sciences

Mar 29, 09:30 - 09:40

KAUST Library

Opening remarks for the 2nd day of the conference. Brief Biography Omar Knio is Professor of Applied Mathematics and Computational Science and Principal Investigator of the Uncertainty Quantification Research Group at the King Abdullah University of Science and Technology, Saudi Arabia Education and early career Knio earned his Bachelor of Engineering degree from the American University of Beirut, and his S.M. and Ph.D. Degrees in Mechanical Engineering from M.I.T. After a period as a Postdoctoral Associate at the Department of Mechanical Engineering at M.I.T., Knio joined the Mechanical

Are we challenging the status quo of compressible computational fluid dynamic frameworks?

Matteo Parsani, Associate Professor, Applied Mathematics and Computational Sciences

Mar 29, 09:45 - 10:20

KAUST Library

Together with the algorithm suitability to exploit current petascale and next-generation exascale supercomputers, robust, accurate, and structure-preserving discretizations are necessary for developing predictive computational tools. In this seminar, we will show how we leverage a multidisciplinary platform that integrates numerical analysis, physics, and high-performance computing for the analysis and development of novel numerical methods for ordinary and partial differential equations with provable properties such as nonlinear stability (entropy stability) and conservation, and structure-preserving techniques. These properties are critical for designing reliable, efficient, and self-adaptive solvers for complex geometries – an essential cornerstone for next-generation computational frameworks. Current classes of partial differential equations that we are working on are the compressible Navier–Stokes equations and the Eulerian model for compressible heat-conducting flows. We also use deep learning to complement and speed up the process of solving efficiently large-scale PDE-based problems. In this talk, we will summarize the progress we made in the last few years in the following areas: - Numerical analysis and algorithm development for robust, smart compressible flow solvers. - Development from the ground up of a new scalable hp-adaptive computational fluid dynamics (CFD) framework, a potential prototype of the future compressible solver as chartered by the NASA CFD 2030 vision. I will show applications and impact in the automotive and aerospace industry and in its extension for improving knowledge of flow physics in aeroacoustics.

An Immerse domain method for fluid-structure interaction with contact

Patrick Zulian, Scientific Collaborator, Euler institute, Università della Svizzera italiana, Lugano, Switzerland

Mar 29, 10:30 - 11:00

KAUST Library

Abstract We present an embedded approach for the numerical simulation of fluid-structure interaction problems. This work focuses on problems where the elastic structures undergo large deformations. Motivating applications are (1) the study of liquid diaphragm pumps used for digital inkjet printing and (2) the study of bio-prosthetic heart valves. In both applications, we assume that the fluid is Newtonian, and we solve a series of contact problems. First, we solve the contact interaction between the different immersed structures. Second, we constrain the movement of such structures to the

Opportunities and Challenges in Industrial HPC

Michael M. Resch, Professor, Institute of High Performance Computing (IHR), University of Stuttgart, Germany

Mar 29, 11:30 - 12:00

KAUST Library

Exaflop Computing is driven by research activities with a focus on the performance but little regard for operational issues as are required by industrial users. In this talk we will address the industrial view of Exascale Computing aiming at getting a better understanding of what it takes to exploit teh new technologies also in the industrial field.

Challenges and Opportunities in Reshaping Linear Algebra Libraries for HPC/AI Workloads: A Testcase with 3D Unstructured Mesh Deformations

Hatem Ltaief, Principal Research Scientist, Computer, Electrical and Mathematical Sciences and Engineering

Mar 29, 12:00 - 12:30

KAUST Library

Traditional HPC simulations and AI / Big Data applications face similar challenges when solving extreme-scale scientific problems: bulk synchronous parallelism, expensive data motion, high algorithmic complexity and large memory footprint. Processors and memory technology scaling have mitigated these challenges thanks to an exponential growth in processor performance but only a constant increase in memory speed and capacity. The free lunch is perhaps over as we approach the hard physical limit of silicon. The energy efficiency gap between communication and computation keeps widening and has even forced the hardware and software communities for an immediate action of co-design. We describe the challenges encountered during the last 15-year journey of reshaping high performance linear algebra libraries for massively parallel systems. We explore disruptive numerical algorithms and programming models required to continue supporting HPC applications as well as emerging AI workloads at the dawn of the exascale age. In particular, we assess our implementation using 3D unstructured mesh deformation based on Radial Basis Function interpolation in the context of the HiCMA numerical library. Our HPC software solution achieves significant performance superiority against state-of-the-art implementations on Shaheen-II (based on dual-socket 16-core Intel Haswell nodes), Hawk (based on dual-socket 64-core AMD Epyc Rome nodes), and Fugaku (based on 48-core Fujitsu A64FX nodes) Supercomputers.

Poster Session - Flow Simulation at the Exascale

Mar 29, 12:30 - 14:30

KAUST Library

We kindly invite you to join our Poster Session and explore new ideas and research efforts on the topics highlighted in this conference.

Massively Parallel Models of Human Hemodynamics

Amanda Randles, Assistant Professor of Biomedical Sciences, Duke University, USA

Mar 29, 18:00 - 18:45

KAUST Library

Abstract The recognition of the role hemodynamic forces have in the localization and development of disease has motivated large-scale efforts to enable patient-specific simulations. When combined with computational approaches that can extend the models to include physiologically accurate hematocrit levels in large regions of the circulatory system, these image-based models yield insight into the underlying mechanisms driving disease progression and inform surgical planning or the design of next generation drug delivery systems. Building a detailed, realistic model of human blood flow, however

Physics-Based Modeling in the Era of Big Data: Enhancing Physics-Based Models using Data-Based Approaches

Michael E. Mueller, Associate Professor, Princeton University, USA

Mar 29, 18:55 - 19:40

KAUST Library

With the increasing prevalence of data from increasingly large full-fidelity flow simulations, there is a growing temptation to rely solely on these rich databases and exclusively data-based modeling paradigms, essentially abandoning physics-based modeling. However, the challenge with exclusively data-based modeling will always be model extrapolation: even with exascale computing and beyond, some regimes of relevant parameter spaces will never be computationally accessible with full-fidelity flow simulations. Additionally, physics-based models, being grounded in fundamental governing equations, need not re-learn fundamental constraints such as conservation laws and thermodynamics bounds. Unfortunately, especially for complex flows, such as turbulent multi-physics flows, every exclusively physics-based model will ultimately need to rely to some degree on crude assumptions or model oversimplification. Therefore, an opportunity exists to develop truly predictive models that can leverage the best of both physics-based modeling and data-based modeling through hybridization. The emerging paradigm starts with a physics-based modeling framework and replaces sub-model components with the crudest assumptions with data-based sub-model components derived from full-fidelity flow simulation databases. This hybridized modeling paradigm allows for not only more accurate models but also models able to account for phenomena that previous exclusively physics-based models were unable to capture. Several relevant examples from turbulent reacting flows will be presented and discussed.

Powering HPC Discoveries through New Developments in the Exascale Computing Project

Dr. Lori Diachin, Principal Deputy Associate Director, Computing Directorate, Lawrence Livermore National Laboratory, Exascale Computing Project

Mar 29, 20:00 - 20:30

KAUST Library

This presentation will introduce work in the U.S. Exascale Computing Project (ECP), which is developing a capable computing software ecosystem that leverages unprecedented HPC resources to work toward predictive science capabilities to solve problems in the areas of climate, energy, and human health. We will give an overview of the applications and software technologies being developed as part of the ECP, where software complexity is increasing due to disruptive changes in computer architectures and the complexities of tackling new frontiers in extreme-scale modeling, simulation, and analysis. We will give examples of the challenges application teams have faced in the development of new algorithms and physics capabilities that perform well on GPU accelerated node architectures. We will also describe our integrated approach to the deployment of a suite of programming models and runtimes, math libraries, data and visualization packaged and development tools that comprise the Extreme-scale Scientific Software Stack (E4S). We will explain how E4S—a portfolio-driven effort in ECP to collect, test, and deliver the latest advances in open-source HPC software technologies—is helping to overcome challenges associated with using independently developed software tools together in a single application. We conclude by showcasing some of the latest results the ECP teams have achieved in developing new capabilities in a number of application areas.

Computational Investigation of the Effects of Chemistry on Mars Retropropulsion Environments

Eric Nielsen, Senior Research Scientist, Computational AeroSciences Branch, NASA Langley Research Center, USA

Mar 29, 20:30 - 21:00

KAUST Library

Human-scale Mars vehicles will require retropropulsion for descent and landing, replacing heritage supersonic parachute systems with an extended phase of powered flight. Due to the limitations of terrestrial testing in Mars-relevant conditions, design and analysis will increasingly rely on computational modeling and simulation. This paper provides an overview of a computational campaign investigating the aerodynamics of a Mars lander concept along various points on a powered descent trajectory including supersonic, transonic, and subsonic conditions using finite-rate chemistry. Simulations using unstructured grids containing billions of elements are performed at scale using thousands of GPUs, enabling run-times of a few days for each simulation presented.

MFEM: Accelerating Efficient Solution of PDEs at Exascale

Tzanio Kolev, Center for Applied Scientific Computing, Director of the Center for Efficient Exascale Discretizations, Lawrence Livermore National Laboratory, USA

Mar 29, 21:00 - 21:30

KAUST Library

Upcoming exascale architectures require rethinking of the numerical algorithms used in large-scale PDE-based applications. These architectures favor algorithms, such as high-order finite elements, that expose fine-grain parallelism and maximize the ratio of floating point operations to energy intensive data movement. In this talk we present an overview of MFEM , a scalable library for high-order finite element discretization of PDEs on general unstructured grids. We also report on recent work in the Center for Efficient Exascale Discretizations , a co-design center in the US Exascale Computing Project focused on next-generation discretization software and algorithms. Our approach to efficient operator evaluation is based on a "matrix-free" representation of the finite element operator, that factors a bilinear form into a series of sparse and dense components corresponding to the parallelism, mesh topology, basis, geometry, and pointwise physics in the problem. The operator decomposition exposes several layers of parallelism, enables the use of batched dgemss and tensor contractions, and only requires quadrature point values to be assembled for computing the action. This "partial assembly" formulation is a natural fit for modern HPC hardware, because it results both in less (nearly optimal) computation and less (optimal) data movement compared to assembling a global sparse matrix, therefore increasing performance and reducing time to solution. In addition to discussing efficient operator evaluation, we will provide an overview of the MFEM capabilities and applications to compressible hydrodynamics and electromagnetics. We will also review our work on performance optimizations for GPU architectures, high-order benchmarks and miniapps, scalable unstructured adaptive mesh refinement, high-order mesh optimization and matrix-free preconditioning. MFEM: Modular finite element library, http://mfem.org. Center for Efficient Exascale Discretizations, http://ceed.exascaleproject.org.