Towards CI-enabled, optimization-driven,

simulation-based decision making

Omar GhattasUltrascale Computing LabBiomedical Engineering

Civil & Environmental EngineeringCarnegie Mellon University

Initial and boundary conditions

• Cyberinfrastructure := integrated, distributedparallel supercomputers, clusters, fast networks, federated databases, middleware, software libraries and tools, algorithms, application codes

• Design := decision making • Focus on high end, high fidelity simulation• Optimization framework

From physical models to simulation-based decision making

physical model of natural physical model of natural or engineered systemor engineered system

mathematical modelmathematical model

numerical modelnumerical modelcomputer computer simulationsimulation

validationvalidation

SimulationSimulation--based decision makingbased decision makinge.g. design, control, operations, e.g. design, control, operations,

disaster response, manufacturing,disaster response, manufacturing,hazard assessment, planning, treatmenthazard assessment, planning, treatment

scalable scalable algorithms algorithms & solvers& solvers

data/observationsdata/observations

multiscalemultiscale models models

advanced geometry &advanced geometry &discretizationdiscretization schemesschemes

parameter inversionparameter inversiondata assimilationdata assimilationmodel/data error controlmodel/data error control

visualizationvisualizationdata mining/sciencedata mining/science

optimizationoptimizationstochastic modelsstochastic modelsuncertainty quantificationuncertainty quantification

verificationverificationapproximation approximation error controlerror control

CyberInfrastructure

CyberEngineering

Earthquake Modeling for Seismic Hazard Assessment Earthquake Modeling for Seismic Hazard Assessment VolkanVolkan AkcelikAkcelik , , JacoboJacobo BielakBielak, George Biros , George Biros ((UPennUPenn), Steven Day (SDSU), Omar ), Steven Day (SDSU), Omar GhattasGhattas, , LoukasLoukas KallivokasKallivokas (Texas), Harold (Texas), Harold MagistraleMagistrale(SDSU), David (SDSU), David O’HallaronO’Hallaron

Region of interest for Region of interest for 1994 Northridge 1994 Northridge earthquake simulation earthquake simulation

Adaptive grid resolves up to 1Hz freq. Adaptive grid resolves up to 1Hz freq. w/100 million grid pts; uniform grid w/100 million grid pts; uniform grid would require 2000x more pointswould require 2000x more points

SCEC geological model provides 3D soil SCEC geological model provides 3D soil properties in Greater LA Basinproperties in Greater LA Basin

Snapshot of simulated ground motion Snapshot of simulated ground motion (simulation requires 3hr on 6Tflops PSC (simulation requires 3hr on 6Tflops PSC machine, running at >80% parallel machine, running at >80% parallel effeff))

Comparison of observation with Comparison of observation with simulation (improved prediction simulation (improved prediction requires requires petaflopspetaflops capability)capability)

Inversion of surface observations Inversion of surface observations for 17 million elastic parameters for 17 million elastic parameters (right: target; left: inversion result)(right: target; left: inversion result)

MultiscaleMultiscale Blood Flow Modeling for Artificial Heart Device DesignBlood Flow Modeling for Artificial Heart Device Design

At macroscopic (device) scales:At macroscopic (device) scales:•• Development of artificial heart assist Development of artificial heart assist

device at device at UnivUniv Pitt Med Center (Pitt Med Center (AntakiAntaki))•• Numerous advantages (size, power, Numerous advantages (size, power,

reliability, nonreliability, non--invasiveness)invasiveness)•• Design challenge: overcome tendency to Design challenge: overcome tendency to

damage red blood cellsdamage red blood cells•• Need macroscopic blood flow theory that Need macroscopic blood flow theory that

accounts for blood (cell) microstructureaccounts for blood (cell) microstructure

At microscopic (cell) scales:At microscopic (cell) scales:•• Macroscopic model fails in smallMacroscopic model fails in small--lengthlength--

scale regions (blade tip, rotor bearing)scale regions (blade tip, rotor bearing)•• Need modeling at cell scales to account Need modeling at cell scales to account

for blood damagefor blood damage•• Our Our mesoscopicmesoscopic simulations resolve simulations resolve

interaction of RBCs elastic membrane interaction of RBCs elastic membrane with plasma fluid dynamicswith plasma fluid dynamics

•• Prospects for 3D simulation of bladeProspects for 3D simulation of blade--tip tip region: 1 week at sustained 1 region: 1 week at sustained 1 petaflops/spetaflops/s

James James AntakiAntaki, Guy , Guy BlellochBlelloch, Omar , Omar GhattasGhattas, Marina , Marina KamenevaKameneva (Pitt), Robert (Pitt), Robert KormosKormos (Pitt), Gary Miller, (Pitt), Gary Miller, K. K. RajagopalRajagopal (Texas A&M), George (Texas A&M), George TurkiyyahTurkiyyah(Washington), Noel (Washington), Noel WalkingtonWalkington

Real time optimization for dynamic inversion & controlReal time optimization for dynamic inversion & control

Inversion and control for airborne contaminant transportInversion and control for airborne contaminant transport

Water network contaminant inversionWater network contaminant inversion•• Nonlinear optimization problem with Nonlinear optimization problem with

>300K variables and >100k controls >300K variables and >100k controls •• Solution time < 2 CPU minutes Solution time < 2 CPU minutes

real time source detection real time source detection •• Algorithm successful on thousands of Algorithm successful on thousands of

numerical tests on several municipal numerical tests on several municipal water networkswater networks

•• Formulation tool links to existing Formulation tool links to existing modeling software (EPANET) and modeling software (EPANET) and powerful NLP solver (IPOPT)powerful NLP solver (IPOPT)

VolkanVolkan AkcelikAkcelik, Roscoe Bartlett (, Roscoe Bartlett (SandiaSandia), Lorenz ), Lorenz BieglerBiegler, , George Biros (George Biros (UPennUPenn), Frank ), Frank FendellFendell (TRW), Omar (TRW), Omar GhattasGhattas, , Matthias Matthias HeinkenshlossHeinkenshloss (Rice), Judy Hill (CMU), David (Rice), Judy Hill (CMU), David Keyes (Columbia), John Keyes (Columbia), John ShadidShadid ((SandiaSandia), Bart van ), Bart van BloemenBloemenWaandersWaanders ((SandiaSandia), Andreas ), Andreas WachterWachter (IBM), David Young (IBM), David Young (Boeing)(Boeing)

•• sensor data provides sensor data provides concentrations of concentrations of hazardous agents hazardous agents

•• inverse problem solved inverse problem solved to reconstruct initial to reconstruct initial conditionsconditions

•• control problem solved control problem solved to find optimal to find optimal remediation strategyremediation strategy

Shape optimization of accelerator structures Volkan Akcelik (CMU) , Lori Freitag (LLNL), Omar Ghattas (CMU), David Keyes (Columbia), Patrick Knupp (SNL), Kwok Ko (SLAC), Lie-Quan (Rich) Lee (SLAC), Mark Shepherd (RPI), Tim Tautges(SNL)

CAD Meshing Partitioning(parallel)

h-Refinementp-refinement

Solvers(parallel)

Refinement

Basic Analysis Loop for given GeometryBasic Analysis Loop for given Geometry

Omega3POmega3P

S3PS3P

T3PT3P

Tau3PTau3P

•• Computer modeling has replaced trial and Computer modeling has replaced trial and error prototypingerror prototyping•• Next generation accelerators have complex Next generation accelerators have complex cavities that require shape optimization for cavities that require shape optimization for improved performance and reduced costimproved performance and reduced cost

•• Shape optimization problem governed by Shape optimization problem governed by electromagnetic electromagnetic eigenvalueeigenvalue problemproblem•• Cost functions involve target frequency, Cost functions involve target frequency, surface integrals of magnetic field, line surface integrals of magnetic field, line integrals of electric field integrals of electric field

medical medical imagingimaging

4D image 4D image registrationregistration

5D model 5D model inversioninversion

diagnosis & diagnosis & planningplanning

imaging imaging lab serverlab server

institutional institutional clustercluster

regional regional supercomputing supercomputing centercenter

physician physician desktopdesktop

ImageImage--based patientbased patient--specific inversionspecific inversion--based cardiac modelingbased cardiac modeling

VolkanVolkan AkcelikAkcelik (CMU) , George Biros (Penn), (CMU) , George Biros (Penn), AlfioAlfio BorziBorzi (Graz), (Graz), ChristosChristos DavatzikosDavatzikos (Penn), Omar (Penn), Omar GhattasGhattas (CMU), William (CMU), William GroppGropp (Argonne), Michael (Argonne), Michael HintermuellerHintermueller (Graz), (Graz), EldadEldad HaberHaber(Emory), David Keyes (Columbia), Jan (Emory), David Keyes (Columbia), Jan ModersitzkiModersitzki ((LubekLubek), ), Jennifer Jennifer SchopfSchopf (Argonne)(Argonne)

Detailed example #1: Real-time inversion for transport of contaminant

• Given local velocity field v (e.g. from mesoscopic weather model), diffusivity k, sensor observations u*, and a terrain model, estimate initial condition u0 of a convection-diffusion equation by solving regularized inverse problem:

• Forward problem can then be solved to predict transport of contaminant

Optimality system

Symbolic optimality system

Greater Los Angeles BasinOnshore flow

Transport over synthetic terrain: forward solution

• Initial concentration is a Gaussian

• Velocity field from laminar Navier-Stokes flow calculation with parabolic inflow

• Images show contours of plume at initial and after 30, 60, and 90 timesteps

• PETSc implementation

Transport over synthetic terrain: inverse solution

• Solution of inverse problem via a reduced Newton-CG solver, preconditioned by a 2-step stationary method

• Original initial condition vs. inverted initial condition for 6x6x6, 11x11x11, and 21x21x21 sensor arrays

• Solution obtained in 58 CG iterations, each requiring pair of forward/adjoint transport solves

Detailed example #2: Earthquake inversion

USGS

Variable-slip kinematicsource model

Earthquake Modeling for Seismic Hazard Assessment Earthquake Modeling for Seismic Hazard Assessment VolkanVolkan AkcelikAkcelik , , JacoboJacobo BielakBielak, George Biros , George Biros ((UPennUPenn), Steven Day (SDSU), Omar ), Steven Day (SDSU), Omar GhattasGhattas, , LoukasLoukas KallivokasKallivokas (Texas), Harold (Texas), Harold MagistraleMagistrale(SDSU), David (SDSU), David O’HallaronO’Hallaron

Region of interest for Region of interest for 1994 Northridge 1994 Northridge earthquake simulation earthquake simulation

Adaptive grid resolves up to 1Hz freq. Adaptive grid resolves up to 1Hz freq. w/100 million grid pts; uniform grid w/100 million grid pts; uniform grid would require 2000x more pointswould require 2000x more points

SCEC geological model provides 3D soil SCEC geological model provides 3D soil properties in Greater LA Basinproperties in Greater LA Basin

Snapshot of simulated ground motion Snapshot of simulated ground motion (simulation requires 3hr on 6Tflops PSC (simulation requires 3hr on 6Tflops PSC machine, running at >80% parallel machine, running at >80% parallel effeff))

Comparison of observation with Comparison of observation with simulation (improved prediction simulation (improved prediction requires requires petaflopspetaflops capability)capability)

Inversion of surface observations Inversion of surface observations for 17 million elastic parameters for 17 million elastic parameters (right: target; left: inversion result)(right: target; left: inversion result)

Inverse problem: Use records of past seismic events to improve velocity model

SCEC Phase III strong motion database: SCEC Phase III strong motion database:

Observations from 28 earthquakes and 281 stationsObservations from 28 earthquakes and 281 stations

Least squares parameter estimation formulation of inverse wave propagation

receiverssourcesdata misfit

inversion fields

displacements

forward wave propagationmodel

Multiscale inversion:Target vs. inverted isosurfaces, level 1

Multiscale inversion:Target vs. inverted isosurfaces, level 2

Multiscale inversion:Target vs. inverted isosurfaces, level 3

Multiscale inversion:Target vs. inverted isosurfaces, level 4

Multiscale inversion:Target vs. inverted isosurfaces, level 5

Multiscale inversion:Target vs. inverted isosurfaces, level 6

Multiscale inversion:Target vs. inverted isosurfaces, level 7

Multiscale inversion:Target vs. inverted isosurfaces, level 8

Multiscale inversion:Target vs. inverted isosurfaces, level 9

Comparison of target and inverted material models: 3D acoustic and elastic

Acoustic medium, pAcoustic medium, p--wave velocitywave velocity Elastic medium, sElastic medium, s--wave velocitywave velocity

Challenges/Needs

• optimization techniques for multiscalesimulation models

• optimization-ready reduced order models• large scale 4D data assimilation methods• real-time optimization algorithms • uncertainty quantification and propagation• simulation-based optimization algorithms

scalable to O(100,000) processors• latency tolerant algorithms for exploiting

distributed computing resources

SummarySummary

•• CI will help catalyze a transformation to highCI will help catalyze a transformation to high--fidelity simulationfidelity simulation--based decisionbased decision--makingmaking

•• But hardware/middleware infrastructure alone is But hardware/middleware infrastructure alone is insufficient to achieve this goalinsufficient to achieve this goal

•• Simultaneous advances on the models, methods, Simultaneous advances on the models, methods, and algorithms that underpin the components and algorithms that underpin the components ––and on their and on their systematic integrationsystematic integration to target to target strategic applications strategic applications –– are crucial for realizing are crucial for realizing the potential of CIthe potential of CI

Final editorial comment

• Advances in models and algorithms have often to led to greater improvements in simulation capability than have improvements in hardware

Example frommagnetohydrodynamics:2.5 orders of magnitude from hardware improvements; 3.5 orders of magnitude from modeling and algorithmic advances (From SCaLeSreport, Vol 2)