XPACC: The Center for Exascale Simulation of Plasma ......PCI Parallel Computing Institute CSE...

PCIParallel ComputingInstitute

CSE Computational Science & Engineering XPACC

Center for Exascale Simulation

of Plasma-Coupled Combustion

XPACC: The Center for ExascaleSimulation of Plasma-CoupledCombustion

Jonathan B. Freund

Parallel Computing Institute (PCI)Mechanical Science & EngineeringAerospace EngineeringUniversity of Illinois

Oxydizer

O - Ar; Air2

Fuel

H ; CH2 4

Coaxial

DBD

Plasma

Pin-to-Pin Spark Discharge

Boundary Layer

Turbulence

Pipe-!ow Jet

Turbulence

E-"elds

A new DOE/NNSA-funded ASC PSAAP MSC Center

1





DOE/NNSA/ASC/PSAAP Motivation

I NNSA — DOE National Nuclear Security Administration

I ASC — Office of Advanced Simulation and Computing

“Established in 1995, the ASC Program supports the U.S. De-fense Programs’ shift in emphasis from test-based confidence tosimulation-based confidence”

I PSAAP — Predictive Science Academic Alliance Program

• Multidisciplinary Simulation Centers (MSC) (3 $3.2M/year)• Single-discipline Centers (SDC) (3 $1.6M/year)

“We expect the PSAAP alliances will continue to help developthe predictive science field and the workforce of the future,wherein simulations will be pervasive and instrumental in im-portant, high-impact decision-making processes” [emphasis added]

— Robert Meisner, director of NNSA ASC

2





XPACC MSC Specific Motivation

I Truly predictive simulations will accelerate fundamentaladvances in the use of plasmas to improve combustion

I Computer Science advances that enable such massive-scalepredictive simulations will have broad impact

3





Specific Target Application

Oxydizer

O - Ar; Air2

Fuel

H ; CH2 4

Coaxial

DBD

Plasma

Pin-to-Pin Spark Discharge

Boundary Layer

Turbulence

Pipe-!ow Jet

Turbulence

E-"elds

I Predict the ignition threshold of a jet in crossflow viaspark-discharge and dielectric-barrier-discharge plasmas

I Iconical combustion flow with new ‘knobs’ for mediatingignition

4





Coupled-Physics Predictive Science Challenge

I Plasmas introduce chemical-reaction-coupling radicals

I Plasma heating triggers chemistry

I Local heating alter flow/turbulence

I Electric fields alter diffusion of charged species

I Electric fields exert body forces, destabilizing flames

I Electrodes age and alter plasma generation efficacy

I Turbulence mixes reactants

I Flows alters plasma structure

I · · ·

5





A Multi-Scale Predictive Science Challenge

I LENGTH:

• nm-scale◦ electrode surface physics◦ mean-free-path reactions

• µm-scale◦ plasma structures◦ flame thickness

• mm-scale ◦ turbulence• m-scale ◦ device

I TIME:

• ns-scale◦ plasma breakdown◦ ion impacts

• µs-scale ◦ reaction kinetics• ms-scale ◦ mixing rates• s-to-∞ ◦ electrode aging

Length Scales

Time Scales

Electrode

Plasma

Flow/Flame

~ m - device

~ mm - turbulence

~ μm - flame thickness

~ μm - plasma sheaths

~ nm - chemical reactions

~ nm - electrode surface

~ s - electrode aging

~ ms - turbulence mixing

~ μs - plasma/reaction kinetics

~ ns - plasma breakdown

6





A Multi-Character Predictive Science Challenge

I Physics/math character ⇒ algorithms ⇒ resource utilization

I Hyperbolic

• Convection: Turbulent mixing

I Elliptic

• Diffusion: Small-scale,molecular mixing

• Electric field• Radiation

I Atomistic/Particle

• Electrode damagemechanisms

• Particle plasma models

I Quantum

• Electrode work functions

Electrode

Plasma

Flow/Flame

Diffusion

Atomistic

Quantum

Diffusion

Atomistic

E & M

Convection

Diffusion

Reaction

Convection

Radiation

7





Extreme-Scale Computing Synopsis

I NOW

• Petascale resources: 1015 operations/sec• Homogeneous: nodes and network• Power hungry

I FORECAST (5+ years)

• Exascale resources: 1018 operations/sec• Ever greater concurrency• Heterogeneity

∗ more varied specialized processing elements∗ CPUs + GPU-like accelerators + · · ·∗ limited local memory

• Limited full-system bandwidth• More components: more potential faults• Input/Output (I/O): costly-to-impractical• Power management essential

8





Use of Exascale (×103 Petascale)

I Vertical: 103 × more physics

• more scales• increasingly fundamental models• ⇒ robustness for extrapolation to true prediction

I Lateral: 103 × more simulations

• establish input–output statistics:model parameters → predictions

• quantified uncertainty adds predictive value

I Both lead to better Quantity of Interest (QoI) prediction

9





XPACC: An Exascale-Simulation Problem

I Models can couple reliably only if firmly grounded in physics

• calibrated sub-models do not guarantee accurate integration• represent coupled-physics in detail for low-uncertainty

prediction

I Representing physics means, in part, representing length andtime scales

• integrating physics broadens the scale range• not computationally additive: depends upon min/max of

represented scales

I For low uncertainty, integrating petascale sub-physics modelswill require exascale

I We will build toward exascale using reduced models, expectingsignificant-but-quantified initial uncertainty

10





Exascale-Mapping Predictive Science Challenge

I Different physical models ⇒ different discretization ⇒different implementation challenges

I Numerics and implementation design to minimize overhead ofmapping onto forthcoming, heterogeneous exascale resources

{

{

Existing CS

MPI, OpenMP

conventional I/O

Needed CS

load balancing

fault tolerance

I/O management

data localization

heterogeneity tools

power awareness

Petascale System

(homogeneous)

Exascale System

(heterogeneous)

Length Scales

Time Scales

Electrode

Plasma

Flow/Flame

Diffusion

Atomistic

Quantum

Diffusion

Atomistic

E & M

Convection

Diffusion

Reaction

Convection

Radiation

GLOBAL

LOCAL

VECTOR

SCALAR

CPU Suited

GPU Suited

Custom

hardware

~ m - device

~ mm - turbulence

~ μm - flame thickness

~ μm - plasma sheaths

~ nm - chemical reactions

~ nm - electrode surface

~ s - electrode aging

~ ms - turbulence mixing

~ μs - plasma/reaction kinetics

~ ns - plasma breakdown

I General, portable CS advances are necessary to complete thismapping

11





Exascale: Criteria of Success

I 5-years outlook: trans-petascale/pre-exascale

I Metrics of success toward exascale (and beyond):

• scale well to levels of concurrency expected in exascale systems• scale well in the presence of performance irregularity in

computing elements and network• perform well, including operations per Joule, with more

specialized computing elements and less hardware support• produce correct results even in the presence of faults• tools must not hinder physics modeling development and

simulation

12





The Team

13





The Team

Principal Investigator/CS Lead

W. Gropp (CS/PCI)

Application Lead

J. Freund (MechSE/AE)

Computer Science

W. Hwu (ECE/PCI)

S. Kale (CS)

D. Padua (CS)

[A. Kloeckner (CS)]

Application Simulations

H. Johnson (MechSE)

C. Pantano (MechSE)

[M. Panesi (AE)]

Experimental Lead

G. Elliott (AE)

Application Experiments

I. Adamovich (MAE at OSU)

N. Glumac (MechSE)

W. Lempert (Chem, MAE at OSU)

External Advisory Board

Campbell Carter (AFRL)

Andrew Chien (UChicago)

Jack Dongarra (UTenn/ORNL)

Max Gunzburger (FSU)

Bob Moser (UT Austin)

Elaine Oran (ONRL)

Chief Software Architect

D. Bodony (AE)

Hardware-Algorithms Integration

Lead: L. Olson (CS)

Executive Committee

14





Specific Configuration

Oxidizer

O - Ar; Air2

Fuel

H ; CH2 4Coaxial DBD Plasma

• Combustion radicals

• Minor heating

Pin-to-Pin Spark Discharge:

• Combustion radicals

• Localized intense heating

Boundary Layer

Turbulence

Pipe-!ow Jet

Turbulence

E-"elds:

• Body forces

• Biased di#usion (in !ames)

I Fuel jet (5 mm; H2, CH4) in cross-flow (50 m/s; O2-Ar, air)

• A canonical configuration• Represents broad class of application configurations

I Plasmas near exit mediate ignition/combustion

• Spark discharge: radicals and thermal• Dielectric barrier discharge: primarily non-thermal via kinetics

I Operate in coupled-physics regime:plasma essential for ignition

15





Plasma Regime

I Collisional

I Non-thermal: dielectric barrier andnanosecond pulse discharges

I Weakly-thermal/thermal: microsecond pulsedischarges

I Temperatures: Tn/i ≈ 300− 3000 K;∼ 1 to 10 eV electrons

I Pressure:

• low (∼ 50 Torr) in some calibrationexperiments

• higher (1 atm) in target application(practical, rich phenomenology)

I Non-magnetic: Larmor radius � mean freepath (rg = v⊥/ωg � λ)

• Electrostatic: Poisson equation∇ · (ε∇)φ = −e(Zini − ne)

Coaxial DBD Plasma • Combustion radicals

• Minor heating

Pin-to-Pin Spark Discharge: • Combustion radicals

• Localized intense heating

Boundary Layer

Turbulence

Pipe-!ow Jet

Turbulence

E-"elds: • Body forces

• Biased di#usion (in !ames)

16





Specific Prediction

Bu

rnN

o B

urn

Burn/No-Burn

Boundary

Parameter Uncertainty

PDFs from physics-targeted

simple-geometry MC assessment

rnN

o B

uN

“Find” Burn/No-Burn Boundary

Identify Key Uncertainties

Adjoint: Gradient

I Predict ignition with uncertainty estimate for

• Flow conditions (jet and crossflow)• Spark operation (kV, duration, rate)• DBD operation (kV, rate)

17





Current Status

}Compressible flow

LES: Smagorinsky SGS

Hard-coded reduced H - air chemistry

Thermomechanically coupled “wall”

Crude heat/radical sources

2START

18





Current Flow Model

(Bodony, Freund)

I Compressible Navier–Stokes

I Reactive mixtures of thermally perfect gases

I Temperature-dependent specific heats

∂

∂t

(Q

J

)+∂F

∂ξ+∂G

∂η+∂H

∂ζ=

S

J

I Solution Q; fluxes {F ,G ,H}; mapping Jacobian JI S source:

• sub-grid-scale models (large-eddy simulation, ...)• chemical/plasma reactions (...)• body forces on charged species (...)• radiation ‘sink’ (...)

I Coupled with thermo-mechanical (electrode) model

I Validated in hypersonic aerodynamics applications

19





Year 0 Demonstration Simulation

Air free-stream Mach number M∞ 0.5H2 jet Mach number Mj 0.5Jet diameter d 5 mmBoundary layer δ99 5 mmReynolds number Re = ρ∞U∞d/µ∞ 46,000

air

H2

Mesh 1: bound. layer 501× 106× 401Mesh 2: r ≈ 0 81× 306× 81Mesh 3: cylindrical 161× 91× 306

No plasma, reaction, electrodes, E -fields, ...

20





Model Development & Integration(Pantano, Adamovich, Bodony, Freund, Johnson, Panesi)

OSU Plasma

Combustion Kinetics

DD-LTE-LEE Plasma

DT Plasma

DS Plasma ( f )

PMC Plasma

Calibrated

2-Reaction Model

SGS Plasma

Transport Models

SGS Combustion/

Mixing ModelsCalibrated Phenomenological

Electrode Aging Model

Physics-based

Electrode Aging Model

DFT Work

Functions Model Refinement

(and De-refinement)

to Target QoI

Uncertainty

Y0}

Compressible flow

LES: Smagorinsky SGS

Hard-coded reduced H - air chemistry

Thermomechanically coupled “wall”

Crude heat/radical sources

2

Y1: QoI

Y2: QoI

Y3: QoI

Y4: QoI

Y5: QoI

START

21





Spark Discharge Electrodes: Aging

I Spark characteristics depends upon surface details of electrode

I Electrodes age:

New 30 Minutes 1 hour

I True predictions will require aging models

I Ion impacts alter mechanical and electronic properties of theelectrode surface

• Roughness focuses electric field, altering ions impacts• Damage alters electron work functions, affecting efficacy

22





Physics-Based Electrode Aging Model(Johnson, Freund)

I Generalize our time-scale bridging crater-function model

• MD ion bombardment• Statistical response used in continuum model• Reproduced ripple-like surface patterns on Si

MD: Ion Bombardment

+Slow

Surface

Evolution

Electrostatics/

E-!eldion trajectories,

surface shape

Density Functional

Theory

Work Functions

unit process/

crater function

E-!eld coupled

Electronic/First-principles Atomic/Mechanical Continuum

INPUT: Electrode material, geometry, voltage, current

OUTPUT: Plasma condition over long damage timescales

damage

morphology

23





Physics-Based Electrode Aging Model(Johnson, Freund)

I Add:

• Electrostatic alteration oftrajectories (build on AFM tipsharpening effort)

• DFT calculations of electron workfunctions

∗ used extensively to quantifyelectronic properties of defectsand nanostructures:Johnson et al.

I Close interaction with experiments

• Physics-targeted experiments onelectrode aging and its effects

• Diagnostics of electrode surfaces(e.g. MRL facilities)

I Build into ‘boundary condition’ forapplication simulation

24





Uncertainty Quantification (UQ): StrategyI Quantified uncertainty increases predictive value

I Quality data and two-way interaction with experiments

• local experiments designed and refined for UQ objective• low-cost, physics-targeted• expert quantitative assessment of measurement uncertainties

I Thorough calibration

• Bayesian inference and sampling• fast, low-dimensional, physics-targeted simulations• develop and employ model-inadequacy models• calibrate models for aleatoric uncertainty (spark discharges)

I Dimensionality reduction

• sensitivity: remove unimportant uncertain parameter• speed: select fidelity for overall QoI uncertainty goals• speed: seek surrogates that respect burn/no-burn boundary

I True QoI prediction

• with quantified uncertainty• compared predictively against experiment

25





UQ: Two-Stage Approach

I Simple configuration, physics-targeted calibration experiments

• published experimental results• designed to emphasize particular physical interactions• designed for specific modeling/simulation needs

I Uncertainty propagation to full-system predictions

26





Physics-Targeted Calibration Experiments(Elliott, Adamovich, Glumac, Lempert)

Quasi-0D

Plasma-Induced IgnitionUse: plasma models, plasma-coupled ignition ki-netics Status: diagnostic capabilities and flowchamber up and running at OSU (Adamovich,Lempert)

27






Quasi-1D

Plasma–PremixedUse: plasma ignition validation, quantificationof electrode aging and effect, aleatoric modelingof sparks Status: inert analog up and runningat Illinois (Elliott, Glumac)

Counterflow Diffusion Flame

+

gnd

Use: kinetics models and effects of aug-mented radical transport; Status: stagnationflow flat-flame variation up and running at OSU(Adamovich, Lempert)

28






Quasi-2D

Plasma–Inert FluidUse: plasma–flow coupling, quantification ofelectrode aging and effect; aleatoric model ofsparks Status: zero-flow version up and runningat Illinois (Elliott, Glumac)

Plasma-Induced Ignition & PropagationUse: flame propagation predictions; Status: di-agnostic capabilities and flow chamber up andrunning at OSU (Adamovich, Lempert)

29





Application / Validation Experiments(Elliott, Adamovich, Glumac, Lempert)

Fully 3D

Plasma-coupled Burning Turbulent JetUse: validation of ignition threshold predic-tion (and other predictions) Status: tunnel andnecessary diagnostics available at Illinois (El-liott,Glumac)

30





Overall Roadmap

Year 3 Year 4Year 2Year 1 Year 5

+

gnd

Prediction UQCalibration UQ

Increasing Sampling

Single Simulation As Physics Models/UQ Necessitates

Ignition (~0D)

Flame (~2D)

Flame kinetics

Ignition kinetics

Spark-flow coupling

Electrode aging

E-field forces/diffusion

Flame propagation/ stability

Complex geometry/turbulence

~0D

~1D

~2D

3D

Data tiling

Over-decompositio

n

GGCG

G G

GM

GPU

accelera

tion

Source-to-source

Load balancing

?

??

??

?

Model Fidelity

GeometricComplexity

Terascale

Petascale

Exascale

}}}

31





UQ: Calibration

(Freund, Panesi)

I Bayesian inference

• low-dimensional, physics-targeted experiments• fast Monte Carlo parameter calibration simulations

I Priors: select to reflect expectations without restricting range

I Include model-inadequacy model calibration(e.g. Kennedy & O’Hagan, 2001; Moser et al. 2012)

32





QoI Uncertainty

(Freund, Panesi)

I QoI: ignition threshold

• find using adjoint-based gradient

I Propagate ~p and ~σ uncertainty toapplication predictions

I Adjoint-based sensitivity estimateof impact of uncertain parameterson QoI

σα∂J

∂pα• Eliminate unimportant

parametric uncertainties• Reduce dimensionality

Bu

rnN

o B

urn

Burn/No-Burn

Boundary

Parameter Uncertainty

PDFs from physics-targeted

simple-geometry MC assessment

rnN

o B

uN

“Find” Burn/No-Burn Boundary

Identify Key Uncertainties

Adjoint: Gradient

I QoI uncertainty: quadrature in uncertainty space

• start: sparse, adaptive, MC• quantify dependencies and seek surrogates, GP, ...

33





Adjoint-Based Sensitivity

(Freund, Bodony, Padua)

I Building on experience withadjoint-based control of unsteady,compressible, free shear flows for noisecontrol (AFOSR)

• Freund et al. (2003,...,2011)• Freund & Bodony et al. (2009,...)

to ControlSensitivity

Flow Solution

Noise

SolutionAdjoint

Time

Aco

ust

ic P

ow

er

0 10 20 30 40 50 60 70 80 902

4

6

8

10

12

Before ControlWith Control

Mach 1.3, Turbulent Jet:

10− 11 10− 10 10− 9 10− 8 10− 7 10− 6 10− 5 10− 4 10− 3 10− 210− 5

10− 4

10− 3

10− 2

10− 1

100

101

Discretized AnalyticFully Discrete

Distance in Control Space

Err

or:

Ad

join

t m

inu

s F.

D.

Numerical Precision Failure

(con!rmed via -real16)

I Developed readily coded exact discreteadjoint for flow-like transport

I Considering a compiler-based approachfor general discrete adjoint (Padua)

34





Example: XPACC in 1D(thanks David Buchta)

I Goal: predict ignition (temperature) versus plasma strength

• Radical addition (R)• Plasma heating (T )

I Physics-targeted experiments → a higher-fidelity model

• Reduced H2–air mechanism of Boivin et al. (2011)• Based on more detailed 21-reaction, 8-species description

35










2 2.2 2.4 2.6 2.8 3

-9

-8

-7

-6

-5

-4

-3

3000.0

2472.4

2037.5

1679.2

1383.9

1140.5

939.9

774.6

638.4

526.1

433.6

357.3

294.5

242.7

200.0

3000K

1000K

200K

500K

2000K

35










I Calibrate simple model:

I : F + O → P + R

II : R + M → F + M

kI = AITnI e−T/T a

I kII = AIITnII e−T/T a

II

35





Example: XPACC in 1D

yk — T (t = 125 µs) 2-reaction trial predictionsdk — T (t = 125 µs) Boivin et al. ‘data’T i

k — initial temperature for datapoint kR i

k — initial radical concentration (atomic-H in Boivin model)k — samples: k = 1, ...,N = 25 in log R i–log T i plane

I Likelihood:

P({yk − dk}|θ, {Rk ,Tk}, I ) =1

σ√

2πexp

[−∑

(yk − dk)2

2σ2

]I Calibrate θ = {AI ,AII , nI , nII ,T

aI ,T

aII , σ} using Bayesian

inference via MCMC

36





Calibration

37





Prediction

2 2.2 2.4 2.6 2.8 3

-9

-8

-7

-6

-5

-4

-3

3000.0

2472.4

2037.5

1679.2

1383.9

1140.5

939.9

774.6

638.4

526.1

433.6

357.3

294.5

242.7

200.0

3000K

1000K

200K

500K

2000K

0 1000 2000 30000

0.002

0.004

0.006

0.008

0.01

0.012

38





Prediction

2 2.2 2.4 2.6 2.8 3

-9

-8

-7

-6

-5

-4

-3

3000.0

2472.4

2037.5

1679.2

1383.9

1140.5

939.9

774.6

638.4

526.1

433.6

357.3

294.5

242.7

200.0

3000K

1000K

200K

500K

2000K

0 1000 2000 30000

0.002

0.004

0.006

0.008

0.01

0.012

I Challenge at burn/no-burn boundaryI Inadequate model-inadequacy model?

• single σ value• biased toward many ‘easy’ predictions away from threshold

I Calibration on joint-PDF (not marginals) essential

• Marginal PDF yields erroneous, broad, bi-model T -predictiondue to correlations (e.g. nI –TI )

38





Overall Approach Summary

I Data with any associated uncertainty

I Candidate physical model

I Inadequacy model

I Calibrate parameters of bothI Identify important uncertainties for QoI

• E.g. 0-D calibration → 1-D prediction sensitivities:

Ti = 447 K Ri = 10−3 σinad. = 48.2

σnI

∂J

∂nI= 20.1 σTI

∂J

∂TI= −33.7 σAI

∂J

∂AI= 6.8

σnII

∂J

∂nII= 11.1 σTII

∂J

∂TII= −16.8 σAII

∂J

∂AII= 0.8

I Propagate important uncertainty to QoI

I Validation: QoI predicted within combined predicted and dataQoI data uncertainty

39





Summary

I Start date.... Jan 1, 2014...

maybe... hopefully

• looking for personnel

I XPACC is simulation based....predictive science enabled by exascale

• bigger simulations: physics faithful across scales• more simulations: rigorously integrated for UQ• ∼ half effort on CS research for utilization of exascale

I Experiments are essential

• within XPACC: for calibration and QoI validation• beyond XPACC: foundational studies will continue to guide

design and integration of models (e.g. AFOSR PAC MURI!)

I XPACC aims to impact combustion beyond its principalPSAAP Predictive-Science goals

40





Summary

I Start date.... Jan 1, 2014... maybe...

hopefully








40





Summary

I Start date.... Jan 1, 2014... maybe... hopefully








40

XPACC: The Center for Exascale Simulation of Plasma ......PCI Parallel Computing Institute CSE...

Documents

Transcript of XPACC: The Center for Exascale Simulation of Plasma ......PCI Parallel Computing Institute CSE...