KarthikDuraisamy

StructuredAutoencoders forOperator-theoreticdecompositionandModelreduction

Thanksto…

Motivation

Decomposition &ReducedOrderModelingofComplexMultiscaleProblems

[K] [W/m3]

LargescalesimulationsO(106)- O(108)CPUhours/runComplexphysics:Flow,turbulence,combustion,heattransfer,etc

TheAutoencoder

4

Structure:encoder(compression)+decoder(decompression)– Encoder𝚽 𝐱; 𝛉𝚽– Decoder𝚿 . ;𝛉𝚿– POD:𝚽→𝐔) · ,𝚿→𝐔 ·– Trainedasonesinglenetwork,𝛉𝚽 and𝛉𝚿 areoptimizedjointly– Automaticallyseparatedintoencoderanddecoderbycuttingatthe“bottleneck”

“Applied Deep Learning”https://towardsdatascience.com/

𝚽 𝚿

x̃ = (·, ✓ ) � �(x, ✓�)x

Embedding(intherightcoordinates)

Part1

Operator-theoreticLearning&Decomposition

Koopman operatorandlinearembedding

ConnectionsofKoopman tootheroperators

L := f ·rx

@u

@t= Lu ; u(·, 0) = h

Kt

h = h � �t

= etLh

Liouville operator

Liouville PDE

Generator

hh,Pt⇢i = hKth, ⇢i

⇢(·, t) = Pt � ⇢Perron-Frobenius operator

Duality

Liouville generatesKoopman

Perron-Frobenius isadjoint ofKoopman

SpectralexpansionofKoopman operators

Koopman operators&“Deep”Learning

Severalworkssince2018

Extracting a Koopman-invariant subspace

Arbabi &Mezic 2019

Goal:ExtractingtheKoopman operatordefinedon

Observationfunctionals

Wearealsointerestedinretrievingthestatex

EnforcingstructureforLearning:“Physicsinformation”

Enforcing structure for Learning : Tractable optimization

“Data-free”, “Physics-informed”

Trajectory data, ”Unknown physics”

EnforcingstructureforLearning:“DMDResNet”

EnforcingstructureforLearning:Stability

x

f(x)

Naïve“Autoencoders”

x

f(x)

Pan. S. & Duraisamy, K., Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability, SIAM J. of Applied Dynamical Systems, 2020.

Puttingitalltogether(deterministicform)

BayesianNeuralNetworks&Variational Inference

Variational Inference

VerificationonModeldynamicalsystemsDuffingoscillator:Eigenfunctions (withuncertainty)

Predictionandsensitivitytodata

100datapoints.1000datapoints10000datapoints

Flowovercylinder:Predictionwithuncertainties

•Gaussian white noise added

Flowovercylinder:Predictionwithuncertainties

Velocitymagnitude(mean)

Velocitymagnitude(standarddeviation)

Physics-InformedProbabilisticLearningofLinearEmbeddings ofNon-linearDynamicsWithGuaranteedStability,Pan,S.,andDuraisamy,K.,SIADS,2020

Proposed framework

Steps:

Ia-priori cross validation to

choose an appropriate

hyperparameter

Imode-by-mode error analysis

Ichoose a trade-o↵ between

reconstruction error and

linear evolving error

Isparse reconstruction of

system with multi-task

learning

4 / 9

Multi-tasklearningframeworktoextractsparseKoopman-invariantsubspaces

Sparsity-promoting algorithms for the discovery of informative Koopman invariant subspaces, Pan, S., N. A-M and Duraisamy, K., arXiv:2002.10637

Turbulent: 3D ship-airwake with impulse side-wind

Itransient behavior is

accurately reconstructed

Istable modes are successfully

extracted from strongly

nonlinear transient data

Ileft mode: due to side edge

of superstructure. right

mode: due to funnel

6 / 9

TurbulentShipAirwake

Sparsity-promoting algorithms for the discovery of informative Koopman invariant subspaces, Pan, S., N. A-M and Duraisamy, K., arXiv:2002.10637

26

Summary

ManyopportunitiestoenforcestructureinAutoencodersè flexibleandpowerfultools

Part 2

LearningReducedOrderModelsofParametricSpatio-temporaldynamics

• B.Kramer,K.E.Willcox,AIAAJournal,2019

• M.Guo,J.S.Hesthaven,CMAME,2018.

• A.Mohan,D.Daniel,M.Chertkov,D.Livescu,arXiv,2019

• S.Lee,D.You,arXiv,2019.

• Q.Wang,J.S.Hesthaven,D.Ray,JCP,2019.

Non-intrusivedata-drivenROMs

qn+1l = f(qn+1

l ,qnl , ....q

n�ll , B(un+1), µ)

Somerecentworks:

BasicComponent:ConvolutionalLayer

29

• Convolutionallayerspreservecomplexspatio-temporal“information”

• Convolutionaloperationonalocalwindoww– 𝑥 ∗ 𝑤 ./ = ∑ ∑ 𝑥.23,/25𝑤3,526

57628378

• Idealfor“localized”featureidentification• Rotationandtranslationinvariant,if

properlyconstructed

http://cs231n.github.io/convolutional-networks/

: “Applied Deep Learning”https://towardsdatascience.com/

TemporalConvolutional• Performsdilated1Dconvolutionaloperationintemporal/sequentialdirection

– 𝑥 ∗9 𝑤 . = ∑ 𝑥.293𝑤3:2;37<

• Exponentialincreaseinreceptionfieldè anincreasinglypopularalternativetoRNN/LSTM

Inputsequence

Conv-1:d=1receptionfield:2

Conv-2:d=2receptionfield:4

Conv-3:d=4receptionfield:8

Output/Conv-4:d=8receptionfield:16

Example:k=2,d=2i

Sourceofimage:github.com/philipperemy/keras-tcn

TrainingMulti-levelconvolutionalAEnetworks

ExampleCAEarchitecture

32

PredictionusingMultilevelAEnetworks

ExampleTCAEarchitecture

PredictionusingMultilevelAEnetworks

Timestepping

37

Example TCN architecture

*: Convolution direction

0 1 0 1 0 1 0

1 0 0 1 0 0 1

Input

𝐐>.,?

Jumpingdilated 1D

convolution

Dense𝐪>.A;

Jumpingdilated 1D

convolution

Jumpingdilated 1D

convolution

𝑛?steps

𝑛> channels

*

Many-to-one

Input

𝐐>.,?

Non-strideddilated 1D

convolution

1D convolution𝐪>.A;

Non-strideddilated 1D

convolution

𝑛? steps

Many-to-many

𝑛> input channels

*

0 0 0 1 1 1 1

0 0 0 0 1 1 1

0 0 0 0 0 1 1

0 0 0 0 0 0 1

*: Convolution direction

Example TCN architecture

38

ComponentCAEreconstruction CAE+TCN

(finalstep)Training Testing

Pressure 0.04% 0.1% 0.12%

Density 0.01% 0.04% 0.14%

Velocity 0.04% 0.08% 0.13%

NumericalTests:Discontinuouscompressibleflow

39

Discontinuouscompressibleflow:Impactofdata

40

NumericalTests:3DShipAirwake

– IncompressibleNavier-Stokes– 576kDOF,400timesnapshots– Globalparameter:slidingangle𝛼– Training:𝛼 =5°:5°:20°– Prediction:𝛼 =12.5

𝛼 =5° 𝛼 =20°

41

NumericalTest:3DShipAirwake

ComponentCAEreconstruction

MLP+TCAE+CAETraining Testing

U 0.12% 0.30% 0.51%

V 0.09% 0.38% 0.89%

W 0.08% 0.29% 0.62%

Relativeabsoluteerror

Truth

Prediction

PredictionvsTruth

Latentvariable

Manuscript“Multi-levelConvolutionalAutoencoderNetworksforParametricPredictionofSpatio-temporalDynamics,”SubmittedCMAME

• FullyData-drivenframework

• Multi-levelneuralnetworkarchitecture• Convolutionsinspace&time

• Non-linearmanifolds• Fasttraining,fasterprediction

• Upto6ordersofreductioninDoF• Totaltrainingtime:3.6hoursononeNVIDIATeslaP100GPUfor3Dshipairwake• Predictiontime:Secondsforanewparameterorhundredsoffuturesteps

Caveats• Requirelargeamountsofdata

• Noindicatorforchoiceoflatentdimensionsè usesingularvaluestofindanupperbound

42Manuscript“Multi-levelConvolutionalAutoencoderNetworksforParametricPredictionofSpatio-temporalDynamics”tobesubmittedtoArXiv inaweek

Summary

• DARPAPhysicsofAIprogram(TechnicalMonitor:Dr.TedSenator)• AirForceCenterofExcellencegrant(ProgramManagers:Dr.Mitat Birkan andDr.

Fariba Fahroo)• OfficeofNavalResearch(Programmanager:Dr.BrianHolm-Hansen)• Computationalinfrastructure:NSF-MRI(Programmanager:Dr.StefanRobila)

43

Acknowledgments

44

𝛼 =5°

NumericalTests