Uncertainty Quantification: What is it and Why it is ... · Introduction to Uncertainty...
40
Uncertainty Quantification: What is it and Why it is Important to Test, Evaluation, and Modeling and Simulation in Defense and Aerospace Peter Qian, Professor in Statistics, University of Wisconsin-Madison, Chief Scientist, SmartUQ Dr. Mark Andrews, UQ Technology Steward, SmartUQ
Transcript of Uncertainty Quantification: What is it and Why it is ... · Introduction to Uncertainty...
Uncertainty Quantification: What is it and Why it is Important to Test, Evaluation,
and Modeling and Simulation in Defense and Aerospace
Peter Qian, Professor in Statistics, University of Wisconsin-Madison, Chief Scientist, SmartUQ
Dr. Mark Andrews, UQ Technology Steward, SmartUQ
Introduction to Uncertainty Quantification (UQ)– Historical Perspective
– Definitions, Terminology and Value of UQ
Industry Challenges
Overview of Methods
Current UQ Technical Bottlenecks
Selected UQ Methods • DOE for Multiple Data Sources
• UQ for Multi-Source Simulations
• Grounding Models in Reality using Statistical Calibration
Outline
2
Before CAE CAE
Historical Overview
However, CAE has new rules . . . have I built the model right? . . . (model verification)have I built the right model?. . . (model validation)have I accounted for real-life uncertainties?
Design verification by virtual testing of computer models
Design verification by prototype testingHands-on learning, tribal knowledge
3
UQ is Rapidly Growing Field With Broad Applications in Aerospace and Defense
62
2789
5667
12572
26095
32634
1959-OLDER 1960-1979 1980-1990 1991-2000 2001-2010 2011-2016
Nu
mb
er
of
Pu
bli
ca
tio
ns
Year Range
Science Direct Search for Publications with "Uncertainty" in Title, Abstract or Keyword
SIAM/ASA Journal on Uncertainty
Quantification (2013)
ASME Journal of Verification,
Validation and Uncertainty
Quantification (2016)
NAFEMS Stochastic
Working Group (2000)
SAE G-11 Probabilistic
Methods Committee (1998)
AIAA Non-Deterministic Approaches
Conference Technical Committee (1999)
4
Materials: Uncertainties in Powder Metallurgy process
Environment and Product Operators
As Designed Strength
Manufacturing: Uncertainties in heat treatment
Systems Integration: Component failure modes
StrengthDistribution
Uncertainty Quantification (UQ) is the science of identifying, characterizing and managing those factors in the analyses of complex systems, physical tests or simulations that impact the accuracy of the results
UQ puts the ‘error bars’ on the simulation resultsMore generally, UQ can put ‘error bars’ on nearly any data or system
5
UQ is a Probabilistic Approach
Load Strength
Deterministic
Strength/Load Design FactorAcceptable?
Deterministic (Design Factor) Approach
• Simple, efficient & easy to understand
• Design Factors do not do not quantify the risk of failure or specify how close the value is to being unacceptable
• Deterministic Approach typically do not include knowing parameter sensitivities
MPa
Freq
uen
cy
Load Strength
Probability of FailureAcceptable?
ProbabilisticProbabilistic Approach
• Iteration of the Deterministic Approach
• Quantifies the risk of failure and specifies how close the value is to being unacceptable
• Provides parameter sensitivity analysis that tells you which model parameters have the greatest influence the simulation response
MPa
Freq
uen
cy
6
There are Many Sources of Uncertainty
Uncertain System Inputs
• Any system input including initial conditions, boundary conditions, and transient forcing functions
Model Form and Parameter Uncertainty
• The inherent inaccuracy of the model and all assumptions made about the modeling parameters
Computational and Numerical Uncertainty
• The uncertainties resulting from the simplifications and approximations necessary to solve the system models.
Uncertainty in Physical Testing
• The uncertainty resulting from experimental error, physical variation, measurement uncertainty, and test set up.
For more information see: https://www.smartuq.com/resources/uncertainty-quantification/
7
Uncertain System Inputs
• Unknown Parameter Values: • Hard to measure system input parameters with fixed, but unknown, values.
• These are often an epistemic uncertainty.
• Examples include: as built material properties, operating environment, and actual loading profiles.
• Variable Parameter Values: • System inputs which are not fixed and may be inherently uncertain.
• Often subject to a known, or at least measurable, probability distribution.
• Examples include: Manufactured part geometry, fuel composition, and exact time of ignition.
0
2
4
6
8
10
0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7 3 3.3 3.6 3.9 4.2 4.5 4.8
An
nu
al C
ycle
s (
x10
00
)
Operating Load
Operator Variability
Use Case
In this graph: The eventual operator is an unknown parameter while the use profile for each operator is a variable parameter.
8
Model Form Uncertainty
• All models are approximations of reality.
• Modeling uncertainty is the result of errors, assumptions, and approximations made when choosing the model.
• Can be broken into: • Model form uncertainty, i.e. uncertainty about the models ability to capture
the relevant system behaviors.
• Parameter uncertainty, i.e. uncertainty about parameters within the model.
Model Form and Parameter Uncertainty
-0.2
-0.1
0
0.1
0.2
0.3
0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Dis
crep
ency
Load
Discrepency/Bias Function
Modeling Accuracy• Handling these types of uncertainty:
• Requires ‘true’ data from the system being modeled.
• Model form uncertainty is best treated with statistical calibration techniques capable of building a discrepancy model.
• Parameter uncertainty may be handled with other unknown or variable system inputs.
9
Computational and Numerical Uncertainty
• Most numerical models including FEA, CFD, and iteratively solved 1D models require simplification or approximation to solve.
• Thus the mathematical description might be perfect but factors such as truncation and convergence error may still introduce uncertainty.
• This will vary between different numerical solvers and different solver settings.
• Handling these types of uncertainty:
• Continuous solver settings may be treated as additional system input variables.
• i.e. the solver tolerance may be modelled as a continuous input.
• Solver type or categorical solver settings may be modelled as a discrete system inputs.
• i.e. similar combustion models may be solved in ANSYS or Converge and different levels of combustion chemistry fidelity may be used in either.
Computational and Numerical Error
10
Uncertainty in Physical Testing
• Arises from uncontrolled or unknown inputs, measurement errors, aleatoric phenomena, and limitations in the design and implementation of tests.
• Results in noisy experimental data.Physical Testing Uncertainty
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0 0.5 1 1.5 2 2.5 3Ti
me
to F
ailu
rePart Load
Time to Failure
Materials and Manufacturing
• Handling this type of uncertainty:• Inverse analysis can be used to
determine the input probability distributions.
• Distributions or empirical measurements can be handled as additional variable system input variables.
11
UQ Enables a One-Time Process for Product Development
• Reduce Costs Driven by Variability
– Prevent unnecessary design iterations
– Shorter development times; fewer tests / prototypes
• Maximize Product Reliability and Durability
– Reduce part-to-part variability increases product life
– Fixing problems in design is cheaper than in the field
• Ensure Simulation Results are Credible and Realistic
– Critical for model validation and what-if scenarios
– Essential for understanding risks for decision making
Build
Test
Design
Design Build Test
12
Major DOD Initiative for Model Verification and Validation
Director of Operational Test and Evaluation Office of the Secretary of Defense, Michael Gilmore
The V&V [Verification and Validation] allows engineers to make fewer model runs and have credible information to support future modeling and simulation evaluations
“Our intent is to make this [Uncertainty Quantification] the default approach that ARDEC uses for all the modeling and simulation work that we do here”, Douglas Ray (Mathematical Statistician Technical Lead at QE&SA Reliability Management Branch of US Army RDECOM ARDEC).
From ‘The Picatinny Voice’ Vol 29, No 17, August 19, 2016. http://www.pica.army.mil/Pictinny/voice/voice/pdfand the DOE Memorandum for Commanding General, Army Test and Evaluation Command, March 14, 2016http://www.dote.osd.mil/pub/policies/2016/20140314_Guidance_on_Valid_of_Mod_Sim_used_in_OT_and_LF_Assess_(10601).pdf
13
Industry Challenges
14
Ohio Aerospace Institute (OAI) and PACE
15
PACE goals
• Establish probabilistic design and validation methodologies as a routinely accepted tool by the aviation, propulsion and power generation industries
• Address development costs, verification and risk analysis in gas turbine engine design
• Leverage resources and technical expertise of its members for the accomplishment of selected research activities
Membership: Aero Engine OEM’s.
Ohio Aerospace Institute (OAI) is an organization under contract with the Air force Research Laboratory (AFRL) to create and manage the Probabilistic Analysis Consortium for Engines (PACE) consortium.
PACE Identified Research Interests
16
Method Challenges:
• Robust computational grid perturbation (from parametric CAD)
• Simulation time
• Stochastic input definition
• Model uncertainty quantification
• Validation data and strategy
Application Area Challenges:
• Secondary flow system temperatures and pressures
• Steady aero performance
• Unsteady aero loading
• Internal turbine cooling
• Depot Inspection Data analysis
• NSMS, TWE, etc., Uncertainty Quantification
UQ for Additive Manufacturing• Additive Manufacturing (AM) is an additive process of
constructing an object rather than the subtractive process of traditional manufacturing.
• AM is used throughout the aerospace community to develop prototypes, produce components with complex, aerodynamic geometries and to fix parts made by conventional manufacturing.
• Sources of AM Uncertainty for Selective Laser Sintering
• Variations in laser power, scan speed, hatch spacing, layer thickness, scan pattern
• Variations in powder morphology, melt and cooling temperatures
• Impact from process uncertainties
• Variations in microstructure, mechanical properties of yield and ultimate strengths and elastic modulus
• Mechanical properties are anisotropic
• Deformed geometry from residual stresses
• Role of UQ in AM
• Prediction of material properties and component performance from AM process parameters
Serve delaminations in Oak Ridge National Laboratory's development of a solely 3D printed car were caused by micro-level uncertainties in the powder. Photo Credit Oak Ridge National Laboratory “Computational Simulations for Additive Manufacturing”.
AM Institutes and Consortiums• National Additive Manufacturing Innovation
Institute (NAMII), manufacturing.gov• Additive Manufacturing Consortium (AMC),
Edison Welding Institute• Consortium for Additive Manufacturing
Materials (CAMM), NIST Office of Advanced Manufacturing
17
UQ for Digital Twins
• Digital Twins serves two main purposes:
• The object can be built and tested in a virtual environment instead of physical prototypes.
• With real-time data, it can optimize the life-span of the object reducing maintenance and downtime costs.
• The Role of UQ with Digital Twins
• Provide margins on predictions and identify uncertainties that cannot be modeled
• Determine the discrepancy between the physical object’s readings and the digital twin’s. A significant difference could be a sign for needed maintenance.
• Use sensitivity analysis to determine the significant parameters and which can be reduced. This will reduce the computational cost of future analysis.
Consortiums• Industrial Internet Consortium
established by AT&T, Cisco Systems, General Electric, Intel and IBM
• Industrie 4.0, Gemany• International Consortium of
Advanced Manufacturing Research, Kissimmee FL
“A Digital Twin is an integrated multiphysics, multiscale, probabilistic simulation of an as-built vehicle or system that uses the best available physical models, sensor updates, fleet history, etc., to mirror the life of its corresponding flying twin.”
- “The Digital Twin Paradigm for Future NASA and U.S. Air Force Vehicles”, E. H. Glaessgen, D.S. Stargel, AIAA 53rd Structures, Structural Dynamics, and Materials Conference: Special Session on the Digital Twin, 2012
Photo Credit: Digitally Cognizant Blog
18
Overview of Methods
19
UQ Has Many Process Components
20
Stat
isti
csHigh-dimensionalIn
terp
ola
tor
Mis
sin
g D
ata
Parallel ComputingNonparametrics
Co
mp
ress
ed S
ensi
ng
Latin Hypercube Design
Design of Experiments
Stochastic Optimization
Active Learning
Deep Learning
Mac
hin
e Le
arn
ing
Karmen Filter
Emulation
Bayesian Methods Inverse Problem
Multi-level Monte Carlo
Sparse Grids
Poly
no
mia
l Ch
aos
Gaussian Process
Data Fusion
Spatial-temporal Data
Dimension Reduction
Seq
uen
tial
Des
ign
L1-M
inim
izat
ion
GPU
Current UQ Technology Bottlenecks
21
• Conducting system level UQ with high-dimensional inputs
• UQ with fusion of simulation and physical data
• UQ with data from multiple simulation sources
• UQ with missing data
• UQ with complex data (e.g., transient and spatial responses)
• UQ with complex geometry
• UQ for design, manufacturing and operation (IoT)
Selected UQ Methods
22
Samurai Sudoku-Based Space-Filling Designs for Data Pooling
23
A Samurai Sudoku-Based Space-Filling Design
• The points can be divided into five overlapped slices.
• Red is the connecting slice and the same symbol denotes the same slice.
• Points in each slice is a Sudoku-based space-filling designs with 16 points.
24
The Connecting Slice of 16 Points
25
One Overlapping of Four Points
26
UQ Applications for Samurai Sudoku
• Quantify and adjust the differences between multiple simulation models
• Make inference for every single source
• Perform combined analysis for all sources
• Cross-validation
• Parallel Computing
27
Multi-Source UQ
• Build statistical emulators using continuous and categorical data types
• Examples • Combine numerical solutions
when solving the same model using different software routines
• Continuous and discrete inputs.
• Advantage: Combined emulator will likely have improved accuracy over separate emulators
• Differences and uncertainty due to solvers used can be quantified
28
Categorical Inputs
...Solver-2
Solver-n
Solver-1
Continuous Inputs
Combined Emulator
Output Predictions
...
Reference: Qian, Wu and Wu (2008)
Ex. 3: Mixed Data Emulation for Multi-Source Application
Collaboration with Caterpillar’s Virtual
Product Development team
Two inputs: UTS and Elastic Modulus at
discrete FE nodal locations
One output: Mean Stress
Analysis Procedure:
• Built emulators from simulation data
• Conduct parameter sensitivity analysis
• Propagate uncertainties
29
Ex. 3: Mixed Data Emulation for Multi-Source Application
Leave One Out Prediction
1.000
0.978
0.957
0.935
0.913
0.891
0.870
0.848
0.826
No
rmal
ized
Mea
n S
tres
s
0.82
6
0.84
8
0.87
0
0.89
1
0.91
3
0.93
5
0.95
7
0.97
8
1.00
0
The Individual Emulator has difficulty fitting the data due to a lack of sufficient
information
1.109
1.087
1.065
1.043
1.022
1.000
0.978
0.957
0.935
0.913
0.891
0.870
0.848
0.826
No
rmal
ized
Mea
n S
tres
s
0.8
26
0.8
48
0.8
70
0.8
91
0.9
13
0.9
35
0.9
57
0.9
78
1.0
00
1.0
22
1.0
43
1.0
65
1.0
87
1.1
09
Leave One Out Prediction
The Combined Emulator shows significant improvement due to sharing of
information between discrete input levels.
Individual Emulator Combined Emulator
30
Method 1: Method 2:
Ex. 3: Propagating Uncertainties from Mixed Data Emulation for Multi-Source Application
31
Response surface suggests bimodal behavior for Modulus
Grounding Models in Reality Using Statistical Calibration
32
Making Simulations More Realistic• Test data are used to calibrate specific parameters in a model to improve fit
between the model and data.
• Typically, calibration parameters that cannot be directly measured in a physical experiment are used.
• Directly measurable inputs are fed into the simulation and the calibration parameters are adjusted.
• Calibration ensures models are physically grounded and increases accuracy.
Recorded Variables
Calibration Parameters
Recorded Variables
Indirectly Observable Parameters
Experiment
Model
Statistical Calibration
Fitted Calibration Parameters
Real World System
System Model
Improved Predictive
ModelDiscrepancy Model
33
Bayesian Calibration
• Use of Expert opinions and beliefs through prior distribution.
• A framework to estimate calibration parameters and discrepancy, simultaneously.
• Estimate calibration parameters as distributions.
• Allow probabilistic predictions to make quantification of uncertainties from multiple sources (input, model, measurement) easier.
• Reference: Kennedy and O’Hagan (2001)
34
Example: Large Deflections of Cantilevered Beam
• Numerical model of a rectangular steel beam subject to a uniformly distributed load, w, and a point load, F, applied at the beam’s free end.
• Statistical Calibration was used to match the Young’s modulus, E, of the beam to an experimental data set which varied the point force, F, and recorded the vertical displacement of the beam’s free end, δy.
• Calibration was performed using 20 simulation runs and 7 physical test points
35
Beam Deflection Calibration Results
0.05
0.1
0.15
0.2
0.25
0.3
0 0.2 0.4 0.6
Tip
Dis
pla
cem
ent
[m
]
Applied Force [N]
Uncalibrated Simulation vs. Physical Data
Experimental
0.05
0.1
0.15
0.2
0.25
0.3
0 0.2 0.4 0.6
Tip
Dis
pla
cem
ent
[m
]
Applied Force [N]
Uncalibrated Simulation vs. Physical Data
Experimental
SmartUQ Cal. (194.15GPa)
0.05
0.1
0.15
0.2
0.25
0.3
0 0.2 0.4 0.6
Tip
Dis
pla
cem
ent
[m
]
Applied Force [N]
Uncalibrated Simulation vs. Physical Data
• The results from the uncalibrated beam bending model do not fit the physical data well.
• Calibrate the material’s Elastic Modulus.
• The calibrated results match the experimental data very well.
36
Results: Model Discrepancy
-0.003
-0.002
-0.001
0
0.001
0.002
0.003
0.00 0.10 0.20 0.30 0.40 0.50 0.60
Dis
cre
pe
ncy
[m
]
Applied Force [N]
Discrepancy with Experimental Data
True Discrepency
-0.003
-0.002
-0.001
0
0.001
0.002
0.003
0.00 0.10 0.20 0.30 0.40 0.50 0.60
Dis
cre
pe
ncy
[m
]
Applied Force [N]
Discrepancy with Experimental Data
True Discrepency
EstimatedDiscrepency
• Calibration yields an estimate of the Young’s Modulus: E = 194.15 [GPa]
• There are two curves plotted
– true discrepancy from test data and simulation
– estimated discrepancy using statistical calibration
• Discrepancy plots give us more than an R2 value between two curves
• Indicate how both model and data uncertainties impact the simulation results
over predictsunder predicts
37
A Chemical Reaction Model
• A Kinetics of the chemical reaction is built for 𝑆𝑖𝐻4 → 𝑆𝑖 + 2𝐻2.
• 𝑦 𝑡 = 𝑦0exp −𝑢𝑡 where𝑦 𝑡 = concentration of 𝑆𝑖𝐻4 as a function of time𝑦0 0 = initial concentration𝑢 is an unknown rate that is specific to this chemical reaction.
• In actual experiments:𝑦 𝑡 = (𝑦0−𝑐)exp(−𝑢𝑡) + 𝑐 + 𝜀, Where c = residual concentration of 𝑐 units is left unreactedand 𝜀 is the measurement error assumed to follow N(0,0.32).
• Initial conditions 𝑦0 = 5c = 1.5, 𝑢 = 1.7, unknown.
38
Results: Bayesian Calibration
Y
Dis
crep
ancy
t
Den
sity
u39
Thank you for your attention.
40
