University of Floridaufdcimages.uflib.ufl.edu/UF/E0/05/21/58/00001/ZHANG_Y.pdf · 4 ACKNOWLEDGMENTS...
Transcript of University of Floridaufdcimages.uflib.ufl.edu/UF/E0/05/21/58/00001/ZHANG_Y.pdf · 4 ACKNOWLEDGMENTS...
USING MULTIPLE PREDICTIONS AND MULTIPLE SOURCES OF DATA FOR DESIGN SPACE EXPLORATION
By
YIMING ZHANG
A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
UNIVERSITY OF FLORIDA
2018
© 2018 Yiming Zhang
To my mom, Cuixia, from whom I have got unconditional love and support
4
ACKNOWLEDGMENTS
First of all, I would like to thank my advisor Dr. Raphael Haftka for his extremely
valuable guidance and support. I would like to thank my co-advisor Dr. Nam-Ho Kim for
his generous sharing and patience. I would also like to thank the members of my
advisory committee, Dr. Sivaramakrishnan Balachandar, Dr. Thomas L. Jackson, Dr,
Herman Lam and Dr. George. Michailidis. I am grateful for their willingness to serve on
my committee and constructive criticism for my work.
I am also grateful to the ones that have collaborated with me and directly
contributed to my work: Dr. Chanyoung Park, Dr. Jaco Schutte, Dr. Siddharth Singh
Thakur, Yash A Mehta, John Meeker, Dr. Waruna Seneviratne, Upul R. Palliyaguru,
Dylan Rudolph, Aravind Neelakantan, Nalini Kumar, Carlo Pascoe, Dr. Vladimir
Balabanov, Dr. Leif Carlsson.
I am thankful to my amazing colleagues from Structural and Multidisciplinary
Optimization Group: Shu Shang, Maria G. Fernandez, Anirban Chaudhuri, Sam Nili,
Dawn An, Na Qiu, Garrett Waycaster, Yiwei Wang, Zhendong Guo, Charles F Jekel Jr,
Sangjune Bae, Ting Dong.
Financial support was gratefully provided by Defense Advanced Research
Projects Agency contract W911QX-13-C-0137 and Predictive Science Academic
Alliance Program contract DE-NA0002378.
5
TABLE OF CONTENTS page
ACKNOWLEDGMENTS .................................................................................................. 4
LIST OF TABLES ............................................................................................................ 9
LIST OF FIGURES ........................................................................................................ 12
ABSTRACT ................................................................................................................... 17
CHAPTER
1 INTRODUCTION .................................................................................................... 19
1.1 Motivation ......................................................................................................... 19 1.2 Function Prediction at Inaccessible Domain using Subsets of Samples ........... 20
1.3 Multi-fidelity Surrogates to Combine Datasets with Different Fidelities ............. 20 1.4 Adaptive Sampling When Sampling Cost is Variable in Design Space ............. 22
1.5 Design Space Sampling by Exploration and Replication .................................. 23 1.6 Approximating Field Data using Proper Orthogonal Decomposition ................. 24 1.7 Objectives ......................................................................................................... 25
1.8 Publications ...................................................................................................... 26 1.9 Outline .............................................................................................................. 27
2 FUNCTION PREDICTION AT ONE INACCESSIBLE POINT USING CONVERGING LINES ............................................................................................ 28
2.1 Background ....................................................................................................... 28 2.2 Principal Idea for Method of Converging Lines ................................................. 30
2.3 One-dimensional Function Estimation using Ordinary Kriging .......................... 32 2.4 Combination of Prediction from Multiple Lines .................................................. 33 2.5 Numerical Properties of Method of Converging Lines ....................................... 37
2.5.1 Algebraic Illustration Functions ................................................................ 37
2.5.2 Method of Converging Lines versus Multi-dimensional Approximation ... 39 2.5.2.1 Extrapolation .................................................................................. 39 2.5.2.2 Interpolation ................................................................................... 41
2.5.3 Applicability to Very High-dimensionality ................................................. 43 2.6 Long-distance Extrapolation and Interpolation of a 2D Drag Coefficient ........... 47
2.6.1 Introduction to the Drag Coefficient Function .......................................... 47 2.6.2 Extrapolation Results............................................................................... 49
2.6.2.1 Extrapolation using the method of converging lines ....................... 49 2.6.2.2 Comparison of 1D and 2D surrogates ............................................ 50
2.6.3 Interpolation Results ................................................................................ 52 2.6.3.1 Interpolation using the method of converging lines ........................ 52 2.6.3.2 Comparison of 1D and 2D surrogates ............................................ 53
2.7 Concluding Remarks ......................................................................................... 54
6
3 MULTI-FIDELITY SURROGATE BASED ON SINGLE LINEAR REGRESSION .... 56
3.1 Background ....................................................................................................... 56
3.2 Linear Regression Multi-Fidelity Surrogate ....................................................... 58 3.2.1 Bayesian Multi-fidelity Surrogate Frameworks ........................................ 58 3.2.2 Proposed Linear Regression Multi-fidelity Surrogate .............................. 59
3.3 Approximation of the Fluidized-Bed Process using LR-MFS ............................ 62 3.3.1 A Benchmark Dataset: Fluidized-Bed Process ........................................ 63
3.3.2 Comparison with the Bayesian MFS using Two-fidelity Dataset .............. 63 3.3.3 Effect of Noise for High-fidelity Dataset ................................................... 66 3.3.4 LR-MFS with Multiple Low-fidelity Datasets ............................................ 67
3.4 Approximation of the Modified Currin Function using LR-MFS ......................... 69
3.4.1 The Modified Currin Function .................................................................. 69 3.4.2 The Effect of Introducing Low-fidelity Dataset ......................................... 71 3.4.3 Comparison with Co-Kriging .................................................................... 72
3.4.4 Fitting High-fidelity Dataset with Different Levels of Noise using LR-MFS .............................................................................................................. 73
3.5 Concluding Remarks ......................................................................................... 75
4 MULTI-FIDELITY SURROGATE MODELING FOR APPLICATION/ARCHITECTURE CO-DESIGN ...................................................... 77
4.1 Background ....................................................................................................... 78 4.2 Related Research ............................................................................................. 80
4.3 Application and Architecture Models ................................................................. 83
4.3.1 CMT-nek .................................................................................................. 84 4.3.2 CMT-bone and CMT-bone-BE ................................................................. 85 4.3.3 Behavioral Emulation (BE) Simulation ..................................................... 85
4.4 Multi-fidelity Surrogates .................................................................................... 86 4.5 Developing MFS Model ..................................................................................... 88
4.5.1 Design Space .......................................................................................... 89 4.5.2 Validations of BE Simulation Results....................................................... 90
4.6 Evaluating MFS Predictions --- Three Case Studies ......................................... 92 4.6.1 Case Study 1: CMT-nek Predictions from BE Simulations ...................... 93
4.6.2 Case Study 2: CMT-bone Predictions from BE Simulations .................... 96 4.6.3 Case Study 3: CMT-nek Predictions from CMT-bone .............................. 97
4.7 Concluding Remarks ......................................................................................... 98
5 ON APPROACHES TO COMBINE EXPERIMENTS AND SIMULATIONS ............. 99
5.1 Background ..................................................................................................... 100 5.2 Open-Hole-Tension Tests ............................................................................... 101
5.2.1 Experiments .......................................................................................... 101
5.2.2 Finite Element Simulations .................................................................... 103 5.3 Approaches to Predict Mean Strength of Untested Points Assisted by
Simulations ........................................................................................................ 104
5.3.1 Model Calibration .................................................................................. 105
7
5.3.2 Algebraic Correction (Multi-fidelity Surrogate) ....................................... 106 5.4 Results for Combining Simulations and Experiments ..................................... 107
5.5 Using Tests from One Material System to Predict Strength of Another Material System ................................................................................................. 110
5.6 Predicting B-basis Allowables of Experimental Strength Assisted by Aimulations ........................................................................................................ 112
5.6.1 Background ........................................................................................... 113
5.6.2 Predicting B-basis Allowable at Untested Points using Surrogates ....... 115 5.6.2.1 Linear regression tolerance intervals ........................................... 115 5.6.2.2 B-basis allowables of Open-Hole-Tension tests ........................... 116 5.6.2.3 Predicting B-basis allowable of Open-Hole-Tension tests using
surrogate............................................................................................... 117
5.6.3 Predicting B-basis Allowable using Multi-fidelity Surrogate ................... 118 5.6.3.1 Simple framework of multi-fidelity modeling ................................. 118
5.6.3.2 Predicting B-basis allowable of using multi-fidelity surrogate ....... 119 5.6.3.3 Risky sampling strategy ............................................................... 120
5.7 Concluding Remarks ....................................................................................... 121
6 A STRATEGY FOR ADAPTIVE SAMPLING WHEN SAMPLING COST IS VARIABLE IN DESIGN SPACE ............................................................................ 122
6.1 Background ..................................................................................................... 122 6.2 Adaptive Sampling using Surrogate ................................................................ 125
6.2.1 The Basic Steps in Adaptive Sampling .................................................. 125 6.2.2 Kriging Surrogate .................................................................................. 126
6.3 Proposed Adaptive Sampling Strategy with Varying Sampling Cost ............... 127 6.3.1 Proposed Methodology .......................................................................... 127
6.3.2 Illustration of AS-C using 1D Algebraic Function ................................... 129 6.4 Multivariate Test Function and Algebraic Cost Functions ............................... 131
6.4.1 The Normalized Branin Function ........................................................... 131 6.4.2 Four Algebraic Cost Functions .............................................................. 132
6.5 Numerical Performance of AS-C ..................................................................... 134 6.5.1 Evaluation Plan ..................................................................................... 134
6.5.2 A Typical Case using the AS and AS-C ................................................. 136 6.5.3 Different Cost Functions and Different Initial Samples .......................... 138
6.6 Concluding Remarks ....................................................................................... 141
7 DESIGN SPACE SAMPLING BY EXPLORATION AND REPLICATION FOR ESTIMATING EXPERIMENTAL STRENGTH ....................................................... 142
7.1 Background ..................................................................................................... 142 7.2 Exploration versus Replication Sampling Schemes for Open-Hole-Tension
Tests .................................................................................................................. 146 7.2.1 Open-Hole-Tension Tests ..................................................................... 146 7.2.2 Experimental Strength Results and Polynomial Fit ................................ 148 7.2.3 Resampling Experimental Strength for Comparing Exploration and
Replication .................................................................................................. 149
8
7.2.4 Identifying the Distribution Type of OHT Strength ................................. 152 7.2.5 Effect of Between-batch Variability ........................................................ 155
7.3 Influence of Distribution Type on Comparison between Sampling Strategies for an Analytical Model of a Composite Plate .................................................... 157
7.3.1 Synthetic Noise using Three Types of Distributions .............................. 159 7.3.2 Fitting Strategy ...................................................................................... 160 7.3.3 Influence of Distribution Type on Sampling Strategies .......................... 161
7.4 Concluding Remarks ....................................................................................... 163
8 FIELD APPROXIMATION USING SURROGATES AND PROPER ORTHOGONAL DECOMPOSITION ..................................................................... 165
8.1 The Drop Falling for Direct Numerical Simulations of Multiphase Flows ......... 165
8.2 Approximating Flow Field using Proper Orthogonal Decomposition and Kriging ............................................................................................................... 166
8.3 Approximating Extreme Scalars using Field Data and Scalar Data ................ 168 8.4 Predicting Border of Multiphase Flow ............................................................. 171 8.5 Concluding Remarks ....................................................................................... 174
9 CONCLUSION ...................................................................................................... 176
APPENDIX
A ANALYTICAL EXPRESSIONS FOR DRAG COEFFICIENT ................................. 180
B ONE-DIMENSIONAL FUNCTION EXTRAPOLATION USING SURROGATES ... 182
B.1 Background .................................................................................................... 182 B.2 Possible Behaviors of Extrapolation using Surrogates ................................... 184
B.3 An Error Metric for 1D Extrapolation ............................................................... 186 B.4 Surrogate Comparison for Extrapolation ........................................................ 187 B.5 Estimating Extrapolation Distance .................................................................. 189
B.5.1 Effective Extrapolation Distance ........................................................... 189
B.5.2 Identification of Effective Extrapolation Distance using Correlation ...... 190 B.6 Concluding Remarks ...................................................................................... 192
C WEIBULL DISTRIBUTION WITH A HEAVY TAIL ................................................. 193
D SPECIMEN STRENGTH SUMMARY FOR OHT TESTING .................................. 194
E VARIABILITY FOR PREDICTION ACCURACY WHEN COEFFICIENT OF VARIATION IS 0.2 ................................................................................................ 198
LIST OF REFERENCES ............................................................................................. 199
BIOGRAPHICAL SKETCH .......................................................................................... 211
9
LIST OF TABLES
Table page 2-1 Target point, variable range and inaccessible domain of 8D algebraic
function ............................................................................................................... 38
2-2 Target point, variable range and inaccessible domain of 100D Styblinski-Tang function ...................................................................................................... 39
2-3 Target point, variable range and inaccessible domain of drag coefficient function ............................................................................................................... 48
2-4 Mean value of prediction and its uncertainty given by standard deviation for extrapolation of drag coefficient function along three lines ................................. 49
2-5 Interpolation results along three lines for drag coefficient function ..................... 52
3-1 Evaluation plan for the approximation of multi-fidelity dataset from Fluidized-Bed Process following [Le Gratiet, 2013] ............................................................ 65
3-2 Major settings for the modified Currin test function ............................................. 70
3-3 Key factors of the evaluation plan to approximate the modified Currin function . 71
3-4 Relative performance of LR-MFS and Co-Kriging for the approximation of the modified Currin function using 100 sets of HF samples ...................................... 73
4-1 Predicting execution times of CMT models based on a typical set of 12 samples and evaluating the prediction using R-RMSE (%) ................................ 94
4-2 Range of the scale factors for LS-MFS ............................................................... 98
5-1 w/D test matrix .................................................................................................. 102
5-2 Layup test matrix .............................................................................................. 102
5-3 Mean value of experimental strength of composite laminate made from MTM45-1 PWC2 3K PW G30-500 Fabric (ksi) ................................................. 102
5-4 Relative error and root mean square error at untested points based on 6 samples ............................................................................................................ 108
5-5 Experimental strength of composite laminate from T700SC-12K-50C/#2510 Plain Weave Fabric (ksi). .................................................................................. 110
5-6 Statistics of OHT tests at the 12 experimental locations ................................... 116
10
5-7 B-basis allowables of OHT tests at the 12 experimental locations using non-parametric approach ......................................................................................... 117
6-1 Details of AS and AS-C for the approximation of Forrester function with a budget of up to 4.5 total cost ............................................................................ 130
6-2 Original input variable space for the four cost functions ................................... 134
6-3 Key factors of the evaluation plan ..................................................................... 134
6-4 Details for a typical DOE of AS and ASC for the approximation of Branin function and linear cost function with up to 10 total cost ................................... 137
6-5 Median value of area metrics 2AR
and maxA (in %) using 100 sets of initial
samples ............................................................................................................ 139
6-6 Median values of total samples for AS and AS-C using 100 sets of initial samples ............................................................................................................ 140
7-1 w/ D ratio test matrix. 18 replicates are selected from three pre-preg batches . 147
7-2 Statistical properties of OHT strengths at given w/D with 18 replicates ............ 148
7-3 Relative difference between surrogate prediction and mean strength using quadratic PRS .................................................................................................. 149
7-4 Mean values of relative RMSE error (%) for surrogate prediction at w/D=3, 4, 6, 8 using 12 specimens and 24 specimens. Mean values reported are based on 1,000 sets of samples .................................................................................. 151
7-5 Statistics of Welch's t-test between specimens from batch 1 and batch 2 ........ 156
7-6 Mean values of relative RMSE error (%) for surrogate prediction at w/D=3, 4, 6, 8 using 12 specimens (3 replicates from 4 configurations for exploration, 4 replicates from three configurations for replication). Mean values reported are based on 1,000 sets of samples ....................................................................... 157
7-7 Parameter settings of composite laminate ........................................................ 159
8-1 Parameter settings ........................................................................................... 171
8-2 Fraction of contribution for first 4 modes to ensemble snapshots ..................... 171
B-1 Average AE for extrapolation of test functions using 30 sets of samples ......... 189
D-1 Specimen strength at w/d=3. Nominal thickness for one ply is 0.0086 inch. ....... 194
D-2 Specimen strength at w/d=4. Nominal thickness for one ply is 0.0086 inch. ........ 195
11
D-3 Specimen strength at w/d=6. Nominal thickness for one ply is 0.0086 inch. ........ 196
D-4 Specimen strength at w/d=8. Nominal thickness for one ply is 0.0086 inch. ........ 197
12
LIST OF FIGURES
Figure page 2-1 Illustration of inaccessible domain for surrogate modeling ................................. 28
2-2 Illustration of space-filling sampling (LHS) and Method of converging lines ....... 31
2-3 A typical set of extrapolation results from 8D Dette function. The three lines are from randomly selected vertices to the target vertex. ................................... 40
2-4 Summary of extrapolation results from multiple lines and 8D Kriging based on Dette function ................................................................................................ 41
2-5 A typical set of interpolation results from 8D Dette function ............................... 42
2-6 Summary of interpolation results from multiple lines and 8D Kriging based on Dette function ..................................................................................................... 43
2-7 A typical set of extrapolation results from 100D Styblinski-Tang function ........... 44
2-8 100 sets of extrapolation results from multiple lines based on 100D Styblinski-Tang function ..................................................................................... 45
2-9 A typical set of interpolation results from 100D Styblinski-Tang function ............ 46
2-10 100 sets of extrapolation results from multiple lines based on 100D Styblinski-Tang function ..................................................................................... 46
2-11 Response of drag coefficient on Re and M ......................................................... 48
2-12 Lines and samples selection for extrapolating and interpolation of drag coefficient function .............................................................................................. 48
2-13 Extrapolation results of drag coefficient function using samples from reduced sampling domain ................................................................................................ 50
2-14 Extrapolation results of drag coefficient function using multi-dimensional Kriging and converging lines............................................................................... 51
2-15 Interpolation results of drag coefficient function ................................................. 52
2-16 Interpolation results using multi-dimensional Kriging and converging lines ........ 54
3-1 Function values of the expT (high-fidelity) and 2T (low-fidelity) from Fluidized-
Bed Process ....................................................................................................... 64
3-2 Illustration of the resampling procedure to generate training and test data ........ 65
13
3-3 Performance of the surrogate models from 100 combinations of training samples and test sets for the approximation of fluidized bed process. ............... 66
3-4 Function values of the expT (high-fidelity) with synthetic noise from normal
distribution .......................................................................................................... 67
3-5 RMSE of LS-MFS to approximate HF samples of fluidized bed process with noise. The boxplots were based on 100 combinations of training samples and test sets. ...................................................................................................... 67
3-6 RMSE of LR-MFS to approximate HF samples from fluidized bed process with different low-fidelity dataset and all the low-fidelity dataset. The boxplots were based on 100 combinations of training samples and test sets. .................. 68
3-7 Scale factors 𝝆 of LR-MFS to approximate HF samples from fluidized bed process with different low-fidelity dataset and all the low-fidelity dataset. .......... 69
3-8 Responses of the modified Currin function in 2D space ..................................... 70
3-9 Effect of introducing low-fidelity samples for prediction of the modified Currin function without noise. PRS indicates the fit based on only HF samples. 3 and 10 are the number of HF samples. The median RMSE is based on 100 sets of samples ................................................................................................... 72
3-10 Comparison of Co-Kriging (CK) with LR-MFS for the approximation of the modified Currin function. 3 and 10 are the number of HF samples. The median RMSE is based on 100 sets of samples ................................................ 73
3-11 Comparison of LR-MFS with different level of noise for the modified Currin function. 3 and 10 are the number of HF samples. The RMSE is based on 100 sets of samples ............................................................................................ 74
3-12 Estimated noise level of LR-MFS with a different number of HF samples for the modified Currin function. 3 and 10 are the number of HF samples. The median STD is based on 100 sets of samples.................................................... 75
3-13 Conceptual illustration of the major epistemic uncertainty for LR-MFS with different number of HF samples ......................................................................... 75
4-1 Hierarchy of the CMT models ............................................................................. 84
4-2 Design of experiments for CMT-nek, CMT-bone, and BE simulations ................ 90
4-3 Validation of BE simulation against CMT-bone-BE ............................................. 91
4-4 Comparing CMT-bone mini-app and CMT-bone-BE skeleton app trends for various parameter values ................................................................................... 92
14
4-5 Difference between CMT-nek validation runs and multi-fidelity predictions based on CMT-bone simulation .......................................................................... 95
4-6 Difference between CMT-bone validation runs and multi-fidelity predictions based on BE simulation ...................................................................................... 96
4-7 Difference between CMT-nek validation runs and multi-fidelity predictions based on CMT-bone ........................................................................................... 97
5-1 Open-Hole Tension Test Specimen Configuration and observed failure mode 102
5-2 Mesh of finite element model for OHT test ....................................................... 104
5-3 Illustration of experimental results and simulations for OHT tests .................... 104
5-4 Failure criteria for a circular through thickness holes in an infinite plate ........... 105
5-5 Selected points from experimental strength (high-fidelity function) used to predict strength at other points ......................................................................... 108
5-6 Performance of different correction models for predicting OHT strength using FEA. Experiments were performed based on composite laminate made from MTM45-1 PWC2 3K PW G30-500 Fabric ......................................................... 109
5-7 Experimental strength of OHT tests from two different material systems ......... 111
5-8 Performance of different correction models for predicting OHT strength using FEA or another material. Experiments for prediction were performed based on composite laminate made from T700SC-12K-50C/#2510 Plain Weave Fabric ............................................................................................................... 111
5-9 Illustration of B-basis allowables based on replicates/repeated tests ............... 114
5-10 Typical histograms (out of 12 locations) based on 18 replicates from OHT tests .................................................................................................................. 116
5-11 Overall accuracy (relative RMSE) for PRS tolerance and direct PRS with increasing number of experimental tests .......................................................... 118
5-12 Overall accuracy (relative RMSE) for MFS and direct PRS with increasing number of experimental tests ........................................................................... 120
5-13 Samples corresponding to the worst three sets of predictions using MFS. Solid points indicate selected experiments out of 12 for MFS. ......................... 120
6-1 Basic steps of adaptive sampling for global approximation. One additional sample is added per iteration............................................................................ 126
15
6-2 Flowchart of the adaptive sampling with varying sampling cost. One additional sample is added per iteration at the point with maximum value to improve the global approximation. .................................................................... 128
6-3 One-dimensional algebraic example Forrester function ................................... 129
6-4 Initial samples and surrogate prediction for the Forrester function ................... 130
6-5 Surrogate predictions for adaptive sampling plans for Forrester function example ............................................................................................................ 131
6-6 Response of Normalized Branin function in the input variable space ............... 132
6-7 Response of the four cost functions in normalized input variable space .......... 133
6-8 An illustration of the evolution of accuracy for adaptive sampling with increasing total sampling cost ........................................................................... 135
6-9 Final samples for a typical DOE the approximation of Branin function with linear cost function ............................................................................................ 137
6-10 Evolution of adaptive sampling with increasing samples for a typical DOE for the approximation of Branin function and linear cost function .......................... 138
6-11 The population of area metrics for the approximation of the Branin function with four cost functions ..................................................................................... 139
6-12 The number of samples used for the approximation of the Branin function with four cost functions using AS and AS-C ................................................. 13940
7-1 Replication strategy vs. exploration strategy using two design variables and 64 samples ....................................................................................................... 144
7-2 OHT test specimen configuration and observed failure mode .......................... 147
7-3 Experimental data and quadratic polynomial response surface fitted to the data. ................................................................................................................. 149
7-4 Illustration for one set of resampled experimental strength representing exploration or replication strategy using 12 points ............................................ 150
7-5 K-S test statistic for OHT strength .................................................................... 154
7-6 Probability plots for the fit of pooled OHT tests................................................. 154
7-7 Statistical properties of specimen strength ....................................................... 155
16
7-8 Configuration and failure response of composite laminate. α is a constant to balance between horizontal and vertical loadings. θ is the orientation angle of lamina. .............................................................................................................. 158
7-9 Three distributions with mean value to be 8.5 ksi. The mean values of the five synthetic batches are 8.1 ksi, 8.5 ksi, 8.8 ksi, 8.4 ksi, 9.0 ksi..................... 160
7-10 Mean value of RMSE for relative error based on 1000 sets of samples with different distribution types. Number of samples indicates the total number of tests including 1~7 replicates for each test point. ............................................. 162
8-1 The front and velocity field in a rectangular box at different times .................... 166
8-2 Illustration of three snapshots at different times. Red point denotes location of minimum pressure ........................................................................................ 169
8-3 Kriging prediction of mode coefficients ............................................................. 170
8-4 Predicting minimum pressure from surrogate estimation and field approximation. Blue line is true value, red line is approximation ...................... 170
8-5 Coefficients of first 4 basis modes in variable space using Kriging ................... 172
8-6 Illustration for validation points, samples and variable space ........................... 173
8-7 Comparing simulation and POD prediction at 4 validation points ..................... 174
B-1 Three one-dimensional test functions for extrapolation .................................... 184
B-2 Extrapolation of three one-dimensional test functions ...................................... 184
B-3 Extrapolation of the logrithmic function ............................................................. 185
B-4 Extrapolation of the logrithmic function from different sampling domain (a) Full sampling domain and (b) Reduced sampling domain ................................ 186
B-5 Four test functions for surrogate selection. Domains close to origin are inaccessible domain. ........................................................................................ 188
B-6 Two types of effective extrapolation distance ................................................... 190
B-7 Correlation and distance ratio over effective distance while extrapolating 30 1D functions...................................................................................................... 191
E-1 Approximating the strength of composite laminate with the trade-off between exploration and replication. Variability of mean RMSE with normal noise, coefficient of variation is 0.2. ............................................................................ 198
17
Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy
USING MULTIPLE PREDICTIONS AND MULTIPLE SOURCES OF DATA FOR
DESIGN SPACE EXPLORATION
By
Yiming Zhang
May 2018
Chair: Raphael T. Haftka Cochair: Nam-Ho Kim Major: Mechanical Engineering
Design optimization and uncertainty quantification need to examine a large
number of candidate configurations. This is commonly done by doing simulations and/or
experiments at a number of design points and approximating at untested points using
surrogates. The objective of this work is to check on combining multiple ways to predict
by integrating various data sources and numerical models. A series of approaches are
proposed as in the following.
Function prediction towards inaccessible points could be encountered for
simulations beyond validation or unaffordable tests. The method of converging line is
proposed based on multiple 1D surrogates towards the inaccessible point. Multiple
predictions based on 1D surrogates are combined using Bayesian inference to increase
reliability. Converging lines proved to be much more reliable and accurate than the
multi-dimensional surrogate for a drag coefficient function and two algebraic functions.
Then the Linear regression multi-fidelity surrogate (LR-MFS) is proposed to
combine datasets with different fidelities. The proposed LR-MFS is expected to be
efficient (low computational cost) and robust for noisy data. The proposed LR-MFS has
18
been evaluated using two benchmark test functions and applied for: (i) estimating failure
strength of composite laminates. (ii) predicting performance of supercomputers
Current sampling plans implicitly assume that all samples have the same cost to
produce. An adaptive sampling scheme with varying cost (AS-C) is developed to handle
varying sampling cost in the design space. Two sets of surrogates are constructed for
the target function (quantity of interest) and the sampling cost. Comparing with standard
adaptive sampling scheme, AS-C led to more samples and more accurate fitting for the
approximation of selected test function.
Two more topics are discussed after that. (1) The first one is on the trade-off for
design space sampling between exploration versus replication. The study is based on a
set of experiments for the strength of a composite laminate. A resampling procedure is
proposed that compared exploration and replication. (2) The second one is on the
approximation of flow field using surrogate and proper orthogonal decomposition. The
field approximation is evaluated for predicting extreme quantities of interest (e.g.
pressure and border of multi-phase flow).
19
CHAPTER 1 INTRODUCTION
1.1 Motivation
Effective prediction of mechanical system response is often the basis for design.
Experimental tests and physics-based simulation are routinely adopted to approximate
system response at given design variables. The test results (referring to both
experiments and simulations) are scattered in variable space and statistical inference
approaches (e.g. surrogate/machine learning/ data mining) are adopted to predict
response at untested points in the variable space. These methods enable near-real-time
estimation of system responses for informed decision making.
Usually, a single approach is used to combine all the information in order to
make a prediction. In recent days, there has been interest in the simultaneous use of
multiple approaches rather than a single one [1-3]. For example, comparing and
potentially combining multiple surrogates proved to be more accurate or at least
equivalent than a single model [4]. Motivated by the combination of different surrogates,
other sources of predictions are examined in the dissertation to make the most use of
available information and therefore characterize mechanical system effectively. We
check on combining multiple ways to predict by integrating various data sources and
numerical models. Firstly, we explore the possibility to develop and combine surrogate
models based on subsets of data. Secondly, we aim to improve the predicative
capability of multi-fidelity surrogates using an algebraic function to combine datasets
with different fidelities. Thirdly, we examine the scheme to incorporate varying sampling
cost for the design of experiments. Fourthly, the trade-off between exploration versus
replication is discussed for the approximation of experimental strength. At the end, we
20
investigate a combination schemes for approximation of field data and approximation of
scalar quantities.
1.2 Function Prediction at Inaccessible Domain using Subsets of Samples
The focus of this chapter is a strategy for making prediction at a point where a
function cannot be evaluated. The key idea is to take advantage of the fact that
prediction is needed at one point and not in the entire domain. We explore the
possibility of predicting a multi-dimensional function using multiple one-dimensional
lines converging on the inaccessible point. The multi-dimensional approximation is thus
transformed into several one-dimensional approximations, which provide multiple
estimates at the inaccessible point. Kriging model is adopted in this work for the one-
dimensional approximation, estimating not only the function value but also the
uncertainty of the estimate at the inaccessible point. Bayesian inference is then used to
combine multiple predictions along lines. We evaluated the numerical performance of
the proposed approach using an 8-dimensional and a 100-dimensional functions in
order to illustrate the usefulness of the method for mitigating the curse of dimensionality
in surrogate-based predictions. Finally, we applied the method of converging lines to
approximate a two-dimensional drag coefficient function. The method of converging
lines proved to be more accurate, robust and reliable than a multi-dimensional Kriging
surrogate for single-point prediction. The chapter 2 on the method of converging lines is
based on the publication [5-7] co-authored by Chanyoung Park, Nam Ho Kim, and
Raphael T. Haftka.
1.3 Multi-fidelity Surrogates to Combine Datasets with Different Fidelities
Multi-fidelity surrogates (MFS) combine low-fidelity models with few high-fidelity
samples to infer the response of high-fidelity model for design optimization or
21
uncertainty quantification. Most publications in MFS focus on Bayesian frameworks
based on Gaussian Process. Other types of surrogates might be preferred for some
applications. In this work, a simple and yet powerful MFS based on single linear
regression is proposed, termed as Linear Regression Multi-fidelity Surrogate (LR-MFS),
especially for fitting high-fidelity data with noise. The LR-MFS considers the low-fidelity
model as a basis function and identifies unknown coefficients of both the low-fidelity
model and the discrepancy function using a single linear regression. Since the proposed
LR-MFS is obtained from the standard linear regression, it can take advantage of
established regression techniques such as prediction variance, D-optimal design, and
inference. The LR-MFS is first compared with three Bayesian frameworks using a
benchmark dataset from the simulations of a fluidized-bed process. The LR-MFS
showed a comparable accuracy with the best Bayesian frameworks. The effect of
multiple low-fidelity models was also discussed. Then the LR-MFS is evaluated using an
algebraic function with different sampling plans. The LR-MFS bested Co-Kriging for
55%~63% cases with increasing number of high-fidelity (HF) samples. For both
examples, the LR-MFS proved to be better than fitting only HF samples, and robust with
noisy data. The chapter 3 on LR-MFS is based on publication [8] co-authored with
Chanyoung Park, Nam Ho Kim and Raphael T. Haftka. The proposed MFS has been
applied to combine the experiments and simulations for two engineering problems. The
first one is behavioral emulation for application/architecture co-design of the exascale
computer. The collection of dataset and details of the experiments were kindly provided
by Aravind Neelakantan, Nalini Kumar and Herman Lam. The chapter 4 on multi-fidelity
surrogate is based on the publication [9] co-authored with Aravind Neelakantan, Nalini
22
Kumar, Herman Lam, Chanyoung Park, Nam Ho Kim and Raphael T. Haftka. The
second one is predicting the strength of composite laminate with a central hole. The
collection of dataset and details of the experiments were kindly provided by Jaco
Schutte, John Meeker and Upul R. Palliyaguru. The chapter 5 on multi-fidelity surrogate
modeling of composite laminate is based on publication[10, 11] and report co-authored
with Jaco Schutte, John Meeker and Upul R. Palliyaguru, Nam Ho Kim and Raphael T.
Haftka.
1.4 Adaptive Sampling When Sampling Cost is Variable in Design Space
Surrogate models have been developed to infer the response of mechanical
systems based on scattered tests/simulations for efficient design optimization. The
method to determine the number and location of the tests is the sampling plan or design
of experiment (DOE). Effective DOE enables surrogate desirable accuracy while
balancing the testing budgets. Current methods for DOE creation implicitly assume that
all samples have the same cost to produce. But the cost for tests/simulations may vary
in the input variable space because some configurations are more expensive to test or
simulate than others. This work explores the possibility to introduce varying sampling
cost for DOE. An adaptive sampling strategy with varying sampling cost (AS-C) is
proposed. Two sets of surrogates are constructed for the target function (quantity of
interest) and the sampling cost. Then a value metric is defined to estimate the
uncertainty to be reduced per cost. The work studies adaptive sampling for global
approximation and one new sample is added per iteration at the point with maximum
value. The proposed AS-C was evaluated using 1D and 2D algebraic functions. Four
different cost functions and 100 sets of initial samples were produced for evaluation. For
a fixed sampling budget, the AS-C was likely to lead to more samples and better
23
accuracy than the standard adaptive sampling strategy. The chapter 6 on adaptive
sampling is based on the publication [12] co-authored with Chanyoung Park, Nam Ho
Kim and Raphael T. Haftka.
1.5 Design Space Sampling by Exploration and Replication
The industry has to revert to experiments for practical design of composite
laminates when physical model/ simulations are inadequate for desirable accuracy or
require excessive computational resources. Surrogates are used to predict strength of
composite laminates in the design space by conducting an array of tests. It has been
reported that an exploration strategy to test as many different configurations as possible
is more effective than replication of fewer points for reducing test noise. The observation
was based on analytical test functions and synthetic Gaussian noise. This work first
studies real experiments to check whether the previous observation stands. We
examined test results of open-hole-tension (OHT) for composite plates that included 18
replicates per test point. We developed a resampling procedure that compared
exploration and replication and found that exploration proved to be more accurate for
prediction than replication. Secondly, we examined the major source of uncertainty for
surrogate prediction, which is variability of strength. The distribution of experimental
OHT strength was found to be not unambiguously independent and identically
distributed normal distribution as commonly assumed. The variation of specimen
strength is correlated rather than independent at different configurations due to the
between-batch variability. Consequently, the influence of distribution type was then
investigated on an analytical test function with three synthetic distributions. The
exploration strategy proved to be better than the replication strategy for all three
distributions. We found that the exploration strategy allows for higher order polynomial
24
surrogate to be used, which is a key point to improve characterization of a function with
complex dependence on design parameters. The chapter 7 on the design of
experiments is based on the publication [13, 14] co-authored with Jaco Schutte, Waruna
Seneviratne, Nam Ho Kim and Raphael T. Haftka.
1.6 Approximating Field Data using Proper Orthogonal Decomposition
Large-scale mechanical system simulation may need to explore the variable
space defining a flow field to understand physical phenomena, perform systematic
validation, sensitivity analysis or design optimization. Surrogate model is frequently
adopted as one major tool to approximate system response in variable space.
Comparing with common system response, which is a scalar, one specialty of
computational fluid dynamic analysis (CFD) is that the whole flow field may be of special
interest. Therefore, reduced order modeling (ROM) has been developed and became a
popular research topic of computational engineering in past decade.
A typical ROM procedure includes feature extraction using mathematical
decomposition [15-17]. Proper orthogonal decomposition (POD) is one popular
approach for model decomposition. Snapshots (matrix for the field of interest) are first
obtained at different variables (e.g. temporal, spatial of dimensional coordinates). The
POD is then performed to decompose the snapshots as a linear combination of basic
modes. The flow field at untested points can be reconstructed based on the
decomposed modes. The coefficients of the modes are traditionally obtained using
projection methods. ROM has also been used to approximate non-linear governing
equations instead of snapshots of flow field [18]. Jeong et. al [19] integrated machine
learning algorithm to approximate Navier-Stokes equations.
25
Kriging surrogate has been studied [20] to approximate whole flow field based
on incomplete information and illustrated competitive performance with ROM. The
combination of POD with Kriging surrogate has also been considered in [21]. After
obtaining basic modes and corresponding coefficients using POD. Kriging surrogates
were developed to estimate coefficients of modes at untested points. This method
proved to be effective for stationary field prediction under subsonic condition in [11].
For design optimization, we are more interested in extreme quantities. ROM
retains all the information from the simulated field [22, 23] and surrogate mainly
approximate specific quantity of interest. It might be useful to combine predicted
extreme values from surrogate with the reconstructed field for integrated analysis. We
propose one approach to correct predicted field data using POD against the
approximated scalar quantities from surrogates. A corrected flow field could be
conveniently used for both qualitative investigation and quantitative analysis.
1.7 Objectives
The main objectives of this research are to address the following challenges:
• Prediction at inaccessible points using multiple surrogates: The function prediction at inaccessible point is challenging and lacks validation. We propose to develop surrogates from different data sets to increase reliability. We apply the idea of multiple data sets to predictions of a single inaccessible point rather than estimating the whole variable space. Data sets are sampled along different lines converging to the target point.
• Multi-fidelity models for noisy data: The mainstream Bayesian multi-fidelity surrogates (MFS) have been commonly studied to combine multi-fidelity data. However, there is a risk for Bayesian MFS of estimating inappropriate hyper-parameters especially in the presence of noisy data. The least-square MFS (LS-MFS) using linear regression is proposed to handle noisy samples.
• Adaptive sampling for variable sampling cost: Current sampling schemes implicitly assume that all samples have the same cost to produce. However, the cost for tests/simulations may vary in the input variable space. We examine the possibility to introduce varying sampling cost for the adaptive sampling. Two sets
26
of surrogates are constructed for the target function (quantity of interest) and the sampling cost.
• Design space sampling by exploration and replication: The study on the trade-off between exploration and replication has been performed using algebraic functions and synthetic noise. The mathematical assumptions are usually violated to some extent for practical design. We tried to study the design of experiments using physical test following regulations.
• Field approximation: Proper orthogonal decomposition has been associated with surrogate models to predict flow field at untested points/configurations. Flow field is useful to provide physical insights, but the extreme quantities (e.g. scalars) are usually of interest for design optimization. We aim to evaluate the performance of field approximation regarding the extreme quantities regarding scalar values and borders of multi-phase flow.
1.8 Publications
The research has produced the following contributions with respect to:
• Function prediction at inaccessible domain using subsets of samples: The method of converging lines are proposed for function estimation towards one single inaccessible point. A multi-dimensional approximation is decomposed as multiple one-dimensional predictions. The extrapolation of 1D function and noisy data are also examined. [5, 6, 24, 25]
• Multi-fidelity surrogate modeling: The linear regression multi-fidelity surrogate is proposed and evaluated using benchmark dataset. The proposed schemes have been accepted and applied for two engineering applications (1) behavioral emulation of exascale computer and (2) predicting strength of composite laminate. [8-11, 26]
• Adaptive sampling with varying sampling cost: We identify the challenge of design of experiments with varying sampling cost. An adaptive sampling scheme is proposed correspondingly and evaluated on algebraic test functions. The proposed scheme has been applied to guide behavioral simulation of exascale computer. [12, 27, 28]
• Design space sampling by exploration and replication: Experimental strength usually suffers large variation. The trade-off between exploration and replication has been carefully studied using a benchmark composite laminate following regulation. [14, 29]
27
1.9 Outline
The organization of this dissertation is as follows. Chapter 2 elaborates the
approach to combine multiple predictions developed from different subsets of data. The
proposed scheme has been evaluated using algebraic test functions and drag
coefficient function. Chapter 3 presents the linear regression multi-fidelity surrogate to
datasets with variable fidelity. The proposed scheme has been evaluated using both
benchmark dataset and numerical examples. Chapter 4 is the application and
evaluation of the linear regression multi-fidelity on behavioral emulation. Multiple low-
fidelity models are discussed. Chapter 5 investigates the approaches to combine
experiments and simulations including model calibration, multi-fidelity surrogate.
Different combination schemes are evaluated for an open-hole-tension test in composite
laminate. Chapter 6 proposes the strategy for adaptive sampling when the sampling
cost varies in the design space. The strategy has been evaluated using algebraic test
functions. The effect of cost functions is elaborated. Chapter 7 discusses the design
space sampling by exploration and replication for estimating strength of composite
laminate. The experiments are thoroughly analyzed including distribution, uncertainty
and surrogate modeling. Chapter 8 talks the field approximation using surrogate and
proper orthogonal decomposition. The field approximation has been compared with
surrogate only to predict scalar quantities. Chapter 9 closes the dissertation with a
summary of achievements and future research opportunities.
28
CHAPTER 2 FUNCTION PREDICTION AT ONE INACCESSIBLE POINT USING CONVERGING
LINES
2.1 Background
In surrogate modeling, it is common to sample a function at scattered points and
fit the samples with an explicit function to approximate function values at un-sampled
points [30, 31]. The estimation procedure is called interpolation when target points are
inside the convex hull of sampled points, while it is termed extrapolation [32, 33]
otherwise, as shown in Figure 2-1. For one-dimensional samples, for example, convex
hull is set by the lower- and upper- bounds of samples. Surrogate models are routinely
used as interpolation schemes and may be less adequate for extrapolation, which is
commonly encountered while approximating high-dimensional functions [34]. A
surrogate is especially useful to estimate function value when the point of interest
cannot be sampled via simulation or experiment due to extreme conditions, lack of data,
limitations of simulation software or an experiment that is too dangerous to perform. For
example, 16 out of 500 simulations were reported to have failed for an aeroelastic
helicopter blade simulation in Glaz et al. [35] during design optimization.
Figure 2-1. Illustration of inaccessible domain for surrogate modeling. A) Interpolating
inaccessible region and B) extrapolating inaccessible region
B A
29
Several valuable efforts have been initiated towards reliable estimation at
inaccessible points. Neural networks, an advanced framework, has been reported to be
misleading for extrapolation beyond the region of samples [33] and is therefore mostly
used for interpolation. Balabanov et al. [36, 37] demonstrated that regularization
methods can improve extrapolation accuracy beyond sample range even if they
compromise interpolation accuracy. Wilson and Adams [38] proposed a Gaussian
process kernel function for better uncovering data patterns and leading to improved
prediction accuracy of forecasting. Richardson extrapolation has been commonly
adopted to predict response of finite element analysis towards extremely refined mesh
that are unaffordable [39, 40]. A specific polynomial form was used for Richardson
extrapolation to incorporate physical knowledge about convergence order. Hooker [41]
proposed an empirical trend function when predicting towards inaccessible point. In
prognosis for remaining life of mechanical systems, application of Gaussian Process
surrogate and neural network model has been discussed based on heuristic study [42].
Wei-Yin proved that setting a bound for extrapolation using linear model trees could
avoid large error [43]. Besides, numerous works can be found on forecasting one
dimensional time series data at inaccessible points (i.e. future) in financial fields and
weather forecasting, e.g. [44, 45].
Based on this literature, it is challenging to estimate prediction accuracy at
inaccessible points. Surrogate predictions, in particular, risk increasing uncertainty when
prediction points are far from samples [36, 46, 47]. This motivated us to estimate a
function value by creating multiple independent predictions from different data sets.
Introducing multiple predictions may reduce the uncertainty of blind predictions. A new
30
sampling pattern, called the method of converging lines, is proposed in this work. As a
first step towards improving the prediction capability for long-distance prediction using
surrogate model, we explore the possibility of predicting a multi-dimensional function
using multiple one-dimensional lines. The main idea is to select samples along lines
towards the inaccessible point. One-dimensional surrogates are then constructed based
on the samples along each line in order to predict the function value at the inaccessible
point. The method of converging lines transforms multi-dimensional function prediction
into a series of one-dimensional predictions, which provide multiple estimates at the
inaccessible point. Combination of multiple predictions based on Bayesian inference is
proposed to enhance the prediction accuracy. This chapter on method of converging
lines is based on the publication [5-7] co-authored by Chanyoung Park, Nam Ho Kim
and Raphael T. Haftka.
2.2 Principal Idea for Method of Converging Lines
The procedure to select locations where the function is to be sampled is called
design of experiments (DoE). For fitting a surrogate, it is common to select space-filling
DoEs (e.g. Latin Hypercube Sampling (LHS) [48]), which are geared to provide a good
representation of the design domain. The method of converging lines, in contrast,
serves the prediction at a single target point, targetx . This method locates sampling points
along several lines all intersecting at the target point.
The difference between LHS and the sampling scheme of the method of
converging lines in two- and three-dimensional space is illustrated in Figure 2-2. The
target point, targetx , is at the origin, and the domain within a normalized distance of 0.5
from targetx is set to be inaccessible. Three converging lines are shown in Figure 2-2 (B).
31
The orientations of line il may vary with applications in order to generate a good
estimate. Along a given line, local coordinate ( )j
localx of jth sample point (j)x is the Euclidean
distance between (j)x and
targetx as
( )j
localx (
ta et
j)
rgx x (2-1)
where ( )j
localx is a non-negative scalar and zero when (j)
targetx x . The local coordinate,
(j)x , is first normalized to be within [0,1] along each dimension to eliminate the effect of
different scales.
Figure 2-2. Illustration of space-filling sampling (LHS) and Method of converging lines. A) space-filling sampling (LHS) and B) Method of converging lines using 15 samples when the target point is at the origin and the domain at distance of less than 0.5 from the origin is inaccessible
A
B
32
The method of converging lines is expected to be ideal in certain conditions,
where sampling domain is relatively convenient to access in terms of cost and
feasibility,and sampling the target point is impossible or extremely challenging. In this
work, we focused on long-distance prediction, namely, the length of inaccessible
domain equals to that from accessible domain.
2.3 One-dimensional Function Estimation using Ordinary Kriging
The underlying assumption for prediction is that the function behaves similarly in
the accessible domain and inaccessible domain. Therefore, surrogate models providing
acceptable accuracy in sampling domain would be appropriate to estimate the function
at inaccessible points within a certain distance from samples. When this assumption
fails, surrogate prediction is likely to fail, whether we use a multi-dimensional surrogate
or the method of converging lines.
Multiple surrogates have been compared for the extrapolation of one-dimensional
problem as shown in Appendix B. Ordinary Kriging based on correlation kernel is
adopted for one-dimensional modeling due to its superior performance for
approximating noise-free data. Surrogate prediction based on Ordinary Kriging is
reported to avoid large errors at inaccessible points [24]. Kriging assumes that the
distance between sample points reflects a spatial correlation. The value of the function
at a point is correlated to the values at neighboring points based on their separation.
The correlation is strong to nearby points and weak with far away points, but strength
does not change based on location. This unique assumption provides a measure of
uncertainty of the prediction. We adopt the Gaussian model to approximate the
correlation between point x and point s as
33
2
1
exp( ), ( ), ( )n
i i ii
C f f x s
x s θ (2-2)
where n is the dimensionality of variable x and s, i is the hyper-parameter along ith
direction [49]. n is 1 for applying one-dimensional prediction using method of converging
line.
Prediction at an untested point x is formulated as
( ) 2
1
ˆ ˆ( ) ,
exp ( )
T
ni
i k k k
k
y
r s x
x r
(2-3)
where ( )is denotes ith sample point, is the mean of samples. The hyper-parameters
are obtained by maximizing the likelihood that the data comes from a Gaussian
process. Ordinary Kriging is implemented using the surrogate toolbox of Viana and Goel
[50].
The prediction of Ordinary Kriging and the corresponding uncertainty are
expressed as a conditional PDF for a given data set y as |p y x y , which follows a
Student's t-distribution [51]. A Student’s t-distribution is well approximated with a normal
distribution and converges to a normal distribution as the number of samples increases.
In this work, the surrogate prediction with uncertainty is expressed using a normal
distribution for computational efficiency. The conditional PDF of a prediction is defined
as a normal distribution 2ˆ ,N y x x where y x is the Kriging predictor and x is
the standard deviation at x.
2.4 Combination of Prediction from Multiple Lines
For making a prediction in n-dimensional space, one-dimensional surrogate
allows higher accuracy than using an n-dimensional surrogate with the same number of
34
samples. The method of converging lines proposes to combine multiple predictions at
the inaccessible point with independently built multiple one-dimensional surrogates
toward the point to reduce the chance of large error. The true response targety x is
predicted with multiple one-dimensional Kriging surrogates and the prediction of the ith
surrogate and the corresponding prediction uncertainty is defined through a conditional
PDF for the given ith data set as target |i ip y x y , which is a normal distribution having a
mean of the Kriging predictor targetˆ
iy x and a variance of the prediction variance at the
target point targetx .
The multiple predictions can be combined to obtain a single most probable
estimate. Various combination schemes are available to combine multiple predictions
[2, 3]. Here we adopt Bayesian inference [52] to combine Kriging predictions. In the
Bayesian inference, the conditional PDF of the true response from the Kriging based on
ith data set can be interpreted as a likelihood functions as
target target| |i ip y l yx y x y . By assuming independence of prediction uncertainties
between lines, the posterior distribution of the true response is obtained by multiplying
the likelihood functions. With m one-dimensional predictions, the posterior distribution is
expressed as
target target
1 1
| |m m
i i
i i
p y l y
x y x y (2-4)
The posterior distribution in equation Eq. (2-4) can be further derived based on
the approximation using normal distributions as
35
2
target target target target
1 1
2
target target
21
target
2
target target
2
target
ˆ| | ,
ˆ( )exp
2
ˆ( )exp
2
m n
i i i
i i
ni
ii
p y p y y
y y
y y
x y x x x
x x
x
x
x
(2-5)
where 2
target target targetˆ| ,i ip y y x x x is a PDF of a normal distribution at targety x for
the given mean of targetˆ
iy x and standard deviation of targeti x . targety x and target x
are the combined prediction and the standard deviation representing the prediction
uncertainty as
target
2
target
target
2
target
target
2
tar t
1
1ge
1
=1
1=
ˆ
1
ˆ
i
i
i
n
i
i
n
i
n
i
y
y
x
xx
x
x
x
(2-6)
Based on Eq. (2-4) and Eq. (2-5), the posterior distribution target
1
|m
i
i
p y
x y
follows a normal distribution, 2
target targetˆN ,y x x . From the expression of target x
in Eq. (2-6), it can be expected that surrogate prediction from the line with minimum
prediction variance will be most influential on the posterior distribution because the
combined prediction is a weighted average of the predictions based on the prediction
variances.
36
Compared with multi-dimensional approximation, the method of converging lines
is essentially a set of predictions from 1D surrogates. The 1D surrogate is likely to be
more robust than multi-dimensional approximation for two reasons: (a) The number of
hyper-parameters to be estimated for building a one-dimensional surrogate is one while
n hyper-parameters have to be estimated for an n-dimensional surrogate. Therefore, the
cost of evaluating samples to make a reasonable prediction is affordable regardless
dimensionality while the cost to build a n-dimensional surrogate increases excessively
(curse of dimensionality). (b) The 1D surrogate can be easily visualized reducing the
chance that failure of the surrogate fitting will go unnoticed.
One important feature for the method of converging lines is that lines may have
diverse performance. Some lines may be friendly for mathematical modeling, namely
enabling small and reliable prediction variance. These good lines will dominate
combined prediction based on Bayesian method. By potentially improving quality of
surrogate models and combining multiple lines, the method of converging lines is
expected to be more reliable regarding to pointwise estimation. Another bonus feature is
that the number of sampling points required along lines for accurate prediction may not
depend on dimensionality of the function. For example, if we were in 20-dimensional
space, prediction with a standard LHS may require hundreds of points, while it is
possible that three lines with 20 points may still suffice for prediction with the method of
converging lines. The effect of dimensionality will be discussed in section 2.5.3.
Several conditions may lead to large prediction errors even under valid
assumption that the inaccessible domain had a similar trend as that of the accessible
domain:
37
• Long-distance prediction usually risks large errors. Lacking samples near inaccessible domain would make it challenging to determine surrogate parameters and interpret goodness of fit.
• Different surrogate models or parameter settings is a major source of uncertainty. It is more challenging to select appropriate models for predictions towards inaccessible points especially when no prior information for noise is available.
• Some lines may not be trusted due to physical complexity or inappropriate modeling. Outlier prediction could be identified and excluded when most lines have similar predictions, but one line is significantly different. Outlier prediction comes from surrogate model with misleading estimation and prediction variance.
Note, however, that these difficulties are likely to be even more acuter if the
same prediction is attempted with a multi-dimensional surrogate, especially that only a
single estimate is obtained.
2.5 Numerical Properties of Method of Converging Lines
Even though it is tough to determine what function could be approximated and
how to predict towards inaccessible points, we tried to identify and discuss several
important factors based on numerical test functions. We examined the effect of lines
regarding to prediction accuracy, compared prediction from multiple lines with that from
multi-dimensional approximation, and effect of high-dimensionality.
Extrapolation has been reported to be more challenging than interpolation in
general [24]. Therefore, the following analysis distinguishes between these two
scenarios. We adopt Ordinary Kriging for prediction considering it could avoid huge
error regarding to prediction towards inaccessible points [24].
2.5.1 Algebraic Illustration Functions
This section examines properties of method of converging lines based on two
algebraic functions, Dette function and Styblinski-Tang function, from virtual library of
simulation experiments [53]. Dette function given in Eq. (2-7) is an 8D algebraic
38
function proposed by Dette et al. [54]. This function is highly curved in some variables
and less in others. The function is evaluated on the hypercube [0,1]ix for all i = 1, …,
8.
8
2 2 2 21 2 2 2 3 3
4 3
( ) 4( 2 8 8 ) (3 4 ) 16 1(2 1) ln 1i
ji j
f x x x x x x x x
(2-7)
For extrapolation, the target point for prediction was selected to be the vertex
where all eight variables were at their upper value, ix =1, i=1,2,…,8 as seen in Table 2-
1. The inaccessible domain was set to have the same length the as accessible domain
along all the variable directions. For interpolation, the target point was set to be =0.5ix
for i = 1, …, 8, which is the center of variable space. The inaccessible domain was set
to be the hypercube centered at the target point with a width of 0.5. Therefore, the
inaccessible domain had the same length as the accessible domain.
Table 2-1. Target point, variable range and inaccessible domain of 8D algebraic function
Target point targetx Variable range Inaccessible domain
Extrapolation ix =1 [0,1]ix [0.5,1]ix
Interpolation =0.5ix [0,1]ix [0.25,0.75]ix
The second test function is the Styblinski-Tang function, which is a popular
multimodal function for testing optimization algorithms with user-defined dimensionality
as seen in Eq. (2-8). The dimensionality d is set to be 100 to study the effect of high-
dimensionality on prediction. The function is evaluated on the hypercube [-1,3]ix for
all i = 1, …, d. Approximating this test function directly is challenging for classical
surrogate models considering its dimensionality.
39
4 2
1
116 5
2( ) i i i
d
i
x x xf x
(2-8)
For extrapolation, the target point for prediction was selected again to be the
vertex, where all the variables were at their upper bounds, =3ix as seen in Table 2-2.
The inaccessible domain was set to be a hypercube cornered at the target point and
had same length as the accessible domain along all the variable directions. For
interpolation, the target point was set to be =1ix , which was the center of variable
space. The inaccessible domain was set to be the hypercube centered at the target
point with a width of 1. Therefore, the inaccessible domain had the same length as the
accessible domain.
Table 2-2. Target point, variable range and inaccessible domain of 100D Styblinski-Tang function
Target point targetx Variable range Inaccessible domain
Extrapolation ix =3 [-1,3]ix [1,3]ix
Interpolation =1ix [-1,3]ix [0,2]ix
2.5.2 Method of Converging Lines versus Multi-dimensional Approximation
We first compared the proposed approach with multi-dimensional approximation
based on Dette function for extrapolation and interpolation respectively.
2.5.2.1 Extrapolation
To start with, the target point for prediction was selected to be the vertex where
all eight variables are at their upper value, ix =1, i=1,2,…,8. For the method of
converging lines, 3 lines were selected towards the target point. The other ends of the
lines were randomly selected from the rest of the 28 vertices of the domain. We selected
100 sets of lines randomly to compensate for the effect of line selection. Inaccessible
40
domain was set to have the same length as accessible domain along all the variable
direction. Six samples were evenly spaced in the accessible domain. A typical set of
extrapolation results based on Ordinary Kriging was shown in Figure 2-3. The function
along line 1 in Figure 2-3 (A) is unimodal like a bowl. Figure 2-3 (B) showed line 2 which
is wavy. Line 3 in Figure 2-3 (C) was close-to-linear and most friendly for approximation.
In the case in Figure 2-3, combined prediction was close to line 3, which had a much
smaller prediction variance. Note that even without knowledge of the true function, the
ability to visualize the data and predictions shown in Figure 2-3 is useful for spotting
which surrogate is likely to be trusted.
Figure 2-3. A typical set of extrapolation results from 8D Dette function. The three lines
are from randomly selected vertices to the target vertex. A) Line 1, B) Line 2 and C) Line 3.
A B
C
41
For multi-dimensional approximation, 18 samples were selected using LHS with
5000 iterations with the Matlab function lhsdesign. Sampling domain was restricted to
be the same as that to generate multiple lines. Inaccessible domain was a hypercube
cornered at target point.
Figure 2-4. Summary of extrapolation results from multiple lines and 8D Kriging based on Dette function
100 sets of predictions were generated using multiple lines and multiple LHS
designs. Prediction accuracy was quantified using absolute percent error and is
summarized using boxplot in Figure 2-4Error! Reference source not found..
Prediction from multiple lines had the error median to be 0.018% and 75% predictions
were less than 0.1%. In contrast, the minimum error of multi-dimensional approximation
was 25%.
2.5.2.2 Interpolation
For interpolation, the target point was set to be =0.5ix for i = 1, …, 8, which is the
center of variable space. Inaccessible domain was set to be the hypercube centered at
the target point with a width to be 0.5. Therefore, the inaccessible domain had the same
42
length as accessible domain. For the method of converging lines, 3 lines were selected
to intersect at the target point. The ends of lines were randomly selected from all
vertices. A typical set of interpolation results using Kriging is shown in Figure 2-5. All the
lines had a bowl shape with different minima. All Kriging models had a similar level of
prediction variance and overestimated function value at the target point slightly. Again,
we note that even without knowing the exact function, figures such as Figure 2-5
provide some protection against the failure of the fitting process.
Figure 2-5. A typical set of interpolation results from 8D Dette function. A) Line 1, B)
Line 2 and C) Line 3.
For multi-dimensional approximation, 18 samples were generated using LHS in
the accessible domain. 100 sets of prediction were generated to estimate randomness
A B
C
43
and summarized in Figure 2-6. Prediction using multiple lines had the error median to
be 3.24% and maximum error to be 3.31%. In contrast, prediction using 8D Kriging had
the error median to be 22.8% and minimum error to be 19.7%. Multiple lines generated
more accurate predictions compared to 8D kriging for this test function.
Figure 2-6. Summary of interpolation results from multiple lines and 8D Kriging based
on Dette function
2.5.3 Applicability to Very High-dimensionality
We adopted Styblinski-Tang function with 100 dimensions to illustrate
applicability to high-dimensionality. The dimension is too high to be approximated by
regular 100-dimensional surrogate models with a reasonable number of samples.
Therefore, the method of converging lines might be the only option for a function with
such a high dimension.
For extrapolation, the target point for prediction was selected again to be the
vertex, where all the variables are at their upper bounds, =3ix . The inaccessible
domain was set to be a hypercube cornered at the target point and had same length as
the accessible domain along all the variable directions. Three lines were randomly
selected from the rest of the 2100 vertices. Six sampling points were evenly spaced in
44
the accessible domain along each line. A typical set of extrapolation results is given in
Figure 2-7. The test function along all three lines was multimodal. Prediction error was
small at domain close to samples and increased with prediction distance in the
extrapolation domain. This example illustrates the difficulty in extrapolating long
distance. Not only the prediction can deteriorate with distance, but for a function that
changes its behavior, even the uncertainty estimate may be flawed.
Figure 2-7. A typical set of extrapolation results from 100D Styblinski-Tang function. A)
Line 1, B) Line 2 and C) Line 3.
100 sets of random lines were generated towards same target point to
compensate for variability. The error of 100 predictions were summarized in Figure 2-8.
The prediction error ranged around 28%. In this case, 100-dimensional surrogates are
A B
C
45
not possible, the performance of the proposed method of converging lines was not
compared with multi-dimensional surrogates, but the prediction error would be huge if
18 samples were able to fit a 100-dimensional surrogate.
Figure 2-8. 100 sets of extrapolation results from multiple lines based on 100D Styblinski-Tang function
For interpolation, the target point is set to be =1ix , which is the center of variable
space. The inaccessible domain is set to be the hypercube centered at the target point
with a width of 1. Therefore, the inaccessible domain has the same length as the
accessible domain. For the method of converging lines, 3 lines are selected toward the
target point. The other end of lines is randomly selected from all vertices. A typical set of
interpolation results is shown in Figure 2-9. All the lines have a bowl shape, and the
Kriging models overestimated function value at the target point. 100 sets of prediction
are generated to compensate for randomness and summarized in Figure 2-10.
Prediction using multiple lines had the error between 7.4% and 9.3%. Based on the
tests using 100D Styblinski-Tang function, the method of lines enables estimation of
high-dimensional function at one point using multiple lines. Such estimation is practically
impossible with a small number of samples using a multi-dimensional surrogate. In
46
particular, it is impossible to fit a Kriging surrogate with 18 points in 100-dimensional
space.
Figure 2-9. A typical set of interpolation results from 100D Styblinski-Tang function. A)
Line 1, B) Line 2 and C) Line 3.
Figure 2-10. 100 sets of extrapolation results from multiple lines based on 100D Styblinski-Tang function
A B
C
47
2.6 Long-distance Extrapolation and Interpolation of a 2D Drag Coefficient
A drag coefficient model of a spherical particle [55] is presented in this section to
demonstrate long-distance function estimation at an inaccessible point using the
method of converging lines.
2.6.1 Introduction to the Drag Coefficient Function
The drag coefficient is a dimensionless quantity that measures the quasi-steady
drag coefficient of a spherical particle in compressible flow. For a given particle, the
drag coefficient, DC , is a function of Mach number, M, and Reynolds number, Re. The
range of applicability for DC in this work is limited to the supersonic (1<M 1.75 ) domain
to avoid the physical discontinuity at M=1. The Reynolds number is limited to
5Re 2 10 to avoid turbulence of the attached boundary layer. It is considered that the
governing physics is consistent within these limits. The analytical expression for DC is
given in DC = (Re, )f M in [55]. The dependence of the drag coefficient on Reynolds
number and Mach number is shown in Figure 2-11, which shows that the behavior can
be made smoother by transforming Re to logarithmic coordinates. The target point,
variable range for analysis and inaccessible domain of drag coefficient function are
summarized in Table 2-3 and plotted in Figure 2-12. For extrapolation, prediction
domain is a square located on the upper right corner as seen in Figure 2-12 (A). For
interpolation, prediction domain is set to be a square in the center of variable range as
seen in Figure 2-12 (B). Length of sampling domain and prediction domain are the
same along each line for both cases.
48
Figure 2-11. Response of drag coefficient on Re and M. A) Natural coordinate and B) log10 coordinate
Table 2-3. Target point, variable range and inaccessible domain of drag coefficient function
Target point targetx Variable range Inaccessible domain
Extrapolation log10(Re)=5.301
M=1.75
log10(Re) [2 5.301] ,
M [1 1.75] ,
log10(Re) [3.65,5.301]
M [1.375,1.75]
Interpolation log10(Re) 3.6505
M=1.375
log10(Re) [2 5.301] ,
M [1 1.75] ,
log10(Re) [2.8253,4.4757]
M [1.1875,1.5625]
Figure 2-12. Lines and samples selection for extrapolating and interpolation of drag coefficient function. Red points denote samples. A) Extrapolation and B) Interpolation.
A B
A B
𝛼3 𝛼2
𝛼1 = 0
49
2.6.2 Extrapolation Results
2.6.2.1 Extrapolation using the method of converging lines
Instead of sampling entire range of sampling domain, six uniform samples were
selected from half of the sampling domain as shown in Figure 2-12 (A). This was
because the prediction variance decreased noticeably when samples were closer to the
target point. When the inaccessible domain is in the interpolation region, the sampling
scheme in Figure 2-12 (B) can be used, which will be considered in the following
section. Table 2-4 summarizes statistics of the surrogate predictions. Predictions of
function value along variable M with constant Re (line 3) had the smallest prediction
variance as shown in Table 2-4 and Figure 2-13 (C), indicating better accuracy along
this line.
Table 2-4. Mean value of prediction and its uncertainty given by standard deviation for extrapolation of drag coefficient function along three lines
Line 1 Line 2 Line 3
i 1.021 0.9352 0.9755
i 0.0786 0.0725 0.0003
By substituting extrapolation results of each line from Table 2-4 into Eq. (2-7),
most probable estimation is 2
target 1 target 2 target 3 target| , , N(0.9755,0.0003 )p y y x y x y x .
Note that the posterior distribution was dominated by the prediction along line 3
because the prediction variance (2) was much smaller than for the other lines. The
small prediction variance denotes high reliability of extrapolation. Comparing with true
function value 0.9766, absolute error is 0.001 and relative error is 0.1%. Note also that
50
the less accurate lines (Line 1 and Line 2) provide a reasonable warning of their lack of
accuracy in their prediction uncertainties, i .
Figure 2-13. Extrapolation results of drag coefficient function using samples from reduced sampling domain. A) Line 1, B) Line 2 and A) Line 3.
2.6.2.2 Comparison of 1D and 2D surrogates
One interesting question is how sensitive the prediction is to the location of lines.
The sensitivity to the selection of lines was tested by varying the angle of lines. In
Figure 2-12, the angle between lines and the horizontal axis with constant M = 1.75
were set to be 1 2 3 , , , respectively, where 1 2 3= 90 0, =45, . In order to test the
effect of lines, 1 varied from 0 to 5 , 2 from 40 to 50 , and 3 from 85 to 90 in
the increment of 1 . 396 sets of three lines were generated based on different
A B
C
51
combinations of 1 , 2 and 3 . Extrapolation results based on each set of lines were
plotted using box-plot in Figure 2-14 (B). All the extrapolation results concentrated near
the true function value, 0.9766, varying from 0.9751 to 0.9756.
Figure 2-14. Extrapolation results of drag coefficient function using multi-dimensional Kriging and converging lines. A) A typical set of samples from LHS and B) accuracy for 100 sets of extrapolations.
In order to examine the performance of multi-dimensional extrapolation, 2-D
Kriging is adopted. 18 samples are selected in the reduced sampling domain using LHS
(5000 iterations maximizing minimum distance) as shown in Figure 2-14 (A). As
converging lines benefitted from the reduced domain, 2-D Kriging also benefited from it
based on our tests. 2-D extrapolation is repeated 20 times to compensate for
randomness of sampling pattern. For a typical 2D extrapolation out of the 20 cases, i.e.
corresponding to the case with median absolute error, the prediction was
2N(0.9869,0.0313 ) . In Figure 2-14 (B) we can see that boxplot of 2-D Kriging prediction
has a median equal to 0.9799 and standard deviation to be 0.0261. 2D Kriging
generated larger variation and bias than converging lines even with 20 sets of data. The
method of converging lines proved to be more reliable and robust for one-point
extrapolation of drag coefficient function.
A B
2 2.5 3 3.5 4 4.5 5 5.5
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.82D Samples(18)
log(Re)
M
Border
Reduced sampling domain
Target Point
52
2.6.3 Interpolation Results
2.6.3.1 Interpolation using the method of converging lines
Interpolation using method of converging lines was performed similarly as shown
in 2-11 (b). We selected samples in the half of sampling domain to reduce prediction
variance. Interpolation results are plotted in Figure 2-15 from three lines. Predictions at
target point are provided in Table 2-5.
Table 2-5. Interpolation results along three lines for drag coefficient function
Line 1 Line 2 Line 3
i 0.9408 0.9943 0.9223
i 0.0146 0.0742 5×10-6
Figure 2-15. Interpolation results of drag coefficient function. A) Line 1, B) Line 2 and C)
Line 3.
A B
C
53
As in the case of extrapolation, the prediction variance on Line 3 is by far the
smallest. By substituting prediction results of each line from Table 2-5 into Eq. (2-7),
most probable estimation is 2
target 1 target 2 target 3 target| , , N(0.9223,0.000005 )p y y x y x y x ,
essentially identical to the result of Line 3. Comparing with true function value, 0.9224,
absolute error is 0.0001 and relative error is 0.01%.
2.6.3.2 Comparison of 1D and 2D surrogates
We have more freedom to select lines for interpolation while fixing intersection
angle between lines. Therefore, the intersection angle between line 1, line 2 and line 3
were fixed to be o45 successively. Three lines rotated counterclockwise gradually from
o0 to o45 in 20 steps. 20 sets of 1D interpolation results were then obtained accordingly.
For 2D Kriging, 18 samples are selected in the sample domain using LHS (5000
iterations maximizing minimum distance for the following analysis as shown in Figure 2-
16 (A). For a typical 2D interpolation out of the 20 cases, i.e. corresponding to the case
with median absolute error, the prediction was 2(0.9317,0.0012 )N . It is not only less
accurate than the method of converging lines, but also has a substantial underestimate
of the uncertainty in the prediction in the prediction variance.
Predictions of 1D and 2D Kriging were summarized in Figure 2-16 (B). 1D Kriging
predictions varied from 0.9223 to 0.9232. 2D Kriging prediction generated a median to
be 0.9255, and standard deviation equal to 0.0195. The box plot showed that for some
DOEs the 2D prediction was grossly inaccurate.
54
Figure 2-16. Interpolation results using multi-dimensional Kriging and converging lines.
A) 2D samples (18) and B) Comparison of 2D and 1D interpolation.
2.7 Concluding Remarks
We presented a method for improving the accuracy of function estimation at an
inaccessible point by taking advantage of the fact that prediction is needed at only one
point instead of everywhere. The proposed method of converging lines transforms a
multi-dimensional prediction into a series of 1D predictions. Most probable estimation is
obtained by combining consistent predictions from different lines using Bayesian
inference while assuming all the lines are independent.
One-dimensional prediction was performed using Kriging surrogate model.
Function prediction is only feasible under the assumption that the function behaves
similarly in the sampling domain and prediction domain. We illustrated its applicability to
high dimensional functions with two algebraic examples, 8D Dette function and 100D
Styblinski-Tang function. The proposed approach proved to be more accurate and
reliable than multi-dimensional approximation for the 8D Dette function. The proposed
approach enabled moderately accurate estimation of 100D Styblinski-Tang function with
18 samples. The two examples also illustrated another advantage of the method, which
A B
55
is easy visualization of the fitted surrogate. Finally, the examples illustrated that
combining multiple predictions using Bayesian inference could increase the chance of
obtaining valid estimation.
Next the method was illustrated for a physical example, extrapolation and
interpolation of a two-dimensional drag coefficient. For this example, as well, the
method of converging lines proved to be much more reliable and accurate than 2D
Kriging.
Converging lines were randomly selected in variable space from current study.
Predicative capability for method of converging lines is expected to be improved while
introducing refined strategies to determine location of lines and samples along a line.
Another interesting topic in the future is consistency between multiple predictions along
lines. The extent of consistency between lines may reveal the major source of
uncertainty such as model-form uncertainty, outlier effect or unexpected behavior of
target function at inaccessible domain.
56
CHAPTER 3 MULTI-FIDELITY SURROGATE BASED ON SINGLE LINEAR REGRESSION
3.1 Background
Surrogate models have been applied for various engineering design optimization
and uncertainty quantification problems, which require many physical tests or
simulations [30, 56]. Since physical tests and high-fidelity simulations are usually time-
consuming or expensive, surrogate models can be developed based on scattered tests
to infer the system response [11, 57]. However, performing a number of tests or
simulations needed for fitting an accurate surrogate is often too expensive. Multi-fidelity
surrogates (MFS) can provide a solution for this problem by combining a small number
of accurate simulations or tests with lower accurate models. Multi-fidelity models are
commonly encountered for engineering analysis, such as finite element simulations with
different resolutions and numerical simulations combined with physical tests. Various
frameworks have been proposed to predict mechanical system responses by combining
data from different fidelities for design optimization and uncertainty quantification, e.g.,
see reviews by Fernández-Godino et al.[58] and Peherstorfer et al. [59].
MFS usually introduces a discrepancy surrogate to model the difference between
low- and high-fidelity samples [60]. The popular Bayesian discrepancy-based MFS was
introduced in [61-64] and demonstrated its effectiveness for the approximation of multi-
fidelity datasets. Part of its success compared to earlier MFS was because of scaling
the low-fidelity model in addition to introducing the discrepancy [65]. Model calibration
associated with surrogate is another approach to combine multi-fidelity dataset. The
physical parameters required by the low-fidelity model are first optimized and fitted by a
surrogate to improve agreement between low-fidelity predictions and high-fidelity
57
samples [66-68]. A comprehensive Bayesian MFS model that uses both calibration and
discrepancy was proposed in [66], offering greater flexibility, although this is the most
complex framework. However, the Bayesian framework requires a model for the
uncertainty structure, and has been applied almost exclusively for Gaussian process
(GP). As demonstrated by Viana et al. [3], there are situations where GP is not the most
accurate surrogate, so there is a merit to having an MFS arsenal that includes other
surrogates.
We propose a simple and yet powerful MFS based on regression, or least-
squares fit, that is LR-MFS. The high-fidelity behavior is approximated by a linear
combination of low-fidelity predictions and a discrepancy function. In this work, the
discrepancy function is represented by a linear combination of monomial basis
functions, but the approach is also applicable to other basis functions such as radial
basis functions. The key idea is to consider the low-fidelity model as one additional
basis function in the multi-fidelity model with the scale factor as a regression coefficient.
The design matrix for least-squares estimation consists of both the low-fidelity model
and discrepancy basis functions. Then the scale factor and coefficients of the basis
functions are obtained simultaneously by solving the regression set of linear equations.
LR-MFS is expected to easily use various tools available for regression fits, such as
prediction variance and D-optimal designs. An effective LR-MFS could serve as the
basis for uncertainty propagation [69, 70] and design optimization[71].
The work is organized as follows. Section 3.2 briefs the Bayesian MFS and
presents the proposed LR-MFS using single linear regression. Section 3.3 evaluates the
proposed LR-MFS using a benchmark dataset from a fluidized-bed process. The LR-
58
MFS is compared with three Bayesian frameworks. In section 3.4, the LR-MFS is further
evaluated using an algebraic test function with different levels of noise and different
sampling plans. This chapter on LR-MFS is based on publication [8] co-authored with
Chanyoung Park, Nam Ho Kim and Raphael T. Haftka.
3.2 Linear Regression Multi-Fidelity Surrogate
The basics of Bayesian MFS is discussed first to understand the major
components of an MFS and potential limitations. Then the Linear Regression Multi-
fidelity Surrogate (LR-MFS) is proposed which also includes a scale factor and a
discrepancy function as with the Bayesian MFS.
3.2.1 Bayesian Multi-fidelity Surrogate Frameworks
Kriging[72] or Gaussian process is one of the most popular surrogate models for
design optimization. The Kriging is a stochastic process (a collection of random
variables), such that every finite collection of those random variables has a multivariate
normal distribution, i.e. every finite linear combination of the variables is normally
distributed. Kriging naturally provides the prediction variance at an untested point. Co-
Kriging developed by [Kennedy and O’Hagan, 2000] [61] introduces the correlation
between multi-fidelity dataset. Let us denote a set of sample 1, , Ln
L L LX x x is from
the low-fidelity model Lf x and 1, , Hn
H H HX x x from the high-fidelity model Hf x .
The corresponding function values are 1, , Ln
L L Ly yy and 1, , hn
H H Hy yy ,
respectively. Co-Kriging is made of two sets of correlated Kriging models. The Kriging
surrogate ˆ ( )Lf x is first constructed based on ,L LX y . The second Kriging ˆ ( )Hf x is then
built based on the discrepancies ˆH L Hfy X as the discrepancy function. The scale
59
factor is estimated from maximum likelihood estimation as part of ˆ ( )Hf x . Variations
of the Bayesian multi-fidelity surrogate are developed to improve accuracy and
computational efficiency [62, 73].
Since Kennedy and O’Hagan (2000) introduced the Bayesian multi-fidelity
surrogate, subsequent studies supported that the GP based surrogate generally
provided accurate predictions [74]. However, the use of GP model also embraces
technical difficulties regarding hyper parameter estimation. Finding the hyper
parameters of the GP model is equivalent to finding a global optimum solution for a
highly nonlinear likelihood function [75, 76]. In addition to the computational burden of
likelihood function evaluations, likelihood functions are often plagued with numerical
instability due to covariance matrix inverse operation. Forrester et al. (2007) introduced
co-Kriging that is featured with better computational efficiency. Qian and Wu (2008)
proposed the use of Marko chain Monte Carlo (MCMC) and sample average
approximation algorithm for hyper parameter estimation [62]. Le Gratiet (2013)
proposed an approach to reduce the computational burden by simplifying the
covariance matrix inversion operation [73]. For data with noise, GP based surrogates
tend to underestimate the noise by overly smoothing the prediction [77].
3.2.2 Proposed Linear Regression Multi-fidelity Surrogate
In this work, we propose the least-squares estimation of the scale factor ρ and
the coefficients of the discrepancy function. The linear regression multi-fidelity (LR-
MFS) is built with two surrogates, ˆ ( )Lf x and ˆ( ) x , which are the linear regression fitted
to the low-fidelity data and the discrepancy data as in Eq. (3-1)
60
ˆ ˆ ˆ( ) ( ) ( )H L
f f x x x (3-1)
where the scale factor ρ and discrepancy function ˆ( ) x are obtained together by
minimizing the difference between the scaled low-fidelity samples and the high-fidelity
samples as in Eq. (3-2), where H
d is difference between high-fidelity samples and
scaled low-fidelity samples as defined in the Eq.(3-3). ( HX , Hy ) denotes the high-
fidelity dataset including n samples.
ˆ, ( )
ˆ ˆmin : ( ) ( )T
H H H H
X d X dx
(3-2)
ˆ ( )H L H H
f d yX (3-3)
The MFS in Eq. (3-1) can be expanded with the discrepancy function being
represented by p basis terms (e.g, monomial bases), as
1
ˆ ˆ ˆ( ) ( ) ( )
ˆ= ( )+ X
H L
p
L i i
i
f f
f b
x x x
x x (3-4)
where Xi x denotes the thi basis term, and
ib is the coefficient of X
ix . In the
proposed LR-MFS, the traditional design matrix is augmented by including the low-
fidelity model with unknown scale parameter ρ. In least-squares estimation, the
relationship between high-fidelity samples and model predictions can be given as
Y XB e (3-5)
for LR-MFS, where
61
(1) (1) (1)(1) (1)1
1
1 ( 1) 1 1
( ) ( )( ) ( ) ( )
1
ˆ ( ) X X
, , ,
ˆ ( ) X X
L H H p HH H
n n p p n
n nn n nH HL H H p H
p
fb
fb
x x xy e
Y X B e
y ex x x
(3-6)
In the above equation, Y is the vector of high-fidelity samples, X is the
augmented design matrix, B is the parameter vector and e is the vector of residual
errors. By augmenting the design matrix, the scale parameter and unknown coefficients
of the discrepancy functions are estimated simultaneously. This means that the low-
fidelity model is considered as an additional basis function. An additional advantage of
the proposed LR-MFS is that it is unnecessary to build a surrogate of the low-fidelity
model when low-fidelity simulations are cheap. The method only requires evaluating the
low-fidelity model at the same locations with the high-fidelity samples. The unknown
parameters in LR-MFS are obtained by using the standard regression technique as
1
T T
B X X X Y (3-7)
Linear regression assumes that the residual errors are normal, independent and
identically distributed. The prediction error is estimated from Eq.(3-8)
2ˆ1
T
n p
e e (3-8)
the prediction variance at a point x is obtained from Eq.(3-9)
1
2ˆ ˆ[ ( )]T T
HVar f
x X x X X X x (3-9)
where the X x denotes the design matrix of linear regression at the point x for
prediction.
62
The scale factor implies the level of trend similarity between ˆ ( )H
f x and ˆ ( )L
f x ,
and plays a critical role to approximate multi-fidelity data. Negative values or extremely
large values of indicates a risky prediction, which are likely to be associated with
undesirable low-fidelity models, inappropriate surrogate forms, or inadequate samples.
The discrepancy function ˆ( ) x is likely to be modeled well by a low-order PRS when
Lf x had a similar trend as Hf x .
There are nice properties of LR-MFS. Firstly, the LR-MFS could be easily applied
to more than two-fidelity models by augmenting the design matrix with multiple low-
fidelity models with multiple scale factors as shown in Eq.(3-10), where k is the number
of low-fidelity models. Secondly, it is possible to use other basis functions beyond
monomials to approximate the discrepancy function as in this work. Thirdly, the
established approaches could be directly used to LR-MFS such as (1) handling noise
from different distributions, (2) confidence intervals, prediction intervals and tolerance
limits for conservative estimation, (3) stepwise regression to reduce model-form
uncertainty, (4) Obtaining optimal design of experiments, such as D-optimal designs.
1 1
ˆ ˆ( ) ( )+ Xpk
H j Lj i i
j i
f f b
x x x (3-10)
3.3 Approximation of the Fluidized-Bed Process using LR-MFS
In this section, the proposed LR-MFS was compared with three Bayesian multi-
fidelity surrogates using the dataset from Fluidized-Bed Process. The effect of different
low-fidelity models was also discussed.
63
3.3.1 A Benchmark Dataset: Fluidized-Bed Process
A comparison is made between the proposed LR-MFS and three Bayesian multi-
fidelity surrogates developed by [Kennedy and O’Hagan, 2000][61], [Qian and Wu,
2008][62] and [Le Gratiet, 2013][73]. This case was selected because the results for the
Bayesian implementation are given in [Le Gratiet, 2013][73]. Using these results from
the literature removes possible implementation bias in comparing the proposed
algorithm with existing algorithms. The multi-fidelity surrogates are used to approximate
the simulation a fluidized-bed process [78]. The quantity of interest for approximation is
the temperature of the steady-state thermodynamic operation point for a fluidized-bed
process. There are six variables affecting the quantity of interest: humidity ( rH ), room
temperature ( rT ), temperature of the air from the pump ( aT ), flow rate of the coating
solution ( fR ), pressure of atomized air ( aP ), and fluid velocity of the fluidization air ( fV ).
28 different process conditions were considered with coating solution used for distilled
water at room temperature. Four-fidelity outputs are available: expT , 3T , 2T , 1T . expT is
the experimental response denoted as high-fidelity data, 3T is the most accurate
simulation considering the adjustments for heat losses and inlet airflow. 2T provides
medium accuracy considering adjustment for heat losses. 1T has lowest accuracy
without adjustment for either heat losses or inlet airflow.
3.3.2 Comparison with the Bayesian MFS using Two-fidelity Dataset
We focus on the prediction of expT (high-fidelity) assisted by 2T (low-fidelity) using
two-level multi-fidelity surrogates as from in [Le Gratiet, 2013][73]. The dataset is
collected at 28 points with 6 design variables for both expT and 2T as visualized in
64
Figure 3-1. 2T is highly correlated withexpT having the correlation coefficient 0.99 and
2T overpredicts the value of expT .
Figure 3-1. Function values of the expT (high-fidelity) and 2T (low-fidelity) from Fluidized-
Bed Process
The evaluation is performed according to the plan proposed by [Le Gratiet,
2013][73, 79] as shown in Table 3-1. 20 samples were randomly selected from 2T as LF
samples as shown in Figure 3-2. Then 10 HF samples were randomly selected such
that the HF samples were nested into the LF samples. The left-out 18 points from expT
were used as test set ,T Tx y . Root-mean-square-error (RMSE) was adopted to
evaluate prediction accuracy as in Eq. (3-11). 100 different combinations of training
samples and test sets were produced for comparison. The model from [Kennedy and
O’Hagan, 2000] (KO) [61]was implemented using R CRAN package “approximator”, the
model from [Qian and Wu, 2008] (QW) [62] was implemented using the WinBugs
software, the model from [Le Gratiet, 2013] (LG)[73] was implemented using the R
CRAN package “MuFiCokriging”. Note that KO and LG are based on the same GP
models and their theoretical characteristics are identical. However, they may give
65
different results for the same problem because of the way of implementations and the
performance of their optimizers to estimate hyper parameters.
Table 3-1. Evaluation plan for the approximation of multi-fidelity dataset from Fluidized-Bed Process following [Le Gratiet, 2013]
Steps Procedures
1 Randomly selected 20/28 LF samples and then select 10/20 HF samples
2 The other 18 HF samples were used as test set
3 Generated 100 different combinations of design and test sets
4 Used root-mean-square-error (RMSE) as prediction metrics
Figure 3-2. Illustration of the resampling procedure to generate training and test data
18 2
( ) ( )
1
1 ˆ ( )18
j j
H T T
j
RMSE f
x y (3-11)
The median RMSE for the three Bayesian approaches are taken from [Le Gratiet,
2013] [73, 79] as shown in Figure 3-3. For the construction of LR-MFS, a linear
polynomial response surface (PRS) was fitted to the LF samples and the LR-MFS was
developed with a constant discrepancy. A linear PRS was developed based on only HF
samples to study the effect of introducing LF models. The LR-MFS was more accurate
than the PRS fit by introducing the LF samples. From Figure 3-3, it is clearly shown that
LR-MFS and LG were most accurate and significantly better than the KO and QW
models. The scale factors 𝜌 of LR-MFS was shown in Figure 3-3 (B) and varied
100
sets ⋮
66
between 0.73 and 1.15 which implied the similar trend between HF samples and LF
samples. The median of 𝜌 was 0.9 which was consistent with the observation that LF
model overpredicted HF model.
Figure 3-3. Performance of the surrogate models from 100 combinations of training
samples and test sets for the approximation of fluidized bed process. A) Median RMSE from PRS based on only HF samples, LR-MFS (LR), the model from [Le Gratiet, 2013] (LG), the model from [Kennedy and O’Hagan, 2000] (KO), the model from [Qian and Wu, 2008] (QW). The results for LG, KO, QW were taken from [Le Gratiet, 2013] (LG). B) Scale factors ρ from 100 sets of predictions for LR-MFS.
3.3.3 Effect of Noise for High-fidelity Dataset
High-fidelity samples might be obtained from noisy physical tests or simulations
with noise from discretization errors. Therefore, LR-MFS was further evaluated for HF
samples with perturbed with synthetic noise from a normal distribution 20,2N . The
noise level is comparable with the difference between HF and LF samples. The
evaluation was repeated based on Table 3-1 for the perturbed HF samples. One
realization of the HF samples with noise is shown in Figure 3-4. RMSE of LR-MFS to
approximate HF samples with noise was shown in Figure 3-5. It is seen that LR-MFS
A B
67
was robust with respect to noise. The median RMSE only increased from 2.1 to 2.2
while the standard deviation of the noise was 2.
Figure 3-4. Function values of the expT (high-fidelity) with synthetic noise from normal
distribution
Figure 3-5. RMSE of LS-MFS to approximate HF samples of fluidized bed process with
noise. The boxplots were based on 100 combinations of training samples and test sets. A) RMSE of LR-MFS for the dataset with or without noise and B) estimated noise level (STD).
3.3.4 LR-MFS with Multiple Low-fidelity Datasets
We compared the effect of different low-fidelity dataset using LR-MFS. expT was
the high-fidelity dataset for prediction. 3T , 2T and 1T were each used as the low-fidelity
dataset to build a two-fidelity LR-MFS prediction and then all the 3T , 2T and 1T were
used to build a four-fidelity LR-MFS prediction. The evaluation was performed according
B
LR-MFs no noise LR-MFS with noise
A
68
to Table 3-1 using 100 sets of design of experiments. The RMSEs of the LR-MFS are
shown in Figure 3-6. All the two-level LR-MFS had similar accuracy around 2. The
accuracy of LR-MFS improved with higher fidelity of iT . One interesting observation is
the four-level LR-MFS using all the iT had the worst accuracy and largest variation. The
scale factors 𝜌 are shown in Figure 3-7 for further investigation of the results. For the
two-fidelity LR-MFS, 𝜌 was about 0.9 with small variability. While for the four-fidelity LR-
MFS, all the 𝜌 were associated with significant variability and sometime negative. The
undesirable behavior of 𝜌 was due to the increasing number of parameters for surrogate
training. Considering the limited number of high-fidelity samples, more parameters
would suffer higher risk of overfitting. Regarding the median value, 3T had the highest
fidelity and largest 𝜌 which implied significant contribution to the LR-MFS prediction. For
2T and 1T , the 𝜌 was close to 0 and LR-MFS assigned larger weighs to the dataset with
higher fidelity.
Figure 3-6. RMSE of LR-MFS to approximate HF samples from fluidized bed process
with different low-fidelity dataset and all the low-fidelity dataset. The boxplots were based on 100 combinations of training samples and test sets.
69
Figure 3-7. Scale factors 𝝆 of LR-MFS to approximate HF samples from fluidized bed process with different low-fidelity dataset and all the low-fidelity dataset.
3.4 Approximation of the Modified Currin Function using LR-MFS
In this section, the proposed LR-MFS was evaluated using the modified Currin
function which has a known expression. The effect of different level of noise and
sampling plans were discussed.
3.4.1 The Modified Currin Function
The modified Currin function [53, 80] with two variables was adopted for further
investigation of the proposed LR-MFS approach. The high-fidelity model Hf x is given
in Eq. (3-12). We modified the original low-fidelity model [81] with a larger scale factor
and added a quadratic function as shown in Eq. (3-13). These variations make the
multi-fidelity modeling more challenging. Major settings of the test function are
summarized in Table 3-2. The function values of ( )Hf x vary between 1.2 and 13.8 with
a range 12.6. The ( )Hf x and Lf x are highly correlated with the correlation coefficient
0.99. The responses of Hf x without noise and Lf x are shown in Figure 3-8.
3 2
1 1 1
3 2
2 1 1 1
2300 1900 2092 601( ) 1 exp
2 100 500 4 20H
x x xf
x x x x
x (3-12)
70
1 2 1 2
1 2 1 2
2
1 2
1( ) [ ( 0.05, 0.05) ( 0.05,max 0, 0.05 )]
8
1[ ( 0.05, 0.05) ( 0.05,max 0, 0.05 )]
8
1( 5 7 )
8
Lf f x x f x x
f x x f x x
x x
x
(3-13)
Table 3-2. Major settings for the modified Currin test function
Input variables Range of Hf x Range of Lf x Correlation coefficient
between Hf x and
Lf x
1 2, [0,1]x x [1.1804, 13.7692] [-0.1867, 4.9498] 0.99
Figure 3-8. Responses of the modified Currin function in 2D space
We have evaluated multiple aspects of the LR-MFS as summarized in Table 3-3.
A PRS was fitted to only high-fidelity samples as the baseline to examine the accuracy
of LR-MFS while introducing the low-fidelity samples. Then the LR-MFS was compared
with the Co-Kriging from [Kennedy and O’Hagan, 2000][61]. The Co-Kriging model was
implemented through the ooDACE Toolbox [82]. The effect of sample size was
discussed by comparing the surrogates from 3 and 10 HF samples. Synthetic noise was
added to HF samples with levels of noise 𝑁(0, 0.12), 𝑁(0, 0.22) and 𝑁(0, 0.32). The
71
prediction accuracy was evaluated from RMSE at 100 100 test grid from Hf x . 100
sets of training samples with size 3 and size 10 respectively using LHS. For the MFS
models, the LF samples were obtained directly from Lf x instead of a surrogate to
avoid approximation error in building the low-fidelity surrogate. LF samples were
generated at the same location of HF samples and test grid. It is also possible to build a
ˆ ( )Lf x based on only low-fidelity data for repeated calls of LR-MFS in practical
applications.
Table 3-3. Key factors of the evaluation plan to approximate the modified Currin function
Key factors Plan
Effect of low-fidelity
samples
Compare LS-MFS with PRS fitted to only high-fidelity samples
Comparison with
Bayesian MFS
Compare LS-MFS with Co-Kriging from [Kennedy and O’Hagan,
2000]
Effect of noise for
HF
Add synthetic noise to HF samples from
𝑁(0, 0.12), 𝑁(0, 0.22), 𝑁(0, 0.32)
Effect of sample
size
Investigate surrogates for 3 HF samples and 10 HF samples
Prediction metric Root-mean-square error
Fitting error to only
LF samples
Low-fidelity sample was from 𝑓𝐿(𝐱), instead of a surrogate
Sampling plan Latin Hypercube Sampling, repeated 100 times with different
samples
3.4.2 The Effect of Introducing Low-fidelity Dataset
We first investigated the effect of introducing low-fidelity samples without noise.
For the approximation of 3 HF samples, a constant PRS and LR-MFS with constant
discrepancy were fitted. For the approximation of 10 HF samples, a quadratic PRS and
72
LR-MFS with quadratic discrepancy were fitted. Accuracy of the PRS and LR-MFS is
shown in Figure 3-9. 3 and 10 are the number of HF samples. The median RMSE is
based on 100 sets of samples. LR-MFS was more accurate than PRS by the order of
magnitude, both LR-MFS and PRS improved noticeably with more samples. The
median scale factors of LR-MFS increased from 1.74 to 1.95 with more samples which
indicated an enhanced contribution of low-fidelity dataset.
Figure 3-9. Effect of introducing low-fidelity samples for prediction of the modified Currin function without noise. PRS indicates the fit based on only HF samples. 3 and 10 are the number of HF samples. The median RMSE is based on 100 sets of samples. A) RMSE of surrogates and B) Scale factors of LR-MFS.
3.4.3 Comparison with Co-Kriging
Then the LR-MFS was compared with Co-Kriging without noise. It is seen in
Figure 3-10, LR-MFS was slightly better than Co-Kriging for the median RMSE and has
smaller variation. Co-Kriging had larger scale factors for the approximation of 3 HF
samples and similar scale factors for the approximation of 10 HF samples with LR-MFS.
The relative performance between LR-MFS and Co-Kriging was summarized in Table 3-
4. Among the 100 sets of HF samples, LR-MFS bested Co-Kriging for 55 cases and 63
cases while having 3 HF samples and 10 HF samples respectively.
A B
73
Figure 3-10. Comparison of Co-Kriging (CK) with LR-MFS for the approximation of the
modified Currin function. 3 and 10 are the number of HF samples. The box plot is based on 100 sets of samples. A) Median RMSE of surrogates and B) Median scale factors of MFS.
Table 3-4. Relative performance of LR-MFS and Co-Kriging for the approximation of the modified Currin function using 100 sets of HF samples
Number of HF samples 3 HF samples 10 HF samples
Cases bested by LR-MFS 55 63
Cases bested by Co-
Kriging
45 37
3.4.4 Fitting High-fidelity Dataset with Different Levels of Noise using LR-MFS
HF samples might be associated with significant experimental variation. The
synthetic noise was added to the HF samples to imitate experimental variation from
normal distributions 𝑁(0, 0.12), 𝑁(0, 0.22), 𝑁(0, 0.32). The accuracy of LR-MFS for HF
samples with noise was shown in Figure 3-11. The LR-MFS provided accurate
predictions for all the cases. The LR-MFS was expected to be robust with noise as
inherited from polynomial response surface. The LR-MFS3 didn’t change much with
increasing noise due to the large model-form uncertainty. In Figure 3-11 (B), we can see
that the low-fidelity samples contributed more on LR-MFS10 than that on LR-MFS3. The
estimated noise was computed according to Eq. (3-8) and summarized in Figure 3-12.
A B
74
For LR-MFS3, the estimated noise level was significantly contaminated by model-form
uncertainty. For LR-MFS10, the estimated noise level was close to the true value. The
sources of epistemic uncertainty for LR-MFS prediction were conceptually illustrated in
Figure 3-13. The contribution of uncertainties varied with conditions (e.g. number of
samples). The hazard of model form and noise effect decreased while introducing more
samples. The effect of model form was minuscule for LR-MFS10 and therefore the
estimated noise level was close to the true noise.
Figure 3-11. Comparison of LR-MFS with different level of noise for the modified Currin
function. 3 and 10 are the number of HF samples. The RMSE is based on 100 sets of samples. A) RMSE of the LR-MFS and B) ρ of the LR-MFS
A
B
75
Figure 3-12. Estimated noise level of LR-MFS with different number of HF samples for
the modified Currin function. 3 and 10 are the number of HF samples. The median STD is based on 100 sets of samples.
Figure 3-13. Conceptual illustration of the major epistemic uncertainty for LR-MFS with
different number of HF samples
3.5 Concluding Remarks
In this work, the linear regression multi-fidelity surrogate (LR-MFS) was proposed
to combine dataset with different fidelity, especially for high-fidelity with noise. The linear
regression is commonly used, balanced between accuracy, cost and simplicity. The LR-
MFS is less likely to overfit noise by limiting the number of parameters. Comparing with
the heuristic deterministic MFS, the LR-MFS is derived from standard linear regression
Model form Noise effect
Epis
tem
ic u
ncert
ain
ty
PRS10 LR-
MFS3
LR-MFS10 PRS
3
76
with analytical solutions to obtain parameters and therefore promising for theoretical
study. The treasure of linear regression could be applied directly to LR-MFS, such as
prediction variance, optimal DoE. The LR-MFS was demonstrated using polynomial
response surface (PRS) in this work. The proposed LR-MFS was first compared with
three Bayesian frameworks using a benchmark dataset from the simulations of a
fluidized-bed process. The accuracy of Bayesian frameworks varied significantly, and
the LR-MFS was comparable to the best GP-based approach. The effect of multiple
low-fidelity datasets was discussed. Introducing multiple low-fidelity models would suffer
a higher risk of overfitting for limited number of high-fidelity dataset. Then the LR-MFS
was evaluated to approximate a non-linear numerical test function with different
sampling plans. The LR-MFS bested Co-Kriging for 55%~63% cases with increasing
number of high-fidelity(HF) samples. For both examples, the LR-MFS proved to be
better than fitting only HF samples, and robust for HF samples with different level of
noise. LR-MFS could avoid the error for fitting LF data as demonstrated for the
numerical example. The LR-MFS heavily relies on the model form of the discrepancy
function. In the future, the selection of basis terms will be examined using existing
schemes such as stepwise regression.
77
CHAPTER 4 MULTI-FIDELITY SURROGATE MODELING FOR APPLICATION/ARCHITECTURE
CO-DESIGN
The HPC community has been using abstractions such as representative
applications (e.g., mini-apps, proxy-apps, skeleton apps) and architecture models to
enable faster co-design cycles. While abstraction developers often qualitatively verify
the correlation of the app abstractions to the actual (also called parent) application, it is
equally important to quantify this correlation to understand how the co-design results
translate to the parent application. In this work, we propose a multi-fidelity surrogate
(MFS) approach which combines data samples of low fidelity (LF) models (using
representative apps and architecture simulation) with a few samples of a high fidelity
(HF) model (using the parent app). The MFS fitted from a small number of
computationally expensive HF samples and larger number of cheaper LF samples, can
provide a more accurate prediction than using either type of data alone. In this work, we
demonstrate the application of MFS using a compressible fluid flow - application -
compressible multiphase turbulence (CMT)-nek (HF data) and its proxy-app, skeleton-
app, and simulation models (all LF data). We constructed a least-squares MFS based
on polynomial response surface (PRS) to mitigate the difference between the LF and
HF data. Our results show that the relative root-mean-square error (RMSE) between
predictions of MFS and the baseline HF models was 4%, which is significantly better
than 74% RMSE in using purely LF data alone, demonstrating that MFS is a promising
approach for predicting the parent application performance (HF model) using an MFS
while staying within a reasonable computational budget. The collection of dataset and
details of the experiments were kindly provided by Aravind Neelakantan, Nalini Kumar
and Herman Lam. This chapter on multi-fidelity surrogate is based on the publication [9]
78
co-authored with Aravind Neelakantan, Nalini Kumar, Herman Lam, Chanyoung Park,
Nam Ho Kim and Raphael T. Haftka.
4.1 Background
As we approach exascale computing, the next frontier in high-performance
computing, it is important that application developers and system designers co-design
to develop better performing and more energy efficient application codes and
machines[83]. For fast and effective turnaround during the co-design process,
application developers create representative applications which are abstract, smaller,
and self-contained descriptions of their application code (also called parent app) and
only capture the key parameters and features that predominantly influence the outcome
of co-design[84]. To further speed up the co-design process and to enable architecture
design-space exploration (DSE), system architects build simulator models to study the
application performance on various underlying architectures. Behavioral Emulation (BE)
[85] is one such coarse-grained approach for simulation of extreme-scale systems and
applications. While the parent application can be used to drive architecture simulations,
abstract application end-point models are often used to represent the parent app to
speed up the co-design process.
Representative applications, in the form of miniapps, proxyapps, or skeleton
apps, have been developed for many scientific HPC codes and are a necessity in cases
where the actual application cannot be shared with the hardware architects[86-89]. After
development, it is important to validate these representative apps against their parent
apps to ensure they are reasonably accurate representations of the application
behavior. Typically, this qualitative validation is performed by comparing ratio of
computation to communication, weak scaling and strong scaling trends, similarity
79
analysis, etc. Similarly, performance prediction results of architecture simulations are
verified against testbed measurements. After validation, both the representative apps
and simulator models are used as platforms to evaluate tradeoffs for improved
performance, power, and resilience, different programming models, compilers, etc. and
guide the refinement of parent application. Qualitative validation is important; however,
it is also important to determine quantitatively how the improvements in a representative
app or architecture translate to the parent app. To the best of our knowledge, no
solution for quantitative validation of representative applications has been proposed in
the literature.
In general, surrogate models are approximations that are fit to the available data
of a phenomenon of interest, herein the parent app. A high-fidelity surrogate model
(HFM) can be constructed from more accurate and computationally expensive high-
fidelity data (e.g., benchmarking data using parent app); and a low-fidelity model (LFM)
can be constructed from computationally cheaper but less accurate low-fidelity data
(e.g., skeleton apps, simulation results). In this work, we propose the use of a multi-
fidelity surrogate model (MFS) for identifying the relation between parent and
representative apps. These multi-fidelity models are constructed using a combination of
HFMs and LFMs.
The concept of MFS has been extensively studied to approximate the high-
fidelity models (HFMs) assisted by cheaper low-fidelity models (LFMs) [58]. To balance
accuracy and computational cost associated with data collection, the MFS approach
aims to develop a surrogate model based mostly on LF samples assisted with only a
80
few HF samples. Typical multi-fidelity models include finite element analysis with
different resolution, physical tests versus numerical simulations, etc. [11, 58, 90].
In this work, we leverage many of the MFS methods developed in other scientific
domains and adopt and apply them to reduce the computational cost of validation of
representative apps used in the HPC co-design process. After a survey of the related
research in Section 4.2, in Section 4.3 we present an overview of the parent application
case study (CMT-nek), its representative miniapp (CMT-bone) and skeleton app (CMT-
bone-BE), and the Behavioral Emulation approach that we use for performance
modeling and simulation. In Sections 4.4 and Section 4.5, we describe a methodology
for developing an MFS for an application from its mini-app (HFM), skeleton app (LFM),
and a simulator model (LFM). In Section 4.6, we demonstrate the usefulness of the
proposed methodology by applying it to a multi-physics simulation software being
developed for exascale systems - CMT-nek.The results show that the root-mean-
square error (RMSE) between predictions of MFS and the baseline HF models was 4%,
which is significantly better than 74% RMSE in using purely LF data alone,
demonstrating that MFS is a promising approach for predicting the parent application
performance (HF model) using an MFS while staying within a reasonable computational
budget.
4.2 Related Research
Miniapps have become extremely important for exascale DSE and performance
optimization. In [91], which presents a validation methodology, the authors state that
miniapps reduce the DSE time by a factor of a thousand, making them extremely useful
for exploring the design space of the parent application. Heroux, et al. in [92] provide a
81
verification and validation (V&V) methodology for assessing the ability of the miniapp to
effectively represent the performance of their parent application. The authors use the
difference between miniapp and parent app performance as their validation metric and
compare it against a threshold. This approach requires equal number of samples for
both the parent app and the miniapp. Since samples for the parent app are typically
more expensive to obtain, it can be a limiting factor in extensive validation studies over
a large design space. In our proposed approach, constructing an MFS requires much
fewer samples of parent app than of the miniapp, thus considerably reducing the
computational budget of conducting performance validation.
Miniapps are used as a tool to evaluate optimization methods to improve the
performance of the parent application. But improving the performance of the miniapp
does not guarantee the same for the parent app, making it important to know how
representative these miniapps are of their parent app [84]. For example, in [84] the
authors seek to improve the performance of the application on new and future systems
using miniapps. Although the optimizations applied improve the miniapp performance,
the impact on actual application performance is not clear. In our work, an MFS for the
parent application, built using high-fidelity application performance samples and lower-
fidelity miniapp performance samples, can help us draw a relationship between the
performance behavior of the two applications.
Several frameworks have been proposed to realize multi-fidelity modeling in
various science and engineering domains [58] [11, 90]. In [90], a Bayesian framework
has been applied to predict the data of nuclear radiation based on simulations. A
variable fidelity optimization framework has been demonstrated for the design of engine
82
piston[93]. A deterministic framework has been proposed to predict the strength of
composite laminate based on finite element simulations in[11]. A large number of
applications to mechanical systems can be found in[58], which reports that the MFS
reduces computational cost drastically while enabling desirable prediction accuracy.
The MFS has also been adopted as a powerful tool for uncertainty propagation[59].
Various MFS frameworks have been proposed for different engineering
applications. The existing frameworks are mainly different from two aspects: (1) the
form of algebraic function and (2) the scheme to determine MFS parameters. Various
MFS frameworks have been proposed to emphasize different aspects of these two
components. For example, the Bayesian MFS based on a scale factor has been applied
to design buildings[94] and flapping flight [95]. The Bayesian MFS incorporating
discrepancy function has been proposed [62, 90] as a popular MFS framework for
various applications. This Bayesian framework is equivalent to the co-Kriging surrogate
[73] with no prior information. Balabanov et al. [60] used a sequential deterministic MFS
based on the discrepancy function to combine finite element simulation with different
resolutions. Zhang et al.[11, 26] proposed a simultaneous deterministic MFS based on
the discrepancy function to combine experimental strength and finite element simulation
for composite laminate. We can apply this methodology in our HPC community to save
computational cost of parent application.
Sampling schemes for multi-fidelity models have been studied correspondingly.
HF samples are usually a subset of LF samples. One representative all-at-once
sampling strategy is the nested design sampling [96]. First, LF samples are generated
using Latin Hypercube Sampling (LHS). Then the HF samples are generated by
83
maximizing the minimum distance between all existing LF samples. Huang et al. [97]
proposed a sequential sampling scheme for design optimization using Bayesian MFS.
Either LF or HF samples are generated iteratively for design optimization. In our
approach, we used Full Factorial Design (FFD) for sampling as it is convenient for
parametric study. LF samples are available at the configurations of all the HF samples.
In this work, we leverage many of the MFS methods developed in other scientific
domains and adopt and apply them to the HPC domain to reduce the computational
cost of validation of representative apps used in the co-design process.
4.3 Application and Architecture Models
In this section, we give an overview of the parent application under study (CMT-
nek), its representative mini-app (CMT-bone) and skeleton app (CMT-bone-BE); and a
BE simulation approach that we use for performance modeling and simulation. The
relationships among their corresponding models are shown in Figure 4-1. The parent
app, CMT-nek, represents a high-fidelity (HF) model, whereas CMT-bone (a mini-app)
and CMT-bone-BE (a skeleton app) are low-fidelity (LF) models, as compared to CMT-
nek. BE simulation is a modeling and simulation of CMT-bone-BE. Thus, BE simulation
is an even lower fidelity model than CMT-bone-BE and CMT-bone.
In this study, our objective is to first perform validation and uncertainty estimation
of the BE simulation results against test samples of CMT-bone-BE (details in Section
4.3.2). Then a multi-fidelity surrogate model (MFS) is developed using mostly samples
from the low-fidelity BE simulation and a few high-fidelity CMT-nek samples. The MFS
model is then used to predict CMT-nek results (Section 4.6, case study 1). This
experiment is repeated between BE simulation (LF) with CMT-bone (now being an HF,
84
as compared to BE simulation) in case study 2, followed by CMT-nek (HF) and CMT-
bone (LF) for case study 3.
Figure 4-1. Hierarchy of the CMT models
4.3.1 CMT-nek
CMT-nek is being developed at the PSAAP-II Center for Compressible
Multiphase Turbulence (CCMT) at University of Florida to perform simulation of
instabilities, turbulence, and mixing in particulate-laden flows under conditions of
extreme pressure and temperature. CMT has applications in many environmental,
industrial, and national defense and security areas. CMT-nek is being developed from a
production release of petascale code Nek5000 [98], a Gordon Bell prize winning open-
source software for simulating unsteady incompressible fluid flow with thermal and
passive scalar transport. It is a highly scalable code with strong scaling to over a million
MPI ranks on ALCF BG/Q Mira. CMT-nek aims to take advantage of this sustained
performance by inheriting the MPI strategies used in Nek5000; and by hooking into the
Nek5000 repository, leveraging any changes and optimizations made to Nek5000.
85
4.3.2 CMT-bone and CMT-bone-BE
CMT-bone is a mini-app that encapsulates the key data structures and compute
and communication kernels of CMT-nek. While retaining the workflow of CMT-nek,
CMT-bone simplifies the number of variables defined and allocated and also the
number of computation and communication operations performed at each time step in
the simulation. Banerjee et al. and Kumar et al. in [86, 89] have validated mini-app
CMT-bone with its parent app CMT-nek and found that the key compute kernels are
well represented by the proxy application.
CMT-bone-BE is a skeleton app of CMT-nek created to support rapid algorithmic
design-space exploration. It models the computation that happens within every
simulation time step to calculate the partial derivative and exchange data between
nearby spectral element meshes. CMT-bone-BE ignores the initial problem setup
including mesh generation. Mesh generation operations can be abstracted and replaced
with a computation model in BE.
4.3.3 Behavioral Emulation (BE) Simulation
Behavioral Emulation (BE) is a coarse-grained modeling and simulation
approach that aims to provide timely, flexible, and scalable estimates of application
performance on existing and future system architectures. In BE, the complexity of large-
scale system simulation is handled by simultaneously dividing the simulation into
different levels of system abstraction (e.g., device, node, rack, system) and abstracting
the behavior of the components at each of these levels. The coarse-grained component
models mimic or emulate the observed execution behavior of the component instead of
its cycle-accurate operation. There are two basic types of BE models - application BE
objects (AppBEOs) and architecture BE objects (ArchBEOs).
86
BE simulations are used to predict the execution time of CMT-bone-BE and have
computational cost less than that of CMT-bone-BE (and much less than that of CMT-
nek and CMT-bone). In our study, BE simulation results are used to produce the LF
data points which are used to construct MFS models to predict the execution time of
CMT-nek and CMT-bone, using very few data points from these two applications, thus
reducing the overall computational budget of the DSE process.
4.4 Multi-fidelity Surrogates
In engineering applications, it is common to have multiple models with different
fidelities for solving the same problem such as finite element simulations with different
grid resolutions, numerical simulations, and physical experiments. A high-fidelity model
(HFM) represents the physical phenomenon more accurately than the low-fidelity model
(LFM) but it is often very expensive. An MFS based approach uses both high-fidelity
and low-fidelity datasets to approximate the HFM in the design space. An effective MFS
is expected to make accurate prediction with a limited budget for sampling. Fern andez-
Godino et al. [58] reviewed recent developments of MFS especially on effectiveness of
applying MFS to practical design. Peherstorfer et al. [59] summarized the technical
details of MFS for inference and uncertainty propagation.
MFS translates LFM against a few HF samples using an algebraic function.
Typical MFS frameworks include two major components: (1) a model to define the
relation between LFM and HFM, and (2) the scheme to find parameters of the MFS and
associated uncertainty of prediction. The LFM could be translated to HFM through (1) a
constant scale factor, or (2) scaling up the LFM and adding to a discrepancy function.
After determining the form of algebraic function, the MFS could be developed either
87
using Bayesian inference through Gaussian process, or using a least-square regression
minimizing error between fitted model and data.
In this work, we investigate the feasibility of MFS to quantify and mitigate the
difference between high-fidelity parent applications (e.g., CMT-nek) and low-fidelity
simulations (e.g., BE simulation) in the area of co-design of large-scale system. The
least-squares MFS (LS-MFS) (equivalent to the LR-MFS in section 3) [8] was selected
for this feasibility study while balancing complexity and predictive capability.
ˆ ˆ ˆ( ) ( ) ( )H L
f f x x x (4-1)
The LS-MFS is built with two surrogates, ˆ ( )Lf x , a polynomial response surface
(PRS) fitted to low-fidelity data, and ˆ( ) x , the fitted discrepancy data as in Eq. (4-1).
The scale factor and discrepancy function ˆ( ) x are obtained to minimize prediction
error at the high-fidelity samples according to Eq. (4-2) and Eq. (4-3). ( HX , Hy )
denotes the high-fidelity dataset containing n samples.
ˆ, ( )
ˆ ˆmin : ( ) ( )T
H H H H
x d x dx
(4-2)
ˆ ( )H L H H
f d x y (4-3)
The multi-fidelity surrogate using a single linear regression is obtained from Eq.
(4-4) to Eq.(4-8). Y is the vector of high-fidelity samples, X is the augmented design
matrix, B is the vector of unknown coefficients and e is the vector for residual errors.
Xi x denotes the thi monomial/basis, and
ib is the coefficient of X
ix . The obtained
discrepancy function ˆ( ) x is shown in Eq.(4-8).
Y XB e (4-4)
88
(1) (1)
1
1 1 1
( ) ( )
, ,
H H
n p n
n n
H Hp
b
b
y e
Y B e
y e
(4-5)
(1) (1) (1)
1
( 1)
( ) ( ) ( )
1
ˆ ( ) X X
ˆ ( ) X X
L H H p H
n p
n n n
L H H p H
f
f
x x x
X
x x x
(4-6)
1
T T
B X X X Y (4-7)
1
ˆ( )= Xp
i i
i
b
x x (4-8)
The LS-MFS scales up the LFM and adds a polynomial function ˆ( ) x to match a
few high-fidelity samples. The scale factor is critical for prediction. Negative or
extremely large values of indicates a prediction with large error, which is likely to be
associated with undesirable LFMs, inappropriate surrogate forms, or inadequate
samples. ˆ( ) x is supposed to be a low-order polynomial function while assuming the the
LFM has a trend similar to HFM. In our study, we adopted a constant ˆ( ) x for less than
10 HF samples and a linear polynomial function as ˆ( ) x for the rest. We approximated
the execution time in logarithmic coordinate to account for the order-of-magnitude
variation of execution time. ˆ ( )Lf x was developed using a quartic PRS.
4.5 Developing MFS Model
Although various performance metrics can be studied for performance simulation
such as energy consumption and communication times between the processors, the
metric of interest in this work is the total execution time for running a typical
computational fluid dynamics analysis using CMT-nek (HFM), CMT-bone (HFM), CMT-
89
bone BE (LFM), and BE simulation (LFM). All the benchmarking of the CMT models is
performed on the Vulcan HPC platform from Lawrence Livermore National Laboratory
(LLNL) [99]. Vulcan is a 24-rack IBM Blue Gene/Q system based on POWER
architecture that consists of 24,576 nodes and 400TB of compute memory. It is
important to ensure that the HF and LF data are obtained from the same hardware. The
accuracy of the MFS model reflects how representative the LF and HF are of each
other.
4.5.1 Design Space
For CMT-nek, the three main application parameters of concern are Element
Size (ES), Elements per Processor (EPP) and Number of Processors (NP). Application
performance can be affected by changing any one of these parameters. We chose 125
experimental points based on five-level full factorial design. “Five-level” denotes the 5
points/grids selected along each application parameter with similar space, as shown in
Figure 4-2. The design of experiment is ES= [5,9,13,17,21], EPP= [8,32,64,128,256]
and NP=[16,256,2048,16384,131072]. The experimental runs require up to 131,072
processors, 34 million elements and 311 billion computational grid points.
As the model fidelity increases so does the cost of obtaining a test sample. We
obtained data for CMT-bone-BE and BE simulation for the entire design space (125
data points); but for CMT-nek and CMT-bone, data was judiciously obtained from a
subset of the design space. For the runs from LFM and HFM, we made 22 runs from
CMT-nek (HFM), 67 runs from CMT-bone, and 125 runs for both LFMs (CMT-bone-BE
and BE simulation).
90
Figure 4-2. Design of experiments for CMT-nek, CMT-bone, and BE simulations
4.5.2 Validations of BE Simulation Results
Recall from Figure 4-1 that the order of fidelity is as follows: parent app CMT-nek
(highest), mini-app CMT-bone, skeleton app CMT-bone-BE, and BE simulation (lowest).
In the next section, we will use the BE-simulation results (LF) to predict the performance
of CMT-nek (HF parent app). Thus, first it is important to evaluate the accuracy of the
BE simulation. To do so, in this section, we will first validate the accuracy of BE
simulation against skeleton app CMT-bone-BE. We then evaluate the accuracy of CMT-
bone-BE by validating its results against those of mini-app CMT-bone.
We first examined the BE simulation versus CMT-bone-BE. Validation is the
process of comparing the BE simulation results to its respective benchmarking result
using CMT-bone-BE. In this study, we have validated the simulation results for the
entire design space on Vulcan, one of the largest high-performance computing system
available at the Lawrence Livermore National Lab. The validation of the design space
91
covered all the calibration points. But to further evaluate the accuracy of the simulator,
true validation was performed by validating points that are not present in the calibration
set. Polynomial interpolation was used in the simulator to predict the execution time of
the application at these validation points. The obtained simulation results are then
validated by running the actual CMT-bone-BE application on Vulcan for those validation
points.
The validation results of BE simulation against CMT-bone-BE are shown in
Figure 4-3. where the blue points represent the predicted CMT-bone-BE time using BE
simulation and the red points are the validation points obtained by running the
application (CMT-bone-BE) on Vulcan. We validated the simulations up to 128k NP on
Vulcan and predicted the time for 256k NP and 512k NP. The average percentage error
between BE simulation and CMT-bone-BE is 4 %, thus demonstrating the accuracy of
the BE simulator.
Figure 4-3. Validation of BE simulation against CMT-bone-BE
Then we compared CMT-bone-BE versus CMT-bone. The validation of the
skeleton app CMT-bone-BE against mini-app CMT-bone (Figure 4-4 (A) and Fig 4-4 (B),
respectively) is done through comparing their trends under the same experimental setup
92
described above. CMT-bone-BE being the skeleton app, takes less time to execute than
that of the mini-app, CMT-bone, and hence the range of their execution time varies.
Therefore, to make it easier to compare the trend, the execution time is plotted on a
color scale with red being the lowest in the range and blue being highest in the range as
shown in Figure 4-4. The step-wise increase shows that the predicted execution time for
CMT-bone-BE and CMT-bone increases monotonically with ES and EPP and does not
change appreciably with NP, and the color scale on the graphs verify the similarity in
trend between CMT-bone-BE and CMT-bone.
Figure 4-4. Comparing CMT-bone mini-app and CMT-bone-BE skeleton app trends for
various parameter values. A) CMT-bone and B) CMT-bone-BE.
4.6 Evaluating MFS Predictions --- Three Case Studies
Three case studies were used to demonstrate the multi-fidelity surrogate (MFS)
approach in which a surrogate model, based mostly on low-fidelity (LF) samples
assisted with only a few high-fidelity (HF) samples, is used to predict the performance of
a high-fidelity (HF).
• Case 1: Multi-fidelity model based mostly on BE simulation (LF) and few CMT-nek (HF parent app) data points to predict the performance of CMT-nek (HF)
• Case 2: Multi-fidelity model based mostly on BE simulation (LF) and few CMT-bone (relatively HF mini-app) data points to predict the performance of CMT-bone (HF)
A B
93
• Case 3: Multi-fidelity model based mostly on CMT-bone (relatively LF mini-app) and few CMT-nek (HF) data points to predict the performance of CMT-nek (HF)
The setup was same in all three case studies. A subset of high-fidelity data was
selected as the validation/test runs to evaluate predictions while the others were used
as training runs to train LS-MFS (least square MFS). The number of samples increased
gradually from the remaining runs (which excludes the validation runs) to investigate the
effect of sampling plan. For each number of samples, random selection was repeated
20 times to account for the effect of sampling plan. The overall difference was
measured using relative root-mean-square error (R-RMSE) between the LS-MFS
predictions and the validation runs. The relative error is defined in Eq. (4-9) where ˆ( )f x
is the surrogate prediction and ( )f x is the function value at test point. The R-RMSE is
defined in Eq. (4-10) where tN is the number of test points. The relative maximum
difference (R-MD) at the validation runs based on the repeated samples was also
provided to understand individual prediction. In this work, we study LS-MFS using
polynomial response surface as it is robust with noise effect. Other frameworks of multi-
fidelity surrogates are also available such as co-Kriging. The comparison between
different multi-fidelity surrogates is beyond the scope of this work.
ˆ ( ) ( )
( )( )
f ferr
f
x xx
x (4-9)
2
1
( )
R-RMSE
tN
i
i
t
err
N
x
(4-10)
4.6.1 Case Study 1: CMT-nek Predictions from BE Simulations
22 runs of CMT-nek are obtained as shown in Figure 4-2. 10 runs (out of 22)
were selected randomly and fixed as the validation runs. We first examined LS-MFS to
94
approximate CMT-nek (HF model) runs using a typical set of 12 samples as shown in
the Case 1 column of Table 4-1. The R-RMSE of LS-MFS is 4.49% using BE simulation
(LF model) at 10 validation runs. The linear fit using only HF CMT-nek runs was also
developed as a comparison with the R-RMSE to be 131.23% at the validation runs. The
R-RMSE between original BE simulation and CMT-nek ( ˆ ( )Lf x ) at all the 12 points,
without translation, is 66.85%. As mentioned before, BE simulation mimics CMT-bone-
BE and not CMT-nek. Since CMT-bone-BE is just a skeleton app with very few
computational kernels, the percentage difference is high. The LS-MFS was much more
accurate than either the ˆ ( )Lf x or the linear fit to ( HX , Hy ), demonstrating its promise to
compensate the difference between high-fidelity and low-fidelity models.
Table 4-1. Predicting execution times of CMT models based on typical set of 12 samples and evaluating the prediction using R-RMSE (%)
Surrogates Case 1: CMT-nek prediction from BE simulation
Case 2: CMT-bone prediction from BE simulation
Case 3: CMT-nek prediction from CMT-bone
Number of validation runs
10 20 10
LS-MFS 4.49% 5.40% 7.34%
ˆLf x 66.85% 61.26% 18.65%
Linear fit to
,H Hx y
131.23% 1901.91% 131.23%
Residual errors of
ˆLf x
0.77% 0.77% 1.05%
Residual errors of the linear fit to
,H Hx y
18.40% 50.71% 18.40%
Next, we investigated the effect of the sampling plan on prediction accuracy. The
LS-MFS predictions for CMT-nek runs with increasing number of samples were
summarized in Figure 4-5 (A). The LS-MFS was unstable using only 2 CMT-nek
95
samples due to over-fitting and became more accurate with increasing CMT-nek
samples. The R-RMSE was less than 10% with more than 9 CMT-nek samples and
ended with 4.49%. The R-MD was less than 20% with more than 9 CMT-nek samples
and ended with 7% as seen in Figure 4-5(B). The order of ˆ( ) x was changed from
constant to linear for more than 9 CMT-nek samples which was critical for the accuracy
of LS-MFS. We specified the order of ˆ( ) x for simplicity in this feasibility study. The
performance of LS-MFS could be improved by choosing appropriate ˆ( ) x . Another
observation is the large variation of R-RMSE while repeating HF samples. The design of
experiments for HF samples affected LS-MFS noticeably. The evaluation of LS-MFS for
CMT-nek is based on up to 12 samples and may suffer large uncertainty due to the
scarce runs.
Figure 4-5. Difference between CMT-nek validation runs and multi-fidelity predictions
based on CMT-bone simulations. A) Relative RMSE and B) Relative maximum difference.
A B
96
4.6.2 Case Study 2: CMT-bone Predictions from BE Simulations
20 runs (out of 67) were selected randomly and fixed as the validation runs. We
performed LS-MFS for CMT-bone (HF model in this case) based on 20 validation runs
and up to 47 samples. Again, LS-MFS was most accurate comparing to ˆ ( )L
f x and the
linear fit to only ( HX , Hy ) as shown in Case 2 column of Table 4-1. The LS-MFS
predictions were much closer to HFM than the original BE simulations.
Once again, we investigate the effect of the sampling plan on prediction accuracy.
The R-RMSE in Figure 4-6 (A) reduced with increasing CMT-bone samples and ended
with 5.4%. The R-MD in Figure 4-6 (B) oscillated with scarce CMT-bone runs at the
beginning and stabilized around 10%. Both R-RMSE and R-MD reduced noticeably with
the first few samples and stabilized to less than 10% thus proving to be a promising
approach.
Figure 4-6. Difference between CMT-bone validation runs and multi-fidelity predictions
based on BE simulation. A) Relative RMSE and B) Relative maximum difference.
A B
97
4.6.3 Case Study 3: CMT-nek Predictions from CMT-bone
In the final case study, the LS-MFS was developed to predict the high-fidelity
parent app (CMT-nek) from its low-fidelity mini-app (CMT-bone). This helps in
quantitative validation of the mini-app. The setup was same as in case study 1, where
CMT-nek was the HF model. From Case 3 column of Table 4-1, we see that LS-MFS
provides the best fit compared to linear fit. The LS-MFS had less-than 10% R-RMSE in
Figure 4-7 (A). It is worth noting the significant jump between 9 and 10 samples while
changing the order of ˆ( ) x in Figure 4-7 (A) and Figure 4-7 (B). CMT-bone was close to
CMT-nek and a different scheme might be preferred to determine ˆ( ) x .
Figure 4-7. Difference between CMT-nek validation runs and multi-fidelity predictions
based on CMT-bone. A) Relative RMSE and B) Relative maximum difference.
A key observation between case study 1 and 3 is that although the CMT-bone
samples were much closer to the CMT-nek samples, the MFS predictions of CMT-nek
(HF) from BE simulations (LF) were more accurate than the MFS predictions from CMT-
bone (LF) as shown in Table 4-1. Fitting CMT-bone was more challenging considering
the scarce samples (67 runs). BE simulations, on the other hand, had all the 125
A B
98
samples in the design space and thus, lead to better MFS predictions. This is supported
by residual errors of ˆ ( )L
f x from Table 4-1.
In all three cases, the error in prediction ended less than 10%; thus, proving it to
be a valuable approach to use for reducing computational budget in the process of co-
design. The range of scale factor for the three cases are summarized in Table 4-2.
The scale factors are around 1 which indicates that the LF and HF have similar trend.
The major difference between HF and LF are well-compensated by the constant/linear
discrepancy function.
Table 4-2. Range of the scale factors for LS-MFS
Case 1 Case 2 Case 3
Minimum 0.8 0.91 0.5
Maximum 0.98 1.08 1.3
4.7 Concluding Remarks
Due to high computational cost, validation samples from HFM are usually
obtained at small scale. But with MFS model, we were able to perform quantitative
validation at a reduced computational budget. In this work, we studied the least-square
MFS (LS-MFS) using polynomial response surface as it is robust with noise effect. For
future work, different multi-fidelity surrogates can be compared. Our ultimate goal is to
predict the performance for exascale computation platform which is essentially long-
range extrapolation far from the validation samples. In the future, we will investigate the
capability of LS-MFS for long-range extrapolation which suffers large uncertainty. We
also noticed the LS-MFS predictions were sensitive with the high-fidelity samples.
Effective design of experiments for validation runs are expected to improve the
accuracy of LS-MFS, which is also a valuable research direction.
99
CHAPTER 5 ON APPROACHES TO COMBINE EXPERIMENTS AND SIMULATIONS
Finite element analysis coupled with failure theory has been widely used for
predicting the failure of composite structures. However, significant discrepancies
between model predictions and experiments may be observed due to modeling errors
and material variability. The problem addressed in this work is how to interpolate or
extrapolate in a parametric design space using results from a limited number of
experiments with the aid of simulations.
In order to make most of simulation and experiments, it should be feasible to
correct model prediction using scattered experimental tests. This modeling procedure
combining both simulation and experiment can be viewed a special case of multi-fidelity
modeling, with experiments being the high-fidelity results and simulations as the lower
fidelity results. In this work, we considered two approaches for combining fidelities. One
approach is to seek an additive (offset) or multiplicative (scalar) correction to the low-
fidelity prediction (here simulation). A second approach is to use experiments to
calibrate a failure criterion used in the simulations.
We compare these two approaches in an example problem which consists of a
composite plate with a hole, which is a benchmark composite structure demonstrating
the effects of a stress concentration. The first approach fits a correction to failure
predictions based on a standard maximum strain failure criterion using a multi-fidelity
surrogate. The second approach calibrates the characteristic distance from the open
hole boundary of the Whitney-Nuismer failure criterion using experiments. These two
approaches are also compared to that of making predictions by fitting a surrogate to
100
experimental results without the aid of simulations. We focused on predicting mean
strength of composite structure.
We also examined the case of using a large number of data points from one
material system to complement small number of data from a second material system.
For this case, we compared fitting the small number of data from the second system to
fitting both using algebraic corrections. The collection of dataset and details of the
experiments were kindly provided by Jaco Schutte, John Meeker and Upul R.
Palliyaguru. This chapter on multi-fidelity surrogate modeling of composite laminate is
based on publication[10, 11] and report co-authored with Jaco Schutte, John Meeker
and Upul R. Palliyaguru, Nam Ho Kim and Raphael T. Haftka.
5.1 Background
Multi-fidelity surrogate models have been proposed to combine computer
simulations with different fidelities. We are mostly interested in this approach for
combining experiments with simulations. Experiments are deemed as high-fidelity
function and simulations are assumed to be low-fidelity function. Significant
discrepancies between model predictions and experiments may be observed due
modeling errors and material variability [100]. The multi-fidelity surrogates are
essentially approaches to correct low-fidelity function against high-fidelity function using
algebraic functions. Different theoretical frameworks are distinct mainly in two features:
form of calibration functions and optimization of calibration parameters [74, 90]. One
approach of multi-fidelity modeling is to seek an additive (offset) or multiplicative (scalar)
correction to the low-fidelity prediction (here simulation) [101, 102]. We adopt a scale
factor and discrepancy function as correction function. A surrogate is first developed
based on simulation. The simulation surrogate is then multiplied by a scale factor and
101
added with a discrepancy surrogate. In contrast to using the sophisticated Bayesian
framework to estimate the correction parameters, we adopted the linear regression
multi-fidelity surrogate (LR-MFS) for multi-fidelity modeling. We apply the framework for
estimating failure strength of composite laminates with assistance of finite element
simulation.
A second approach to incorporate experimental strength with simulation is to
calibrate a failure criterion tailored to the particular structural element by using
experiments [103]. We compare model correction and calibration for a composite plate
with a hole which is a benchmark composite structure [13, 104]. The first approach fits a
correction to failure predictions based on a standard maximum strain failure criterion
using a multi-fidelity surrogate framework. The second approach calibrates the
characteristic distance from the open hole boundary of the Whitney-Nuismer failure
criterion using experiments[105]. These two approaches are also compared to that of
making predictions by fitting experimental results without simulations
5.2 Open-Hole-Tension Tests
5.2.1 Experiments
The experiments were performed by Upul R. Palliyaguru from the Wichita State
University. The OHT test was designed to investigate the effect of an unfilled hole on
the tensile failure strength. The test specimen geometry is shown in Figure 5-1. The
experimental OHT strength was calculated based on the nominal area (i.e. gross cross-
sectional disregarding the hole). The width of the specimen and the diameter of the hole
are denoted by w and D, respectively. Specimens were made of three distinct material
batches.
102
Figure 5-1. Open-Hole Tension Test Specimen Configuration and observed failure
mode.
Two design parameters varied in the tests: w/D and layups. Table 5-1 details the
test matrix for different w/Ds. Table 5-2 listed the test matrix for different layups
quantified by the fraction of ±45° plies. Each structural configuration was composed of
three pre-preg batches, with each batch containing 6 replicates. Mean values of the
experimental strength were summarized in Table 5-3.
Table 5-1. w/D test matrix
w (in.) D (in.) w/D Batches/Replica
tes
No. of Specimens
0.75 0.250 3 3 x 6 18
1.00 0.250 4 3 x 6 18
1.50 0.250 6 3 x 6 18
2.00 0.250 8 3 x 6 18
Total = 72
Table 5-2. Layup test matrix
Layup % 0°/45°/90°
Ply Stacking Sequence Batches/Replicates No. of
Specimens
40/20/40 [0/90/0/90/45/-45/90/0/90/0]s
3 x 6 18
25/50/25 [(45/0/-45/90)2]s 3 x 6 18
10/80/10 [45/-45/90/45/-45/45/-45/0/45/-45]s
3 x 6 18
Table 5-3. Mean value of experimental strength of composite laminate made from
MTM45-1 PWC2 3K PW G30-500 Fabric (ksi) N45 =0.2 N45 =0.5 N45 =0.8
w/D=3 88.67 58.20 37.41 w/D=4 92.08 67.58 42.63 w/D=6 98.52 70.10 49.80 w/D=8 101.65 72.33 51.86
Failure at lateral gage middle
F F
D w
103
5.2.2 Finite Element Simulations
The finite simulations were produced by Jaco Schutte and John Meeker using
the inhouse tool at the Materials Sciences Corporation. The finite element solver
ABAQUS was utilized to predict the strains and stresses in the open hole tension
specimens. The open hole tensile coupon finite element model consisted of linear
continuum brick elements with a linear displacement controlled loading. The laminate
properties were defined by representative volume elements (RVE) assembled of three
sub-cells. Mechanical properties of the laminia MTM45-1 PWC2 3K PW G30-500 Fabric
could be found from [11]. The sub-cells are a transversely isotropic representation of the
resin impregnated fiber bundles, in the form of a set of concentric cylinders, where the
center cylinder consists of the fiber material and the surrounding cylinder the resin
material. Two of the three sub-cells represent each of the orthogonal directions of the
resin impregnated fibers in the fabric and the third sub-cell is a pure cylinder of resin.
Failure was modeled by maximum strain failure criterion applied to the fiber
directions of each sub-cell. The net strains in each sub-cell were calculated from the
state of strain of each element of the global analysis. Failure of the specimen is
assumed when one of the sub-cells exceeds a maximum stain in the fiber direction,
determined with unidirectional test coupons.
Figure 5-2 depicts the mesh of the FE model, including a magnified mesh around
the hole. Critical points of failure were observed at the hole edge. The simulations were
conducted at 221 different combinations of (w/D, N45) uniformly spaced, where w/D
varied between 2 to 10 with a step size of 0.5, and the fraction of 45° plies, N45 varied
between 0.2 to 0.8 with a step size of 0.05. Mean value of experimental strength and
simulation results are shown in Figure 5-3. The strength observed from experiments
104
and simulations are both monotonic in both variables and slightly curved over the
design space.
Figure 5-2. Mesh of finite element model for OHT test. A) Overall mesh of the whole
plate and B) zoomed mesh near the hole.
Figure 5-3. Illustration of experimental results and simulations for OHT tests
5.3 Approaches to Predict Mean Strength of Untested Points Assisted by Simulations
We have examined two approaches to predict experimental strength at untested
points assisted by simulations: model calibration and algebraic correction. In addition, a
surrogate model has been fitted to experiments alone in order to check whether the
A B
105
simulations provide improved accuracy. All three approaches used polynomial response
surface as the surrogate due to its robustness with noisy data.
5.3.1 Model Calibration
The tension and compression strength of composite laminates with holes drops
greatly with increasing hole diameters. This drop cannot be fully predicted with the use
of a classical stress concentration factor. Whitney and Nuismer [105]assumed that
failure of an open hole in an infinite plate under tension or compression can be
determined when the stress at some characteristic distance away from the edge of the
hole reaches the un-notched strength of the material system. Figure 5-4 illustrates the
assumption where an infinite plate containing a hole with radius r is placed under a
uniform tensile stress and y is defined as the stress concentration at a distance x from
the hole. The Figure 5-4 shows a point at distance x from the hole where the stress is
the un-notched strength of the material system 0 .
Figure 5-4. Failure criteria for a circular through thickness holes in an infinite plate
106
The Point Stress Criterion (PSC) [105] gives the ratio of the laminate un-notched
strength 0 to the open hole strength N as function of the radius (r) and the
characteristic distance 0d as shown in Eq. (5-1). The distance 0d can be calibrated by
experiments for a given material system. The un-notched failure strengths 0 for each of
the material systems were obtained with a finite element analysis of the composite
laminate without a hole using max strain failure.
2 4
0 0
2,
2 3
N r
r d
(5-1)
We combined experiments and simulations by fitting the Whitney-Nuismer
characteristic distance as function of 45( / , )w D N . At untested points, N is the quantity
of interest, we can obtain 0 based on finite element analysis of un-notched laminate at
a given 45N . At each of the experimental points the notched strength is available from
mean value of replicates and 0d can be computed using Eq. (5-1). A polynomial
response surface (PRS) was fitted to the characteristic length. Estimated failure
strength was obtained by combining un-notched strength from finite element analysis
with predicted characteristic length.
5.3.2 Algebraic Correction (Multi-fidelity Surrogate)
Simulations at untested points may allow us to improve accuracy of the
prediction. Here we use failure predictions from FEA based on strains at the edge of the
hole that substantially under-predicted the failure strength. We adopt a multi-fidelity
surrogate (MFS) for correcting the predictions. The quantity of interest for prediction is
the mean value of experimental strength. MFS provides a fit combining a small number
of expensive high-fidelity data (here experiments) and a large number of less expensive
107
low fidelity data (here simulations) as function of a design parameter vector x. The high
fidelity data (experimental mean strength) is denoted as ( Hx , Hf ), while the low fidelity
data is ( Lx , Lf ), where Hx is a subset of Lx .
In the simple framework [8] (equivalent to the LR-MFS in section 3), the MFS is
described with two surrogates ˆ ( )Lf x and ˆ( ) x which represent a fit to the low fidelity
data and a discrepancy function or correction, respectively,
ˆ ˆ ˆ( ) ( ) ( ) ( )H Lf f x x x x (5-2)
where ρ is a scalar and ( ) x denotes random noise due to experimental variability and
numerical instability. ρ and ˆ( ) x are obtained simultaneously through optimization as
shown in Eq. (5-3) and Eq. (5-4).
ˆ, ( )
ˆ ˆmin : ( ) ( )T
H H H H
x
x d x d (5-3)
ˆ ( )H L H Hf d x f (5-4)
Polynomial fits are used for surrogates, with the order for the low-fidelity model
being selected using leave-one-out cross validation based on ˆ ( )Hf x at experimental
points. Large scale factor indicates significant contribution of ˆ ( )Lf x in the multi-fidelity
model.
5.4 Results for Combining Simulations and Experiments
We first selected a subset of experimental strength results, shown in Figure 5-5,
for modeling. The other points were used to check the accuracy of the predictions of the
three approaches. We first examined root-mean-square error (RMSE) at untested points
as shown in Table 5-4. For multi-fidelity modeling, all the finite element simulations were
used to develop low-fidelity model.
108
Figure 5-5. Selected points from experimental strength (high-fidelity function) used to predict strength at other points
Table 5-4. Relative error and root mean square error at untested points based on 6 samples
w/D, N45 3, 0.5 4, 0.5 8, 0.5 3, 0.8 4, 0.8 8, 0.8 RMSE
Experiment only 11.6% 0.06% 7.6% 9.1% 1.8% 3.5% 7.0%
MFS with rho=0.5 14.1% 3.5% 9.8% 4.7% 0.65% 0.35% 7.4%
WN with quartic PRS 9.2% 2.2% 0.19% 19.3% 9.7% 1.2% 9.6%
It is seen that expf had the best accuracy, followed by WNf and then by MFSf based
on RMSE. All the three models had reasonable accuracy because of the almost linear
variation of the strength that made six data points sufficient for good fit. Introducing
simulation in this case resulted in some decrease in prediction accuracy. We concluded
that expf should be first choice when enough data is available. MFSf is preferred when the
trends between high- and low- fidelity functions are similar. WNf introduces a physical
model to simplify the quantity of interest for approximation. We will examine the effect
of cross-validation to rank these three surrogate models.
We then further investigated different forms of model correction. Various subsets
of experimental strength were selected for modeling. Root-mean-square error (RMSE)
109
were computed using the mean value at all the experimental points to reflect overall
accuracy. For multi-fidelity modeling, all the finite element simulations were used to
develop low-fidelity model. We examined two partial versions of the multi-fidelity model
as shown in Eq. (5-5) and Eq. (5-6) besides MFSf in Eq. (5-2). They represent the use of
only multiplicative correction or only additive corrections, and were needed when the
number of data from the new system is too low to permit both additive and multiplicative
corrections. The parameters and ˆ( ) x were determined similarly as that from MFSf .
ˆ ˆ( ) ( )Lf f x x (5-5)
disˆ ˆ ˆ( ) ( ) ( )Lf f x x x (5-6)
Figure 5-6. Performance of different correction models for predicting OHT strength using FEA. Experiments were performed based on composite laminate made from MTM45-1 PWC2 3K PW G30-500 Fabric
Mean RMSE of 100 sets of prediction are compared in Figure 5-6. The corrected
models are all better than the FEA simulations. It is seen that ˆ ( )f x could avoid large
error when fewer than 3 samples are available. ˆ ( )f x dominated disˆ ( )f x with varying
110
number of samples. expf and MFSf had similar accuracy when more than 3 samples
were available.
The conclusion from this analysis, is that the response of OHT tests is close to
linear and therefore surrogates fitted to experiments could provide reasonable accuracy
with three or more experimental data points. However, simulations combined with model
correction is needed when only one or two experiments are available.
5.5 Using Tests from One Material System to Predict Strength of Another Material System
Next we compared predicting the OHT strength of a target material system (high-
fidelity system) based on available experimental data on a similar material system (low-
fidelity function). Our goal was to see whether large number of data points from one
system can be used to help improve predictions when only a small amount of data is
available from another material system.
The available 12 points of OHT tests were used to develop a low-fidelity model.
Then we used 6 points from the literature for a different material system as high-fidelity
data shown in Table 5-5. All the OHT tests are illustrated in Figure 5-7. As we wanted to
explore the effect of the number of points from the new system, we considered using
only subsets of the available 6 points. To get meaningful conclusions, we randomly
selected a subset of samples from high-fidelity data for modeling, and repeated the
process 100 times.
Table 5-5. Experimental strength of composite laminate from T700SC-12K-50C/#2510 Plain Weave Fabric (ksi).
45/ ,w D N (3, 0.2) (4, 0.2) (6, 0.2) (8, 0.2) (6, 0.5) (6, 0.8)
Strength 59.6 63.2 66.5 69.3 54.5 41.1
111
Figure 5-7. Experimental strength of OHT tests from two different material systems
Figure 5-8. Performance of different correction models for predicting OHT strength using FEA or another material. Experiments for prediction were performed based on composite laminate made from T700SC-12K-50C/#2510 Plain Weave Fabric
Mean RMSE of 100 sets of prediction are compared in Figure 5-8. Similar to
Figure 5-6, ˆ ( )f x dominated disˆ ( )f x with varying number of samples. expf and MFSf had
similar accuracy when more than 3 samples were available. One interesting observation
112
is that all the corrected models are more accurate based on FEA simulations than those
based on another material.
5.6 Predicting B-basis Allowables of Experimental Strength Assisted by Aimulations
We investigate the prediction of B-basis allowables using surrogate in this
section. The B-basis allowable is widely used as conservative estimation of composite
strength. The various methods for calculating B-basis allowable from replicate tests of a
single structural configuration are well documented. Combining such calculations from
several configurations in order to predict B-basis allowables at design configurations
that lack supporting validation tests is less well explored. Methods for improving B-basis
predictions at untested points by adding simulations (e.g. based on finite element
models) have received almost no coverage in the literature. This section explores the
options of predicting B-basis allowable with surrogate that is built on using experiments
and simulations.
We first explore two options. The first approach is regression tolerance limits,
which uses all replicates in the experiments to develop surrogates for mean values and
corresponding B-basis allowables. This implementation has been derived strictly for
independent and identically distributed (i.i.d.) normal random variables. The second
approach is direct regression, which calculates the B-basis at each test point by existing
single-point estimation and uses only this data to develop surrogates. Different single-
point estimation schemes of B-basis allowable could be adopted based on distributions.
Single-point estimation refers to the calculation based on replicates from a specific
configuration. The regression tolerance limits had noticeable bias for modeling the OHT
tests because the replicates do not satisfy the requirement of i.i.d. normal random
113
variable. In contrast, the direct regression was more flexible to associate with the
selected single-point estimation.
Then the direct regression approach was studied further by incorporating finite
element simulation. We combined simulations with experiments using multi-fidelity
surrogate. A few experiments were selected to correct simulations and predicative
capability of corrected simulation was examined at all the experiments. Fitting
experiments alone is preferable over multi-fidelity surrogate for the OHT tests except
when very few (1 and 2) experiments are available. We also found significant errors for
long-distance extrapolation.
5.6.1 Background
Composite materials have been extensively used for industrial design in past
decades. But large challenges remain for simulating strength of composite laminates
with complicated geometries or progressive damage. Strength of composite laminate is
usually associated with large variability, which makes the simulation even more
challenging. Therefore, substantial testing has to be conducted to measure variability
and estimate the resulting conservative allowables, such as B-basis, empirically. The B-
basis is a value that bounds the true (or population) 10th percentile value with 95%
confidence as shown in Figure 5-9. Computing B-basis allowables for fixed
configuration based on replicated tests have been explored, and many are listed in
Composite Materials Handbook 17. Bhachu et al. [106] compared several common
approaches for fatigue crack growth problems. Romero et al. [107, 108] tested the
performance of normal TI method, Pradlwarter-Schuëller kernel density method,
Johnson method, and non-parametric method.
114
Figure 5-9. Illustration of B-basis allowables based on replicates/repeated tests
When experiments are available for several structural configurations, it may be
possible to estimate B-basis allowable at untested points by interpolation or
extrapolation via a surrogate fit to the available data. Surrogate models, such as
polynomial response surfaces or kriging, are routinely used for approximating system
response as a cheap alternative of expensive simulations or tests [7, 30, 109, 110].
They have been used for fitting experimental data for mean strength [13, 77].
We compared two approaches for predicting B-basis allowable. The first
approach employed all the replicated tests and predicts B-basis allowable at untested
points based on rigorous derivation while assuming the experiments are independent
and identically distributed (i.i.d.) normal random variables. The second approach is
direct modeling of the B-basis allowable at each test point obtained from single-point
estimation and use only this data to develop surrogates. Different single-point
estimation schemes of B-basis allowable, which are estimations based on replicates at
a specific configuration, could be adopted based on distributions for direct modeling.
When it comes to predicting mean strength at untested points, it is common to
use simulations that combine finite element analysis (FEA) and failure theories,
Material
properties
Specimen configuration
Estimated 10th percentile
B-basis allowable: lower 95% C.I.
Tested properties with replicates
115
calibrated or corrected by experiments [103, 111]. To combine simulations and
experiments in surrogate it is common to use multi-fidelity surrogate model [90, 101,
112]. However, this is rarely done for predicting variability or conservative allowable at
untested points.
5.6.2 Predicting B-basis Allowable at Untested Points using Surrogates
5.6.2.1 Linear regression tolerance intervals
A polynomial response surface (PRS) can be fitted when test results are
available at several points, there is clear trend of polynomial dependence with a design
parameter and the standard deviation or coefficient or variation are approximately
constant [113]. PRS-based tolerance method calculates one-sided regression tolerance
limits with confidence level (1-𝛼) for a proportion P of the future replicates are within
certain limits (tolerance limist), and 𝛼=0.05, P=0.9 for B-basis. The details for calculating
B-basis allowables are given in Eq. (5-7) to Eq. (5-9) while assuming that strength
follow independent identical normal distribution.
1,ˆ ˆ( , )i i i iB basis x y y k (5-7)
* * *
;1
1,*
=n p i p
i
i
t n zk
n
(5-8)
2*
2
ˆ=
ˆ. .i
i
ns e y
(5-9)
where 𝑖 denotes the index of untested point (𝑥𝑖, 𝑦𝑖) for prediction, ��𝑖 is the PRS
prediction of the mean value at (𝑥𝑖, 𝑦𝑖), �� is the estimated noise level, calculated by the
root mean square error between PRS predictions and samples. 𝑡𝑛−𝑝;1−𝛼∗ (𝑥) is the (1-𝛼)
th quantile of a non-central t distribution with t degree of freedom and non-centrality
parameter 𝑥. n is sample size, in Eq. (5-8). 𝑛𝑖∗ is called the “effective number of
116
observations. 𝑧𝑝∗ is P-th quantile of standard normal distribution. ˆ. . is e y is the standard
error of PRS at ��𝑖, details for the derivation of PRS could be referred at [114].
5.6.2.2 B-basis allowables of Open-Hole-Tension tests
The statistics of OHT tests are provided in Table 5-6. The standard deviation
(Std) varies from 0.85 ksi to 5.42 ksi, and the coefficient of variation (CV) varies from
0.020 to 0.059. The large variations of STD and CV are against the assumption for
identical distribution. We then examined the distributions of OHT tests. Typical
histograms for OHT tests based on 18 replicates are shown in Figure 5-10. No clear
pattern could be observed for the OHT tests which was also supported by Kolmogorov–
Smirnov test (KS test).
Table 5-6. Statistics of OHT tests at the 12 experimental locations No. 1 2 3 4 5 6 7 8 9 10 11 12
R45 0.2 0.2 0.2 0.2 0.5 0.8 0.5 0.5 0.5 0.8 0.8 0.8
w/D 3 4 6 8 6 6 3 4 8 3 4 8
Mean(ksi) 87.35 91.46 97.44 100.12 71.07 50.30 58.11 67.62 72.39 37.68 42.68 51.84
Std(ksi) 3.55 5.42 5.20 4.90 2.84 1.68 2.48 2.70 3.28 1.34 0.85 1.33
CV 0.041 0.059 0.053 0.049 0.040 0.033 0.043 0.040 0.045 0.036 0.020 0.026
Figure 5-10. Typical histograms (out of 12 locations) based on 18 replicates from OHT
tests. Horizontal axis is the strength of the plate in ksi and vertical axis is the number of replicates.
Consider no clear pattern for the distributions, nonparametric approach was
adopted to compute the B-basis allowables (𝐵𝑛𝑜𝑛𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐) [115]. If the sample does not
117
demonstrate clear distribution pattern (e.g. due to limited number of samples), the
calculation can be performed according to order statistics. The strength of samples is
first ranked according to the order of magnitude. 𝑥(𝑟) denotes 𝑟𝑡ℎ largest data. 𝑥(1)
denotes the smallest one. B-basis could be computed based on Eq. (5-10) where k is a
factor depending on sample size. B-basis allowables for the OHT tests are calculated in
Table 5-7.
(1)
( )
( )
k
r
r
xB x
x
(5-10)
Table 5-7. B-basis allowables of OHT tests at the 12 experimental locations using non-parametric approach
No. 1 2 3 4 5 6 7 8 9 10 11 12
R45 0.2 0.2 0.2 0.2 0.5 0.8 0.5 0.5 0.5 0.8 0.8 0.8
w/D 3 4 6 8 6 6 3 4 8 3 4 8
Bnonpara
(ksi)
80.30 81.93 89.79 94.47 64.58 45.28 47.09 58.80 63.95 33.54 38.06 45.89
5.6.2.3 Predicting B-basis allowable of Open-Hole-Tension tests using surrogate
The non-identical and non-normal behaviors of replicates may lead to
unexpected predictions using PRS tolerance. But PRS were likely to provide reasonable
approximation considering the response of OHT tests were monotonic and close-to-
linear in the variable space. Therefore, we considered to estimate B-basis allowables at
single point first and then approximate B-basis allowables directly using PRS (Direct
PRS) in the variable space. PRS tolerance and Direct PRS were compared using the
robust test plan. We don’t have the true 10th percentile and distributions for the OHTS
tests. Therefore, 𝐵𝑁𝑜𝑛𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 were adopted as the quantities of interest for prediction.
The root-mean-square error here is essentially to illustrate the difference between Direct
PRS and PRS tolerance rather than accuracy evaluation. Relative RMSE for PRS
tolerance and direct PRS are given in Figure 5-11. Constant PRS was used for 1-2
118
experiments, linear PRS was used for 3-5 experiments and quadratic PRS was used for
more than 6 experiments. It is seen both approaches had significant error based on 1-2
experiments. The prediction accuracy increased drastically after using linear PRS and
gradually improved by introducing more experiments. The Direct PRS was always
closer to 𝐵𝑁𝑜𝑛𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 than PRS tolerance. Direct PRS could also be associated with
other point-estimation besides nonparametric approach for robust estimation of B-basis
allowables beyond i.i.d. normal distribution.
Figure 5-11. Overall accuracy (relative RMSE) for PRS tolerance and direct PRS with increasing number of experimental tests
5.6.3 Predicting B-basis Allowable using Multi-fidelity Surrogate
This section discussed predicting B-basis allowable using direct PRS assisted by
finite element simulations. Then the effect of sampling strategy was elaborated for multi-
fidelity modeling.
5.6.3.1 Simple framework of multi-fidelity modeling
Simulations at untested points may allow us to improve accuracy of the
prediction. Here we use failure predictions from finite element simulations based on
strains at the edge of the hole that substantially under-predicted the failure strength. We
adopt a simple framework of multi-fidelity surrogate (MFS) for correcting the finite
119
element simulations[11] (equivalent to the LR-MFS in section 3). MFS provides a fit
combining a small number of expensive high-fidelity data (here experiments) and a
large number of less expensive low fidelity data (here simulations) as function of a
design parameter vector x. The high fidelity data (B-basis allowables) is denoted as (
Hx , Hf ), while the low fidelity data is ( Lx , Lf ).
In the simple framework, the MFS is described with two surrogates ˆ ( )L
f x and ˆ( ) x
which represent a PRS fit to the low fidelity data and a discrepancy function or
correction, respectively,
ˆ ˆ ˆ( ) ( ) ( ) ( )H L
f f x x x x (5-11)
where ρ is a scalar and ( ) x denotes random noise due to experimental variability and
numerical instability. ρ and ˆ( ) x are obtained simultaneously through optimization as
shown in Eq. (5-12) and Eq. (5-13).
ˆ, ( )
ˆ ˆmin : ( ) ( )T
H H H H
x d x dx
(5-12)
ˆ ( )H L H H
f d x f (5-13)
We adopted a cubic PRS as ˆ ( )L
f x to fit all the simulations in the variable space.
For ˆ( ) x , constant PRS was used for 1-2 experiments, linear PRS was used for 3-5
experiments and quadratic PRS was used for more than 6 experiments.
5.6.3.2 Predicting B-basis allowable of using multi-fidelity surrogate
Finite element simulations were introduced to assist estimation of B-basis
allowables at untested point using direct PRS. The high-fidelity samples were B-basis
allowables at selected experiments estimated using nonparametric approach. Low-
fidelity samples were the finite element simulations. The MFS predictions were
120
evaluated using robust test plan and compared with direct PRS as shown in Figure 5-
12. MFS predictions dominated direct PRS for 1-2 experiment and less desirable for the
rest.
Figure 5-12. Overall accuracy (relative RMSE) for MFS and direct PRS with increasing number of experimental tests
5.6.3.3 Risky sampling strategy
The sampling strategy is critical for predictive capability of surrogates. Samples
corresponding to the worst three relative RMSEs using MFS are shown in Figure 5-13.
The selected samples were located on the borders and most domain suffered from
long-distance extrapolation and lead to significant errors.
Figure 5-13. Samples corresponding to the worst three sets of predictions using MFS. Solid points indicate selected experiments out of 12 for MFS.
121
5.7 Concluding Remarks
In this work, we investigated the approaches for predicting experimental strength
of composite laminate using multi-fidelity surrogate model. Open-Hole-Tension tests
with two design variables were adopted for the benchmark study. We first examined the
prediction of mean experimental strength assisted using either simulation or historical
data from another material system. Then the multi-fidelity surrogate was used to
prediction B-basis allowables of composite laminate at untested points. The classical
regression tolerance intervals was intended for independent and identically distributed
(i.i.d.) normal variables. The replicates of OHT tests didn’t follow normal distributions
and had different level of variations. The violation of i.i.d. assumptions may lead to large
prediction error. We were motivated to develop PRS based on estimated B-basis
allowables using nonparametric approach (Direct PRS). The Direct PRS was flexible to
associate with various single-point estimation and proved to be more flexible than
regression tolerance intervals while estimating the nonparametric B-basis allowables.
Then we examined the prediction of B-basis allowables using multi-fidelity
surrogate (MFS). MFS predictions were much more accurate than direct PRS for 1-2
experiments and less desirable for the rest. We also found significant errors for long-
distance extrapolation. Hope this effort could guide the prediction for B-basis allowables
of composite laminates.
122
CHAPTER 6 A STRATEGY FOR ADAPTIVE SAMPLING WHEN SAMPLING COST IS VARIABLE IN
DESIGN SPACE
6.1 Background
Design optimization of mechanical systems usually requires extensive
simulations and tests to achieve desirable performance. Surrogate models are often
introduced as an efficient tool to approximate the response of mechanical systems from
scattered simulations/tests in the input variable space. An effective surrogate model
enables a cheap prediction of mechanical responses at a given design
configuration/input. Design optimization based on the surrogate model has been
adopted for numerous mechanical systems with significantly reduced simulation costs
or test period [30, 116]. Besides design optimization, surrogate models have been
applied for various engineering analysis. For example, advanced materials or innovative
designs are emerging which might lack effective theoretical models such as lattice
structure and composite material [11]. Surrogate models may serve as an empirical
model for the fundamental mechanics to enable multi-scale analysis. Surrogates model
is also the key technology for verification, validation and uncertainty quantification such
as the multi-fidelity surrogates which is developed to compensate for the discrepancy
between experiments and simulations [26, 58].
The accuracy of surrogate models strongly depends on the location and number
of samples (experiments and simulations). Methods to select samples are termed
sampling plan or design of experiment (DOE). Desirable DOE enables accurate
surrogate models while balancing the testing budgets. Adaptive sampling is a popular
DOE, which adds samples iteratively with updated surrogates. Various adaptive
123
sampling strategies were developed and proved effective for the approximation of
system response.
One major component of adaptive sampling is the uncertainty metric. Two types
of uncertainty metrics are mainly used for adaptive sampling: model-based prediction
variance and data-driven prediction variance. The prediction error at a point could be
estimated by prediction variance. The prediction variance quantifies the variation of the
potential system response, which is assumed a statistical distribution. The points with
large prediction variance imply risky approximation and could be improved by adding
more samples. The commonly used prediction variance is usually available for a few
types of surrogates by inherently assuming a distribution for the samples. For example,
the Gaussian process [49] assumes a multivariate normal distribution for the samples
and polynomial response surface assumes a normal distribution for the residual
errors[117]. Recent effort has been made to obtain the uncertainty metric for an
arbitrary surrogate based on data-driven approach such as cross-validation (CV). Jin et
al. proposed using the difference between the surrogates from leaving-one-out CV and
the surrogate using full samples to estimate the uncertainty of predictions [118]. Salem
et al. proposed the development of a series of surrogates from leaving-one-out CV.
Then a weighted scheme was used to fit the empirical cumulative distribution function
(ECDF) from the predictions of the surrogates. The prediction variance is then
generated from the ECDF to indicate the estimated uncertainty [119]. Xu et al.
associated surrogate prediction with Voronoi Diagram to determine the uncertainty of
surrogate predictions [120].
124
The current practice of adaptive sampling implicitly assumes that all samples
have the same cost, which is applicable for most applications. However, in some
applications, the cost to obtain samples may substantially vary in the input variable
space. For example, the computational resource of a computational fluid
dynamics(CFD) analysis relies heavily on the mesh density. The mesh density of CFD
analysis changes with Reynold number and Mach number. The effect of sampling cost
might be significant in the high-performance computing (HPC) scenarios. The HPC
architecture and algorithm might change in the input variable space for optimum
performance such as power consumption, an important part of sampling cost, for a
given CFD simulation task [121] [122].
This work explores a DOE strategy with varying sampling cost. We examine the
adaptive sampling for adding samples iteratively. A value function is proposed to
balance the varying sampling cost and expected gain for sampling. In addition, the
actual cost of a sample may not be known in advance. Therefore, two sets of surrogates
are developed: one for the target function and the other for the sampling cost. The value
function is defined as the ratio between the standard deviation of prediction and the
sampling cost estimated from the two surrogates. A new sample is then added at the
point with maximum value. As an initial attempt, the study is based on algebraic test
functions with algebraic cost functions for sampling. We evaluated the proposed
approach based on 1D and 2D algebraic functions, which are convenient for
visualization and discussion.
In the rest of the work, details on the standard adaptive sampling using Kriging
are introduced in section 6.2. The adaptive sampling strategy with varying cost is
125
proposed in section 6.3. Section 6.4 introduces the multivariate algebraic test function
and four algebraic cost functions. Section 6.5 investigates the numerical performance of
the proposed approach. Effect of different cost functions with increasing complexity and
different initial samples are discussed. This chapter on adaptive sampling is based on
the publication [12] co-authored with Chanyoung Park, Nam Ho Kim and Raphael T.
Haftka.
6.2 Adaptive Sampling using Surrogate
6.2.1 The Basic Steps in Adaptive Sampling
Adaptive/sequential sampling refers to the systematic procedure to add the
samples iteratively in order to achieve a cost-effective DOE. We consider here adaptive
sampling that relies on an uncertainty model for the surrogate, and which seeks to
improve the surrogate everywhere rather than toward a target such as the global
optimum. Basic steps for such adaptive sampling are provided in Figure 6-1. The initial
samples are first generated with a part of the sampling budget. Then a surrogate model
is developed with prediction variance estimating the prediction errors in the input
variable space. One additional sample newx is added at the point with maximum
prediction variance as shown in Eq. (6-1), where Var x is the prediction variance of a
surrogate, x denotes the point in the input variable space.
126
Figure 6-1. Basic steps of adaptive sampling for global approximation. One additional
sample is added per iteration.
arg max Varnewx x
x (6-1)
Other criteria are also available for adding samples. For example, expected
improvement might be used to search for the minimum function value among the input
variable space[123]. Multiple points can be added per iteration to take advantage of
parallel computation and reduce the number of iterations. The adaptive sampling and
modeling are iterated until reaching prescribed stopping criterion based on total cost or
estimated accuracy of surrogates[124]. In this work, Kriging is used for surrogate
modeling, and the prediction variance is based on its inherent assumption for
multivariate normal distribution. The stopping criterion adopted here is the total
sampling cost. Namely, the adaptive sampling procedure stops just before the total
sampling cost exceeds a prescribed sampling budget.
6.2.2 Kriging Surrogate
Kriging [117] with a constant trend is adopted in the work for surrogate modeling
as given in Eq. (6-2),
Initial design of experiments with partial budget for sampling
Develop surrogate model and prediction variance
Perform sampling/test at the point with maximum prediction variance
Update surrogate model
Stopping criterion (based on cost, estimated accuracy etc)
Done
No
Yes
127
f Z x x (6-2)
where is the mean value of the samples, Z x is assumed to be a Gaussian
process. The covariance function of the Gaussian process is set to be the squared
anisotropic exponential function and given in Eq. (6-3),
22
, ,
1
, expp
m i m j m
m
Cov Z Z x x
i jx x (6-3)
where ix and jx denote two points from the p dimensional space, 2 is the
process variance, m is the hyper-parameter with m=1,2,…,p. The parameters of Kriging
are obtained according to maximum likelihood estimation. The Kriging predictions
interpolate between samples. The Kriging is based on the assumption of multivariate
normal distribution and naturally provides the prediction variance at an untested point.
Kriging has proved effective to approximate response of various systems and been
used as a major surrogate for adaptive sampling.
6.3 Proposed Adaptive Sampling Strategy with Varying Sampling Cost
6.3.1 Proposed Methodology
The classical adaptive sampling strategy is revised to incorporate the effect of
varying sampling cost. For this purpose, a second surrogate C x is constructed to
approximate the sampling cost in the input variable space using adaptive sampling.
Then a value metric ˆ( )V x is defined as the ratio between the standard deviation of the
prediction x and C x as shown in Eq. (6-4). The ˆ( )V x is evaluated at a grid in the
input variable space. A grid with 100 p points is used in this work for the selection of newx .
The one additional sample newx is generated at the point with the maximum value ˆ( )V x as
128
in Eq. (6-5). The ˆ( )V x indicates the uncertainty to be reduced per cost. x and ˆ( )V x
decrease to 0 after adding the sample at x . The x is used to define ˆ( )V x
considering x is proportional to the confidence interval, which is the key measure of
estimated prediction error. The adaptive sampling with varying cost (AS-C) becomes
classical adaptive sampling (AS) when the cost function is a constant. The flowchart of
AS-C is summarized in Figure 6-2.
ˆˆ( )
ˆV
C
xx
x (6-4)
ˆargmax ( )newx Vx
x (6-5)
Figure 6-2. Flowchart of the adaptive sampling with varying sampling cost. One
additional sample is added per iteration at the point with maximum value to improve the global approximation.
Initial design of experiments with partial budget for sampling
Develop surrogate model to approximate the target function and
to approximate the sampling cost
Perform sampling/test at the point with maximum
Update surrogate models and
Stopping criterion based on total sampling cost
Done
No Yes
Compute the value function from and
129
6.3.2 Illustration of AS-C using 1D Algebraic Function
The popular Forrester function[125] is selected for illustration and preliminary
investigation of the proposed AS-C. The Forrester function is a one-dimensional
algebraic function on [0,1]x as given in Eq. (6-6):
2
6 2 sin 12 4f x x x (6-6)
A linear cost function is introduced in Eq. (6-7). The cost function is always
positive in the input variable space. The response of Forrester function and associated
cost function are visualized in Figure 6-3.
0.1C x x (6-7)
Figure 6-3. One-dimensional algebraic example Forrester function. A) Response of the
objective function and B) cost function for sampling.
For the Forrester function example, the initial design of experiment is 3 samples
at x=0, 0.5 and 1, respectively. The initial samples and surrogates are shown in Figure
6-4 (A) for the target function and Figure 6-4(B) for the cost function. It is clear that the
accuracy of f x is poor due to the sparse initial samples. The cost function has a
simple linear trend and C x matches the true cost function even with just 3 samples.
For a complicated cost function, C x is expected to be less accurate. Comparing with
A B
130
AS, the AS-C is based on two surrogates and expected to suffer from larger uncertainty
when the cost function has a complicated response.
Figure 6-4. Initial samples and surrogate prediction for the Forrester function. A) The
objective function and B) linear cost function
After the construction of surrogate from initial samples, one sample is added per
iteration using Kriging. The sampling budget is set to be up to 4.5. Final predictions
using AS resulted in 7 samples as seen in Figure 6-5(A). And final predictions using AS-
C ended with 8 samples as shown in Figure 6-5(B). Details of the sampling are
summarized in Table 6-1. Comparing with AS, the proposed AS-C enabled more
samples in the input variable space for a fixed budget. This observation might be
general for other test functions. Based on the maximum error, AS-C was more accurate
than the AS. In the following section, the AS-C is investigated further using a 2D test
function with different cost functions and different initial samples.
Table 6-1. Details of AS and AS-C for the approximation of Forrester function with a budget of up to 4.5 total cost
Number of samples Total cost Max error
AS 7 4.0 1.46
AS-C 8 4.1 0.87
A B
131
Figure 6-5. Surrogate predictions for adaptive sampling plans for Forrester function
example. A) AS and B) AS-C.
6.4 Multivariate Test Function and Algebraic Cost Functions
The complexity of cost function has a significant effect on the AS-C. In this
section, a 2D test function is selected for further evaluation of AS-C. Four different cost
functions are introduced to imitate the effect of varying sampling cost.
6.4.1 The Normalized Branin Function
The Branin function is selected as the target function to evaluate AS-C. The
function is in two-dimensional input variable space for convenient visualization and
inspection. The original Branin function is given in Eq. (6-8) and defined in
1 2[ 5,10], [0,15]x x . For the study of varying sampling cost, the input variable space
of Branin function and cost function is normalized within 1 2, [0,1]x x for consistency.
The function value of Branin function and cost function are normalized within
[0.1,1.1]f x for convenient quantitative study. The mapping/scaling of input variable
space and function value is performed through linear transformation. The response of
A B
132
normalized Branin function is shown in Figure 6-6. The Branin function mentioned in the
following analysis refers to the normalized Branin function.
2
2
2 1 1 12
5.1 5 1( ) 6 10 1 cos 10
4 8Braninf x x x x
x (6-8)
Figure 6-6. Response of Normalized Branin function in the input variable space
6.4.2 Four Algebraic Cost Functions
Four cost functions have been adopted to illustrate different cases of varying cost
in the input variable space. The four cost functions are constructed from a linear
function, an exponential function, the Rosenbrock function and the Damper function.
The linear function is shown in Eq. (6-9). The exponential function shown in Eq. (6-10)
increases more drastically in the input variable space. The Rosenbrock function is given
in Eq. (6-11) which has a response with moderate nonlinearity. The Damper function
[126] is given in Eq. (6-12) to imitate the resonance effect. The original input variable
space of the four test functions is provided in Table 6-2 to obtain the response. For the
study of varying sampling cost, the input variable space of cost functions is normalized
within 1 2, [0,1]x x for consistency with the target function. The value of cost function is
normalized within [0.1,1.1]f x for convenient quantitative comparison. 0.1 indicates
133
the basic sampling cost. This has the effect of creating a cost ratio of 1/11 for all the
cost functions between the cheapest and most expensive points. The response of
normalized cost functions are visualized in Figure 6-7.
1 2( ) 100 100 50LinearC x x x (6-9)
1 2( ) exp 5 exp 5 50ExpC x x x (6-10)
2 22
Rosen. 2 1 1( ) 100 1C x x x x (6-11)
2
2
22 2 2 2
2
2 2 2
1 1 2 1 2 1 1 2
11
( )
1 1 1 1 1 11 +4
Damper
xC
Rx x x x x x x
x (6-12)
Figure 6-7. Response of the four cost functions in normalized input variable space. A)
Linear cost function, B) Exponential cost function, C) Rosenbrock cost function and D) Damper cost function.
A B
C D
134
Table 6-2. Original input variable space for the four cost functions
Variable ( )LinearC x ( )ExpC x Rosen.( )C x ( )DamperC x
1x [0,1] [0,1] [ 1.5, 0.5] [1,1.1]
2x [0,1] [0,1] [2,3] [1,1.1]
6.5 Numerical Performance of AS-C
6.5.1 Evaluation Plan
The proposed AS-C was compared with AS using the Branin function. For AS,
only one surrogate was developed to approximate the target function and the true cost
function was used to evaluate the total cost of sampling. The stopping criterion of AS-C
and AS was based on the total sampling cost. The sampling procedure stopped just
before the total sampling cost exceeds 10. The sampling procedure was based on
Kriging with constant trend. The initial samples were generated by the Latin Hypercube
Sampling (LHS) with 5000 iterations and 6 initial samples. 100 sets of initial samples
were generated to account for the effect of data scarcity. The key factors of the
evaluation plan are summarized in Table 6-3.
Table 6-3. Key factors of the evaluation plan
Key factor Plan
Initial sample 100 sets of 6 samples generated from LHS
Cost function Four cost functions with varying complexity
Total sampling cost Evolution for R-squared and maximum error of AS and AS-C
at different total sampling cost
Prediction metric R-squared to measure overall accuracy in the input variable
space; max error to measure local accuracy in the input
variable space; area metric to measure the effect of total
sampling cost
Comparisons
between AS and
AS-C
(1) details of a typical case, (2) median performance with
different initial samples (3) variation with different initial
samples
135
The prediction accuracy of surrogate was examined at a matrix of 100 100 grid
jx , j=1,2,…,10,.000. For a given set of initial samples, two prediction metrics were
adopted to estimate the global and local performance of the surrogates, respectively.
The global accuracy was quantified by the coefficient of determination 2R as given in
Eq. (6-13), where jf x denotes the function value at the jth grid point, ˆjf x denoted
the surrogate prediction, f was the mean of jf x . An 2R of 1 indicates that the
surrogate perfectly fits all the function values. The local performance of surrogates was
quantified by the maximum error maxe in Eq. (6-14). The worst prediction/maximum error
is usually of critical interest in addition to the overall accuracy.
100 100 2
12
100 100 2
1
ˆ
1
j j
j
j
j
f f
R
f
x x
x f
(6-13)
max[1,10000]
ˆmax j jj
e f f
x x (6-14)
Figure 6-8. An illustration of the evolution of accuracy for adaptive sampling with
increasing total sampling cost
2R and maxe measured the prediction accuracy at a given cost. The relative
performance of AS and AS-C might alternate with increasing samples as illustrated in
Total sampling
cost
10
AS
AS-C
136
Figure 6-8, where initialc is the cost of initial samples. The performance of an adaptive
sampling scheme should be interpreted over the total sampling cost. The area metrics
2AR
and maxA were proposed to measure the relative performance of AS and AS-C as
given in Eq. (6-15) and Eq. (6-16), where H is the Heaviside step function defined in
Eq. (6-17). The evolution of adaptive sampling with increasing total sampling cost is
interpolated between samples. 2AR
measures the percentage of times when AS-C is
more accurate then AS over increasing samples with accuracy measured by 2R . maxA
measured the percentage of times when AS-C dominates AS over increasing samples
with accuracy measured maxe . An area metric equal to 50% indicates similar
performance of AS and AS-C. The median and variation of 2AR
and maxA with different
initial samples are also presented in the following sections.
2
10
2 2
-
A = 100%10
initial
AS C AS
c
Rinitial
H R c R c dc
c
(6-15)
10
max,AS max,AS-C
maxA = 100%10
initialc
initial
H e c e c dc
c
(6-16)
0, 0
1, 0
nH n
n
(6-17)
6.5.2 A Typical Case using the AS and AS-C
A typical case of the AS and AS-C to approximate the Branin function and linear
cost function is presented in this section. Six initial samples were generated from LHS.
One additional sample was added per iteration until the total cost of samples was about
to exceed 10. The initial samples and final samples are shown in Figure 6-9 using AS
137
and AS-C. The AS-C allocated more samples around the origin due to the cheap
sampling cost. Details of AS and ASC are summarized in Table 6-4. AS ended with 16
samples and the total cost was 9.5, AS-C ended with 18 samples and the total cost was
9.3. Adding one more sample at the location requested by the sampling algorithm would
exceed the total sampling budget of 10. However, it is clear that cheaper samples can
be added.
Table 6-4. Details for a typical DOE of AS and ASC for the approximation of Branin function and linear cost function with up to 10 total cost
2R maxe 2AR
maxA Total sampling cost Number of samples
AS 0.96 0.1 99.7% 100% 9.5 16
AS-C 0.99 0.05 9.3 18
Figure 6-9. Final samples for a typical DOE the approximation of Branin function with
linear cost function. The solid circles denote the initial samples. A) AS and B) AS-C.
Both AS and AS-C approximated the global response well ( 2R were 0.96 and
0.99, respectively) with AS-C being more accurate. The difference between maxe was
also significant (0.1 vs. 0.05) considering the function value varies between [0.1,1].
Evolution of adaptive sampling with increasing total sampling cost is shown in Figure 6-
10. The 2AR
was 99.7% and the maxA was 100%, namely AS-C was almost always
A B
138
better than AS with increasing total sampling cost. The performance of AS-C oscillated
at the beginning in Figure 6-10. This was mainly due to the large uncertainty of the
Branin surrogate from scarce samples. The cost function was estimated accurately
even with the initial samples.
Figure 6-10. Evolution of adaptive sampling with increasing samples for a typical DOE
for the approximation of Branin function and linear cost function. A) R-squared and B) max error.
6.5.3 Different Cost Functions and Different Initial Samples
The performance of adaptive sampling heavily depends on the quality of
surrogate and initial samples. AS and AS-C were evaluated using the four cost
functions. For the evaluation using each cost function, AS and AS-C were repeated 100
times with different initial samples. Therefore, 100 sets of 2AR
and maxA were generated.
The median value of the 100 2AR
and maxA are used to indicate the general performance
of AS and AS-C.
When 2AR
or maxA is equal to 50%, this indicates similar performance between
AS and AS-C with different total sampling cost. 2AR
and maxA larger than 50% indicate
higher accuracy of AS-C over AS-C. As seen in Table 6-5, 2AR
was close to 100 % for
A B
139
all four cost functions, therefore AS-C proved to be accurate more often, but not
overwhelmingly so. The lowest 2AR
came from the Damper cost function which has a
complicated response and lead to a large uncertainty of estimation. The values of maxA
were also high and indicated preference of AS-C over AS. Similarly, the maxA was lower
for Rosenbrock cost function and Damper cost function which having complicated
responses.
Table 6-5. Median value of area metrics 2AR
and maxA (in %) using 100 sets of initial
samples
Area metric ( )LinearC x ( )ExpC x Rosen.( )C x ( )DamperC x
2AR
93.2 96.9 93.4 89.9
maxA 98.3 93.9 77.8 84.9
Figure 6-11. The population of area metrics for the approximation of the Branin function
with four cost functions. A) Area metric for R-squared and B) area metric for max error.
The collection of 2AR
and maxA was visualized using boxplot in Figure 6-11. Almost
all the values of 2AR
and maxA were larger than 50% namely, AS-C was preferred for the
approximation of the Branin function. The boxplot also reveals significant variability of
A B
140
2AR
and maxA . 6 samples were generated using LHS using 5000 iterations. The number
of initial samples was based on heuristic experience. The performance of adaptive
sampling might be more robust by increasing the number of initial samples or
introducing other advanced schemes for the initial DOE.
AS-C might lead to more total samples than the AS. The total number of samples
was collected for the 100 sets of evaluations to validate this observation. Table 6-6
documents the median value of total samples for AS and AS-C from 100 sets of initial
samples. AS-C resulted in more samples in general. The difference between the
number of samples increased while more samples were used. Compare with AS, AS-C
adopted 3 more samples using the linear cost function and 8 more samples using the
damper cost function. The collection of the total number of samples was visualized
using boxplot in Figure 6-12. The AS-C lead to more samples than AS most time.
Table 6-6. Median values of total samples for AS and AS-C using 100 sets of initial samples
Sampling scheme ( )LinearC x ( )ExpC x Rosen.( )C x ( )DamperC x
AS 16.0 25.0 36.0 27.0
AS-C 19.0 31.0 44.0 33.5
Figure 6-12. The number of samples used for the approximation of the Branin function
with four cost functions using AS and AS-C
141
6.6 Concluding Remarks
This work studies the effect of varying sampling cost on the design of
experiments. A strategy for adaptive sampling with varying sampling cost is proposed
(AS-C). Two surrogates are developed to approximate the target function and cost
function, respectively. A value function is defined as the ratio between the standard
deviation of prediction and estimated cost. The value function is used to indicate the
uncertainty for reduction per cost. A new sample is then added at the point with
maximum value. The proposed AS-C method was evaluated using 1D and 2D algebraic
functions and algebraic cost functions. The adaptive sampling stops just before the total
sampling cost exceeding the prescribed budget. Four different cost functions and 100
sets of different initial samples were produced to evaluate the AS-C. AS-C was
compared with standard adaptive sampling (AS) regarding R-squared, maximum error,
and evolution with increasing total sampling budget. For global approximation of the test
function in the whole input variable space, AS-C led to more samples than AS, and AS-
C was more accurate than AS in the large majority of DOEs.
142
CHAPTER 7 DESIGN SPACE SAMPLING BY EXPLORATION AND REPLICATION FOR
ESTIMATING EXPERIMENTAL STRENGTH
7.1 Background
Composite materials have been routinely used in aerospace applications due to
their outstanding capability to be tailored to specific load paths and conditions, resulting
in weight efficient designs. To achieve weight savings, effective and accurate
characterization of structural strength is essential. Significant progress has been
achieved on composite mechanics in past decades. Failure criteria have demonstrated
reasonable accuracy for predicting strength of benchmark structures. For example,
Whitney and Nuismer [103] proposed a failure criterion to predict strength of composite
laminates with a hole. Tsai and Wu [127] proposed a phenomenological material failure
theory which is widely used for anisotropic composite materials. The World Wide
Failure exercises [128, 129] summarized the effort to provide experimental data and
benchmark different modelling strategies for failure criteria of composites.
Various commercial software is available to simulate the response of composite
structures and used for routine analysis. However, each new material system, structural
configuration, and fabrication process requires a large, costly, and time-consuming
program to obtain simulation with reasonable accuracy [104, 130]. Innovative
approaches are being developed to simulate specific structures with certain configures
such as open-hole-tension test [131]. Large challenges still remain for simulating
strength of composite structures with complicated failure mechanisms and variability in
failure response due to progressive damage.
Industry has resorted to empirical approaches where testing has been the focus
for the characterization of structural failure when simulations are inadequate for
143
desirable accuracy or require excessive computational resources. The quantity of
interest for experimental analysis may be as simple as an averaged pass/fail criteria
based on a single load, or may be extended to include mixed mode loadings and more
involved statistical analysis such as strength allowables [132-134]. Handler et al. [135]
discussed the experimental procedures to develop a test database for composite
structures. Carlsson et al. [111] and the Composite Materials Handbook-17 provide in-
depth guideline for systematic experimental analysis.
Experimental results are usually obtained for an array of configurations for
different combinations of important design parameters. Then surrogate models (such
as polynomial response surfaces) are often fit to the data in order to estimate the
structural response over design space. Forrester et al. [125] discussed about
approximating noisy data with Kriging. Glaz et al. [136] adopted multiple surrogate
models for design optimization to reduce vibration of rotor blade problem. Chaudhuri et
al. [137] proposed an adaptive sampling for reliability-based design optimization. In this
context, a surrogate model serves three purposes. Firstly, a surrogate allows for the
statistical averaging or weighed averaging of multiple test results or replicates at the
same configuration (i.e. multi-batch testing). Secondly, it allows interpolation or
extrapolation of the failure response at locations in the design space where test data is
not available. Lastly, it yields a functional representation of a failure criterion which can
easily be incorporated in the design process.
Surrogate prediction consists of two parts: design of experiments and model
development. Design of experiments, which is also termed sampling strategy, refers to
the approach of selecting test configurations. Prediction accuracy of surrogate can be
144
significantly affected by sampling strategy. Wang et. al [116] and Mackman et. al [138]
reviewed popular approaches to generate design of experiments which had significant
effect on optimization results. For design of experiments, we may explore different test
configurations in design space or replicate the same test configuration multiple times.
We would like to examine the tradeoff between exploration and replication when limited
testing resources are available as shown in Figure 7-1. The replication strategy is
represented by 4 × 4 test matrix with 4 replicates at each point, while the exploration
strategy is represented by 8 × 8 test matrix with no replicates.
Figure 7-1. Replication strategy vs. exploration strategy using two design variables and
64 samples
Exploration interrogates the design space to find potential unforeseen failure
modes and unexpected responses, while replication at the same configuration
quantifies experimental scatter and improves accuracy at test points. In the replication
strategy, test variability is considered by averaging replicated tests, which may allow the
use of an interpolating surrogate that passes through the averaged data points. Without
replication, filtering out experimental noise is accomplished by a regression (i.e., least
squares fit) surrogate model in the exploration strategy. In the literature [77, 139] it was
found that exploration may be more efficient than replication in testing structural
elements for a given budget. Exploration is more likely to reduce surprises from un-
𝑥1
𝑥2
𝑥1
𝑥2
Trade-off
145
recognized failure modes, and it yields surrogate models that are more accurate when
applied to approximation at untested designs. However, the results obtained by [77,
139] were with manufactured data to the assumption that experimental strength was a
random variable with normal distribution, independent and identically distributed.
This work first presents a study of how this conclusion stands using real
experimental results produced by the National Institute for Aviation Research (NIAR) on
Open-Hole-Tension (OHT) tests [29, 140] according to ASTM standard [141]. These
OHT experiments were intended to investigate the impact of the coupon width to hole
diameter ratio on the failure of a composite panel. The failure mode and strength were
evaluated at four structural configurations with replicates. We examined the effect of
exploration vs replication on surrogate prediction based on subsets of experimental
results. Different designs of experiments were generated with the same total number of
tests.
The experimental data exhibited deviations from a normal distribution. So we
were prompted to consider other distributions that could be alternatives for
characterization of composite strength, such as the Weibull distribution [142] and
irregular distribution without analytical expression caused by batch to batch variability
[143]. Influence of strength distributions on the sampling methods was investigated
using one analytical function, composite laminate with highly nonlinear failure response.
Polynomial response surface (PRS) was selected for surrogate modeling due to its
robustness and excellence with approximating noisy data. The surrogate toolbox
provided by [72] was adopted to develop PRS prediction.
146
The work is arranged as follows: Section 7.2 introduces the OHT test
configuration and surrogate prediction of OHT tests. Surrogate models are developed
and compared based exploration strategy and replication strategy. In order to
compensate for experimental variability, we resampled subsets of experimental results
emphasizing either exploration or replication and repeated the modeling procedure. In
Section 7.3 we investigate the effect of distribution on sampling plans using one
analytical function and three synthetic distributions. Exploration and replication
strategies are compared. This chapter on the design of experiments is based on the
publication [13, 14] co-authored with Jaco Schutte, Waruna Seneviratne, Nam Ho Kim
and Raphael T. Haftka.
7.2 Exploration versus Replication Sampling Schemes for Open-Hole-Tension Tests
7.2.1 Open-Hole-Tension Tests
The Open-Hole-Tension (OHT) test was designed to investigate the effect of hole
size on the tensile strength of composite laminates. OHT tests were conducted
according to ASTM D5766 [141]. The standard test specimen geometry (w/D=6) is
shown in Figure 7-2. The width of the plate and diameter of the hole are denoted by w
and D, respectively. The plates with a hole were made of three distinct material batches
of Toray T700SC-12K-50C/#2510 plain weave fabric. The specimen laminate ply
orientations were [0/90/0/90/45/-45/90/0/90/0]s with a nominal thickness of 0.172 inch.
Tests were conducted in ambient laboratory conditions with an as-fabricated moisture
content. The tensile strength of the plate with a hole was calculated based on tensile
load and the nominal area (disregarding the hole). Table 7-1 details the test matrix at
four configurations, with w/D = 3, 4, 6, 8. Eighteen replicates were manufactured at
147
each configuration to quantify strength variability. Three pre-preg batches were adopted
to quantify batch to batch variability. All pre-preg batches were manufactured with the
same processing specification but at different dates. Loading rate of the tension tests
was 0.05 in./min. In all the tests, the dimension of “D” is the same, but the specimen
width “w” is different. The smaller lateral gauge may result in higher possibility of
manufacturing defects. Therefore, panels were examined using through-transmission
ultrasonic C-scan prior to machining specimens to guarantee the manufacturing quality.
Figure 7-2. OHT test specimen configuration and observed failure mode. A) Illustration
for the geometry of the composite plate with a hole and A) Typical tension failure mode
of composite laminate with a hole.
Table 7-1. w/ D ratio test matrix. 18 replicates are selected from three pre-preg batches
w (in.) D (in.) w/D Replicates No. of Specimens
0.75 0.250 3 3 x 6 18
1.00 0.250 4 3 x 6 18
1.50 0.250 6 3 x 6 18
2.00 0.250 8 3 x 6 18
Total = 72
Failure at lateral gage middle
F F
D w
A
B
148
For the OHT tests, only a single failure mode was observed, Lateral Gage Middle
and the strength response is likely to be smooth without jump by visual check as seen in
Figure 7-3. The strength of OHT tests was assumed to be a smooth curve perturbed by
random noise. We focused on the prediction accuracy of mean strength with varying
w/D ratio in this work.
7.2.2 Experimental Strength Results and Polynomial Fit
Physical models [111, 144] have been proposed to predict the strength of
composite plate with a hole. We limited ourselves to data-driven approaches only as a
valuable complement to design optimization when the physical models /simulations are
inadequate for desirable accuracy or require excessive computational resources. A
quadratic polynomial response surface (PRS) was selected to approximate the mean
value of failure strength after trying other fits. The PRS was developed using all 72
specimen strength values at four w/D ratios. The OHT strength of each laminate is
shown in Figure 7-3, along with a quadratic PRS fitted to the mean value of specimen
strengths and 95% confidence interval of the prediction variance. The OHT strength
increased gradually with w/D. Statistical information on the replicate OHT strengths
were summarized in Table 7-2, where iS denote tested strength of 18 replicates at
w/D= 3,4,6,8. Detailed experimental results are provided in Appendix D.
Table 7-2.Statistical properties of OHT strengths at given w/D with 18 replicates
Data
set
Mean
(ksi)
Standard
Deviation(ksi)
Coeff. of
variation
(%)
Range of strength
(ksi)
3S 59.64 3.52 5.90 [54.27, 66.33]
4S 63.22 2.79 4.41 [58.66, 68.07]
6S 66.53 3.01 4.52 [61.83, 71.89]
8S 69.27 3.04 4.39 [64.51, 76.24]
149
Figure 7-3. Experimental data and quadratic polynomial response surface fitted to the
data.
In Table 7-3, we can see that the maximum discrepancy between surrogate
prediction and mean strength was 0.8 % or less. Different from the interpolation
surrogate models (i.e., Kriging) that pass all the samples, the PRS filters noise and
preserves monotonicity in design space. Therefore, a quadratic PRS seemed
reasonably accurate to approximate OHT tests and was considered as the true function
value in the following analysis of prediction accuracy. The difference between this PRS
model and any other surrogate models based on subsets of OHT test results were
assumed to be from sampling strategies.
Table 7-3. Relative difference between surrogate prediction and mean strength using quadratic PRS
w/D=3 w/D=4 w/D=6 w/D=8
Relative difference (%)
0.4 0.8 0.5 0.2
7.2.3 Resampling Experimental Strength for Comparing Exploration and Replication
To compare sampling strategies for exploration and replication, we simulated
situations of paucity of data, by resampling partial experimental strength out of the 72
0 2 4 6 8 1045
50
55
60
65
70
75
80
w/d
Str
ength
(ksi)
Quadratic PRS
Estimation
Samples
95% Confi.
150
available experimental results without replacement. For the exploration strategy, we
sampled from all values of w/D, while for the replication strategy we left out one value of
w/D but had more replicates of the other three. This resampling approach was
motivated by bootstrapping [145], which is a nonparametric approach for statistical
inference. Strength of composite material is often associated with large variability (as
seen in Appendix D), the evaluation of sampling strategy should be interpreted in the
context of stochastic effect. Resampling procedure make the most use of limited
experimental strength and is likely to enable insightful understanding of sampling
strategy excluding stochastic effect. All the resampling plans were repeated 1000 times
using Monte-Carlo simulations to account for randomness. We compared surrogate
prediction for the cases of a total of 12 samples and of 24 samples.
Figure 7-4. Illustration for one set of resampled experimental strength representing
exploration or replication strategy using 12 points. A) Illustration of exploration and B) Illustration of replication strategy while omitting w/D=3.
For the exploration strategy, with 12 samples, 3 samples were randomly selected
from each of the 3 4 6 8, , ,S S S S sets. For the replication strategy, 4 samples were selected
from each of , ,i j kS S S sets, where i j k as shown in Figure 7-4. We skipped one of
w/D and used 4 replicates for each of the remaining three. Replication strategy was
A B
151
repeated four times by omitting one w/D in turn. With 24 samples, the same scheme
was used but with twice as many samples for each w/D.
A quadratic PRS was then constructed with the selected samples. The relative
error, ( )err x in Eq. (1) was computed at the four test configurations to quantify the
prediction accuracy,
( / ) ( / )
( / )( / )
i ii
i
f w D f w Derr w D
f w D
(7-1)
where ( / )f w D denoted predicted failure strength and ( / )f w D denoted the true strength
obtained from the previous PRS with all 72 samples. The root-mean-square error
(RMSE) given in Eq. (2) was used to quantify the overall performance of surrogate
prediction.
42
1
( / )
RMSE4
i
i
err w D
(7-2)
Table 7-4. Mean values of relative RMSE error (%) for surrogate prediction at w/D=3, 4, 6, 8 using 12 specimens and 24 specimens. Mean values reported are based on 1,000 sets of samples
Sampling
strategy
Exploration Replication omitting specimen from specific w/D
w/D=3 w/D=4 w/D=6 w/D=8
12 samples 2.27 3.07 2.33 2.76 6.75
24 samples 1.57 2.22 1.63 1.89 4.70
Table 7-4 shows the accuracy of surrogate models using exploration and
replication strategies with 12 and 24 total samples. The RMSE of relative error for 1000
sets of samples were computed at four configurations, w/D =3, 4, 6, 8. It is obvious that
for this problem the exploration strategy was more accurate than the replication
strategy. When modeling surrogates using samples from (w/D=4, 6, 8) and (w/D=3, 4,
6), the estimation errors at w/D=3 and w/D=8 were significantly large due to the
152
necessity for extrapolation. Also, the largest RMSE errors among the different sample
selections occurred when extrapolation was required.
The influence of number of samples on prediction accuracy was also observed.
When twice as many samples are used to average the noise, the RMSE error is
expected to be reduced by the square root of 2. Indeed, the errors were reduced by a
factor ranging from 1.38 to 1.45. The error in surrogate prediction strongly depends on
variability of experimental strength, and the results in Table 7-4 show that exploration
filters out the noise in individual measurement as effectively as or more effectively than
replication. We further discuss the effect of strength variability on surrogate models in
the following section to make the study more general.
7.2.4 Identifying the Distribution Type of OHT Strength
In their previous work, Matsumura et al. [77] assumed that the distribution of
strengths at the same configuration was normal, independent and identically distributed
(i.i.d.). Independent and identical normal distributions are frequently used for theoretical
studies of stochastic effects. Here we examined to what extent this assumption holds for
OHT test data. We assume all iS follow the same type of distribution, but possibly with
different distribution parameters for each w/D. Three candidate distributions were
selected to characterize variability of strength: a normal distribution, two-parameter
Weibull distribution, which are commonly used for strength characterization, and a
uniform distribution for comparison purposes.
The iS distribution was analyzed using the Kolmogorov–Smirnov (K-S) test,
which is a non-parametric test to quantify the goodness-of-fit between a given
probability distribution and empirical distribution of samples [146]. The p-value of K-S
153
test indicates the probability that samples do not reject the hypothetical distribution. A
high p-value denotes high probability that samples are from the hypothetical distribution.
Effectiveness of K-S test strongly depend on the number of samples. Therefore, we
pooled iS together (i.e. from different w/Ds) for better estimation before applying the K-S
test. iS distributions had similar coefficients of variation (CV) as seen in Table 7-2, which
implied that it was possible to combine the four sets of iS together to query the
underlying distribution. Samples were transformed into the standard form, namely with
location parameter equal to zero and scale parameter equal to one. Eq. (3) was used to
standardize iS while assuming normal distribution or uniform distributions. Eq. (4) was
used to standardize iS while assuming Weibull distribution.
( )
( )
i i
i
mean
std
i
S SN
S, where i = 3, 4, 6, 8 (7-3)
( )
i
istdi
SW
S, where i = 3, 4, 6, 8 (7-4)
The p-values for pooled strengths were computed for the three candidate
distributions as shown in Figure 7-5. A 5% significance level was adopted to exclude
distributions that were less likely to occur (the red horizontal line in Figure 7-5).
Candidate distributions with a p-value lower than the significance level were rejected.
The figure showed that there was a high chance that OHT strength follows a normal
distribution, a lower chance that it came from the Weibull distribution, and the uniform
distribution was rejected. Probability plots for OHT strengths while assuming normal
distribution and Weibull distribution are provided in Figure 7-6 for further comparison.
The OHT strengths matched both distributions in general except at the tails. The
inconsistency at tails may be due to outliers in the experiments and the inability of
154
statistical distributions to approximate physical phenomenon, as well as the possibility
that the real distribution may be different. We also repeated the analysis of fitting of the
distributions considering samples from each batch independently by removing the bias
associated with batches. The P values using Normal distribution were around 0.9 for all
the three batches which were larger than or comparable with the P values using Weibull
distribution. Therefore, Normal distribution seemed desirable to approximate the
variation of the specimen strength.
Figure 7-5. K-S test statistic for OHT strength.
Figure 7-6. Probability plots for the fit of pooled OHT tests. A) Normal distribution and B)
Weibull distribution
Normal Weibull Uniform0
0.2
0.4
0.6
0.8
1
Candidate distribution
p-v
alu
e f
or
K-S
test
5% significance level
-2 -1 0 1 2
0.0050.01
0.050.1
0.25
0.5
0.75
0.90.95
0.990.995
Normalized Strength
Pro
babili
ty
Probability plot for normal distribution
p-value of Kolmogorov-Smirnov test:
Normal dist.: 0.936
10-0.05
10-0.01
100.03
0.01
0.05
0.1
0.25
0.5
0.75
0.90.950.99
0.999
Normalized Strength
Pro
babili
ty
Probability plot for Weibull distribution
p-value of Kolmogorov-Smirnov test:Weibull dist.: 0.49
A B
155
7.2.5 Effect of Between-batch Variability
We examined the distribution of OHT test results and concluded that the
assumption that structural strength followed i.i.d. normal distribution was not perfectly
satisfied as assumed in Matsumura et al. [14, 15]. The strength of composite laminates
could be associated with significant variability as seen in section 2.3. In practical
designs, specimens may be made from different pre-preg batches. Batch-to-batch
variability could be a major source of strength variation as reported by ASTM standard.
Notable batch-to-batch variability would invalidate the assumption of using i.i.d. normal
random variability to approximate structural strength. Figure 7-7 compares iS obtained
from different batches. Let /Dw
ib be the strength of specimen tested from batch i at w/D,
where i=1,2,3 and w/D=3,4,6,8. At given w/D, maximum values of min ( /Dw
ib ), max ( /Dw
ib
) and mean ( /Dw
ib ) were mostly from batch 2 indicating that specimens made from batch
2 tended to be stronger.
Figure 7-7. Statistical properties of specimen strength
3 4 6 850
55
60
65
70
75
80
w/D
str
ength
(ksi)
OHT test specimen strength
Batch 1
Batch 2
Batch 3
mean of batch 1
mean of batch 2
mean of batch 3
95% C.I.
156
We assumed /Dw
ib followed a normal distribution based on the results of the
previous non-parametric analysis. /Dw
ib was likely to have different mean value with
same w/D ratio. Welch's t-test was adopted to compare central values of two Gaussian
populations [147]. This test assumes that the two populations have normal distributions
and unequal variances. The p-value was used to indicate probability of the null
hypothesis that the two population means are equal (using a two-tailed test). A high p-
value represents high possibility that the two distributions under consideration are
equal. Table 7-5 documents the p-value for Welch's t-test between specimens made
from batch 1 and batch 2. The p-value at w/D= 3,6,8 are smaller than the 5%
significance level; i.e., the assumption that the two samples were from the same
distribution was rejected. The p-value at w/D=4 was 0.15, which is slightly larger than
significance level 0.05. Based on Figure 7-7 and Table 7-5, strengths of specimens from
same batch obviously shared systematic bias. The between-batch variability resulted in
a violation of normality assumption and i.i.d. condition even though iS was close to
normal distribution in overall.
Table 7-5. Statistics of welch's t-test between specimens from batch 1 and batch 2
w/D=3 w/D=4 w/D=6 w/D=8 /D
1
wb /D
2
wb /D
1
wb /D
2
wb /D
1
wb /D
2
wb /D
1
wb /D
2
wb
Mean
value
57.6
ksi
63.3
ksi
63.8
ksi
65.2
ksi
66.7
ksi
69.8
ksi
67.3
ksi
70.7
ksi
p-value 0.0004 0.15 0.01 0.04
It would be interesting to check the effect of between-batch variability on
prediction accuracy. This is an extended study on tradeoff between exploration of more
batches and replication ignoring the between-batch variability. We assumed each pre-
157
preg batch was equally important, and selected at least one sample from each batch to
conserve batch-to-batch variability. The sampling plan conserving batch-to-batch
variability is termed three-batch sampling and can be viewed as emphasizing
exploration, while random sampling may omit a batch and have more replicates in the
other batches. We compared surrogate models based on random sampling and three-
batch sampling. Table 7-6 shows the accuracy of surrogate models using exploration
and replication strategies with 12 specimens. The relative RMSE for 1000 sets of
samples were computed at four configurations, w/D =3, 4, 6, 8. It is seen that we benefit
from both types of exploration, using full set of w/Ds, and all three batches.
Table 7-6. Mean values of relative RMSE error (%) for surrogate prediction at w/D=3, 4, 6, 8 using 12 specimens (3 replicates from 4 configurations for exploration, 4 replicates from three configurations for replication). Mean values reported are based on 1,000 sets of samples
Sampling
strategy
Exploration Replication omitting specimen from w/D=i
w/D=3 w/D=4 w/D=6 w/D=8
Random
sampling
2.27 3.07 2.33 2.76 6.75
Three-batch
sampling
1.62 2.45 1.69 2.14 4.9
7.3 Influence of Distribution Type on Comparison between Sampling Strategies for an Analytical Model of a Composite Plate
The experimental results indicated that the strength of composite laminates was
not a perfectly independent and identically distributed Gaussian random variable.
Therefore, we examined the influence of distribution type on sampling strategies using
an analytical test function. While the results aligned with the previous numerical studies
showing that exploration was better than replication, the OHT case was relatively
benign compared to the numerical examples of Matsumura et al. [77]. First, there is only
one design parameter in the OHT test results; second, the failure stress varies smoothly
158
allowing accurate approximation with low-order polynomials; and third, the variation of
the distribution is relatively small. To obtain further insight, we investigate sampling
strategies and distribution types using one analytical function of the failure load of an
unnotched composite laminate with highly non-linear failure response. The geometry of
the plate and the dependence of the failure load on the two design parameters are
shown in Figure 7-8.
Figure 7-8. Configuration and failure response of composite laminate. α is a constant to
balance between horizontal and vertical loadings. θ is the orientation angle of lamina. A) Illustration of composite laminate and B) Response for critical failure load in the design space.
The composite laminate structure had three ply angles [0°/−θ°/+θ°]s which was
intended to have a more complex failure response as shown in Figure 7-8. The laminate
is subject to forces Nx and Ny along x and y axis defined by parameter α, where Nx
=(1−α)F and Ny = αF. There are two design parameters for this structure: θ ∈ [0°, 90°]
and α ∈ [0,0.5]. Table 7-7 details the material properties and strain allowables. The
strains are predicted by the Classical Lamination Theory (CLT). Critical failure mode
was due to ply axial strain. The failure loads were calculated analytically as in
Matsumara et al. [77].
00.1
0.20.3
0.40.5
20
40
60
80
2
4
6
8
10
12
x 105
Strength from multi-source variability
F
A B
159
Table 7-7. Parameter settings of composite laminate
Properties E1(GPa) E2(GPa) ν12 G12(GPa) Ply
thickness
(μm)
ε1allow ε2allow γ12allow
Quantities 150 9 0.34 4.6 125 ±0.01 ±0.01 ±0.015
The analytical failure loads are perturbed with synthetic noise to imitate
experimental variability. Three types of distribution were considered in this work: normal
distribution, Weibull distribution with heavy tail, and multi-source distribution to imitate
batch-to-batch variability. The numerical study focused on average performance of
surrogates based on the synthetic samples from Monte Carlo simulation. Variability of
the performance of surrogates are attached in Appendix D for reference.
7.3.1 Synthetic Noise using Three Types of Distributions
We considered three types of distributions to simulate strength variation. Normal
and Weibull distributions were considered as two popular distributions to approximate
the structural strength. The normal distribution was set to be ( ( , ), ( , ))N NN ,
where ( , )N was the mean value of structural strength from analytical function at
configuration ( , ) , and ( , )N was obtained assuming constant coefficient of
variation (CV) to be 0.05.
The Weibull distribution was first shaped to follow ( , )W a b with a heavy tail to
amplify the difference with the normal distribution, where the scale parameter a=2, the
shape parameter b=1.5. Then ( , )W a b was scaled through a linear transformation to
imitate the experimental variability of the analytical test function. Details of the Weibull
distribution were given in the Appendix C.
160
The multi-source distribution was proposed to imitate between-batch variability
based on findings from the OHT test results. We first generated 5 random values iM as
the mean strengths of 5 batches from 2(0,0.035 )N and fixed to imitate the between-
batch bias for the following analysis. The mean values were then scaled up through
linear transformation at different configurations ( , ) as shown in Eq. (5).
( , ) ( , ) ( , )i im f M f (7-5)
Synthetic samples were then generated from 2( , ), ( , ) 0.035i iN m m
assuming constant coefficient of variation (CV) to be 0.035. Typical probability density
functions (PDF) of three distributions are illustrated in Figure 7-9 for 8.5 ksi mean value
of strength. The synthetic specimen strength in the design space is assumed to come
from one of the five batches. The multi-source distribution should be skewed due to
between-batch for five specific distributions as seen in Figure 7-9 (C).
Figure 7-9. Three distributions with mean value to be 8.5 ksi. The mean values of the five synthetic batches are 8.1 ksi, 8.5 ksi, 8.8 ksi, 8.4 ksi, 9.0 ksi. A) Normal distribution, B) Weibull distribution and C) Multi-source distribution.
7.3.2 Fitting Strategy
A polynomial response surface (PRS) surrogate was selected because of its
robustness and good performance for predictions based on noisy data. Leave-one-out
cross validation was adopted to select an appropriate polynomial order for each
A B C
6 7 8 9 10
x 104
0
0.005
0.01
0.015
0.02Strength from Normal distribution
Strength
Em
piric
al P
DF
6 7 8 9 10
x 104
0
0.005
0.01
0.015
0.02
0.025
Strength
Em
piric
al P
DF
Strength from Weibull distribution
6 7 8 9 10
x 104
0
0.005
0.01
0.015
Strength
Em
piric
al P
DF
Strength from multi-source variability
161
individual set of samples. PRS assumed constant variance instead of constant
coefficient of variation. Therefore, function values of sampling points were preprocessed
using logarithmic transformation with base 10 to obtain close-to-constant variance.
As in Matsumura et al. [77], our test matrices ranged from 4 × 4 to 7 × 7 with
evenly spaced test points in order to investigate the effect of the density of matrix on the
accuracy of approximation. For each test matrix, we replicated the same test
configuration up to seven times. Each test matrix was generated and fitted 1000 times
to compensate for randomness of samples and obtain the average accuracy. Accuracy
of surrogate prediction was computed using a 20 × 20 matrix of test points (400 points
in total). Relative RMSE from Eq. (4) was adopted to measure overall accuracy of the
surrogate model.
7.3.3 Influence of Distribution Type on Sampling Strategies
We first developed surrogate models to approximate the analytical function
without noise for reference. Figure 7-10 (A) summarizes prediction accuracy of
surrogate prediction at all test matrix without noise. The typical orders for polynomial
response surface selected using cross validation were quadratic, cubic, quartic and
quantic respective for the four sets of test matrix. The composite laminate has a highly
nonlinear response (see Figure 7-8) that greatly benefited from the higher order
polynomials made possible by using more test points. This example corresponds to a
case that is more challenging to fit than the OHT case. The RMSE of relative error
reduced from 0.2 to 0.11 with increasing points for exploration. While modeling the
composite laminate strength, a typical PRS model based on 4×4 matrix was quadratic
and a typical PRS model based on 7 × 7 matrix was quantic. Note that since there is no
noise, RMSE remains constant through replication.
162
Figure 7-10. Mean value of RMSE for relative error based on 1000 sets of samples with
different distribution types. Number of samples indicates the total number of tests including 1~7 replicates for each test point. A) Without noise, B) Noise under Normal distribution, C) Noise under Weibull distribution and D) Noise under multi-source distribution.
Noise was then added to the data, and the influence of noise distribution on
sampling strategies was compared. Figure 7-10 documents the accuracy of surrogate
models for the composite laminate. From Figure 7-10 (B-D), we can observe a clear
advantage of exploration over replication with different distributions. The prediction
accuracy did not change much with increasing replicates for the same configuration set.
This was due to the highly non-linear response that implies that surrogate model error is
the major concern. Increasing replicates could mitigate mostly the effect of randomness
B
C D
A
163
(noise). Allocating samples for exploration enables high-order PRS for closer
approximation to the response. That is, model error was more significant than sampling
error corresponding to experimental variability for this example. We also examined the
influence of magnitude of noise on prediction accuracy in Appendix E. Noise with a
coefficient of variation varying from 0.05 to 0.2 only had a limited effect on predicative
capability of surrogates comparing with model-form.
7.4 Concluding Remarks
To predict strength of many composite structural elements, it is necessary to
perform structural strength tests for a matrix of configurations as function of loading,
geometry and material properties. Then a surrogate fit allows interpolation or
extrapolation to be performed to obtain strengths of untested configurations. For a given
number of tests, we must balance between exploration, meaning testing many structural
configurations, and replication of each tested configuration to overcome the effects of
noise. Previous work has shown that exploration is more effective than replication
especially when the functional form of response is complicated, but it was based on
simulated test data assuming independent and identically distributed normal noise.
This work first tested the conclusion favoring exploration using actual test data of
strength of open hole composite plates as function of the ratio of plate width to hole
diameter, w/D. We proposed a resampling procedure based on subsets of experimental
strength to compare exploration and replication. The experimental data consists of 72
samples, indicating failure strengths for three batches of laminates with 6 replicates per
batch and four values of w/D. The two strategies were compared by using 12, 24 or all
72 samples. For each case, testing all four w/D configurations with fewer replicates was
compared with testing only three of the four with more replicates. Using 12 or 24
164
samples randomly drawn from 72 samples, the accuracy of fitted surrogates was
checked 1,000 times in a Monte Carlo sampling strategy. The results showed that
testing all four configurations with fewer replicates led to more accurate predictions than
testing only three configurations with more replicates, thus confirming the previous
simulated results.
We then investigated the major source of uncertainty for prediction, which is the
variability of strength. We found the distribution of OHT strength is not perfectly
independent and identically distributed normal distribution as commonly assumed. This
was likely due to systematic bias in strength for different prepreg batches. To generalize
our observation on sampling strategies, we investigated the influence of distributions
based on synthetic data using non-normal distributions and correlations. Synthetic
samples were generated based on the strength response for an un-notched composite
laminate using normal distribution, Weibull distribution and multi-source distribution.
Prediction of polynomial response surface(PRS) with fixed order/terms had similar
accuracy while increasing replicates for different distributions. Introducing more
replicates did not benefit prediction accuracy noticeably. An exploration strategy
enables a higher order PRS to be used, which is key to increase prediction accuracy for
complex structural behavior.
These results are limited to the accuracy of the mean value of the failure
strength. Future work will focus on estimating the tolerance limits, such as the B-basis
using surrogate models for designing structures with a limited number of available
experimental results. We are also searching for additional composite test types to
investigate the application of surrogate.
165
CHAPTER 8 FIELD APPROXIMATION USING SURROGATES AND PROPER ORTHOGONAL
DECOMPOSITION
In this chapter, we investigate the approximation of flow field using surrogate and
proper orthogonal decomposition (POD). The accuracy for predicting extreme quantities
of interest are evaluated using both field approximation and only surrogate. All the study
is demonstrated using a simple CFD analysis of drop falling implemented in Matlab.
8.1 The Drop Falling for Direct Numerical Simulations of Multiphase Flows
In order to investigate flow field approximation, we adopt a simple Navier-Stokes
solver to simulate a drop falling in a rectangular box [148]. The code for the simulation
was obtained from section 3.4 in [148]. A forward in time, centered in space
discretization is used. The density is advected by a front tracking scheme and a
stretched grid is used, allowing us to concentrate the grid points in specific areas. Four
snapshots at different simulated time have been taken in Figure 8-1 for illustration. The
rectangular box is set to be one-unit length vertically and horizontally. Initial radius of
the spherical drop is 0.15. Density of the flow and drop are set to be 1 and 2
respectively. Viscosity is a constant 0.01(water at 20°C). The viscosity of a fluid is a
measure of its resistance to gradual deformation by shear stress or tensile stress.
This CFD analysis seemed a feasible test function for studying field
approximation due to fast computation and reasonable complexity. Three variables
could be parameterized to apply the methods of snapshots: time, viscosity and density
difference. Complexity of the front increases as time goes.
166
Figure 8-1. The front and velocity field in a rectangular box at different times. A) 0 unit
time, B) 50 unit time, C) 100 unit time and D) 200 unit time.
8.2 Approximating Flow Field using Proper Orthogonal Decomposition and Kriging
Proper orthogonal decomposition(POD) has been proposed to associate with
Kriging surrogate model to realize field prediction [21]. POD is adopted to obtain major
modes for characterizing the flow field and Kriging is used to approximate mode
coefficients. Comparing with traditional projection methods for estimating mode
coefficients, this flow field prediction method is independent of the governing equations.
The numerical instabilities caused by viscosity and compressibility for POD-projection
A B
C D
167
method could be avoided. The POD decomposes the original flow field into a linear
combination of basic modes. We adopt K snapshots to solve basic modes. The modes
could be obtained through optimization procedure that constrains the modes to be
normalized and orthogonal. The scalar value representing flow field at each pixel or
grids make up a matrix ( ) ( )
( , )
k k
i jP p . Here, k denotes indices for snapshots (samples)
and (i, j) are the indices of the pixels, where i=1,2,…,I and j=1,2,3…,J. Each flow field is
a linear combination of M basis modes m and corresponding coefficients ( )k
mc as seen
in Eq.(8-1) , where m is mode index, M=K while all the modes are used for
approximation. The basic modes and mode coefficients ( )k
mc are obtained by minimizing
the root-mean-square difference between the snapshots and predictions as in Eq.(8-2).
The constraints for normalization and orthogonality are provided in Eq.(8-3) and Eq.(8-
4).
( ) ( )
1
Mk k
m m
k
P c
(8-1)
2
( ) ( )
1 1
minK M
k k
m m
k m
P c
(8-2)
subject to
( )
,
1 1
1I J
k
i j
i j
p
(8-3)
and
,
,
( ) ( ) ,
1 if
0 f
m p m p
m p
x x dx
i j
i i j
(8-4)
168
Since ( )m x is normalized only to reflect data pattern, ( )k
mc contains the amplitude
that the corresponding basic modes contribute to the reconstructed flow field. ( ) 2( )k
mc (so
called energy for velocity field) quantifies the contribution of mth mode to the kth flow
field. The POD mode spectrum is often used to describe the fraction (energy function for
velocity field) each mode captures of the total ensemble of snapshots.
( ) 2
1
( )K
k
m m
k
C c
(8-5)
1
/M
m m i
i
Frac C C
(8-6)
The mode order of the POD spectrum is defined according to the magnitude of
mFrac . Most variation of the flow field could be captured from the first few modes.
Coefficients of modes at untested points are approximated using Ordinary Kriging [72].
8.3 Approximating Extreme Scalars using Field Data and Scalar Data
Field approximation using Proper Orthogonal Decomposition (POD) is usually
adopted to build database of expensive CFD for future efficient design optimization.
Various extreme values are the most important quantity of interest for design
optimization. We compare approximation of fields and approximation of scalars
regarding to accuracy of extreme values. Pressure field is usually of critical interest for
engineering analysis. We examine approximations of minimum pressure for the drop
falling simulation with varying time. Considering symmetry of the flow field, only half
domain is adopted for analysis as seen in Figure 8-2.
169
Figure 8-2. Illustration of three snapshots at different times. Red point denotes location of minimum pressure
Four snapshots were adopted at different times for analysis. We extracted four
modes using POD from the snapshots. The mode coefficients at varying time were
estimated using Kriging as seen in Figure 8-3. The minimum pressure was then
obtained from the estimated fields. We also developed a Kriging surrogate to estimate
the minimum pressure directly based on the four samples. The prediction accuracy is
shown in Figure 8-4. For approximation of scalar using Kriging, root-mean-square error
(RMSE) of relative difference with true function is 3.8%, maximum relative difference is
10.3%. For approximation of scalar using POD and Kriging, RMSE of relative difference
with true function is 15.0 %, maximum relative difference is 34.7%. It is obvious that
direct approximation of scalar using Kriging is more accurate.
170
Figure 8-3. Kriging prediction of mode coefficients with varying time. A) 1st mode, B) 2nd
mode, C) 3rd mode and D) 4th mode.
Figure 8-4. Predicting minimum pressure from surrogate estimation and field
approximation. Blue line is true value, red line is approximation. A) Surrogate estimation alone, B) Field approximation using Kriging and POD.
A B
C D
A B
171
8.4 Predicting Border of Multiphase Flow
Besides the minimum pressure, we explore the approach to improve POD
prediction further regarding to approximating other extreme quantities. The border of
multiphase flow is one quantity of interest for engineering analysis. The CFD analysis is
based on incompressible flow, uniform viscosity for the two flows. We transformed the
density field of multiphase flow to be binary value 0 and 1 to indicate different phases.
We then performed field prediction using binary snapshots based on two variables:
viscosity and time for transient simulation. Parameter settings for the CFD analysis are
provided in Table 8-1.
Table 8-1. Parameter settings
Parameters Viscosity Simulation
Time
CFD
grids
Pixels Density
for drop
Density for
flow
Setting 0.005~0.015 150~250 32×32 656×875 2 1
Most variation of the flow field could be captured from the first few modes. We
adopted 99 snapshots and the first 50 modes are adopted for prediction. Contribution of
first 4 modes are summarized in Table 8-2. Mode 1 is most significant compared with
the rest.
Table 8-2. Fraction of contribution for first 4 modes to ensemble snapshots
Coefficients of modes at untested points were approximated using Ordinary
Kriging as shown in Figure 8-5. The coefficients illustrated smooth response and less
likely to have noise.
mFrac m=1 m=2 m=3 m=4
Value 97.5% 0.84% 0.59% 0.22%
172
Figure 8-5. Coefficients of first 4 basis modes in variable space using Kriging. A) Mode
1, B) Mode 2, C) Mode 3 and D) Mode 4.
We proposed to use filter and projection [149] for post-processing of field
prediction. A field value at (i,j) was updated by the average field value within certain
distance from (i, j), namely filter domain as seen in Eq.(8-7). We used a square domain
with side length to be 2R as filter domain. ( )
,
k
f i jp denotes filtered field value at (i,j).
Projection approach is essentially a step function as seen Eq.(8-8). is the threshold
value for binary predictions. and R were empirically estimated based on cross-
validation.
( )
,
1 1( )
,
R RI Jk
i j
i jk
f i j
R R
p
pI J
(8-7)
B A
C D
173
( )
,( )
, ( )
,
1, if
0, if
k
f i jk
pi j k
f i j
pp
p
(8-8)
4 validation points and 99 samples are shown in Figure 8-6. The graphical
complexity increased with reduced viscosity and increasing time. Comparison between
POD prediction and simulation are summarized in Figure 8-7. POD predictions were
almost the same as simulation regarding to drop fronts. We may reduce the number of
samples for this specific case considering the excellent prediction accuracy.
Figure 8-6. Illustration for validation points, samples and variable space
Increasing graphical complexity
Incr
easi
ng
grap
hic
al c
om
ple
xity
(1)
(2)
(3) (4)
174
Figure 8-7. Comparing simulation and POD prediction at 4 validation points. A) Point 1,
B) Point 2, C) Point 3 and D) Point 4.
8.5 Concluding Remarks
We have examined the numerical performance of field approximation using POD
and Kriging surrogates. Flow fields are predicted as a linear combination of selected
basis modes generated by POD. The mode coefficients are approximated using Kriging
models with varying variables. We compared predicative capability for extreme quantity
of interest using field approximation and scalar approximation based on a simple CFD
simulation. Field prediction is found to be less accurate with minimum pressure from the
flow field comparing with scalar prediction.
We have also investigated the prediction of border between multiphase flow. The
border between immiscible liquids is important to investigate physical interaction. We
found the approximation using POD prediction is blurred as grey domain. Therefore, we
D C
A B
175
proposed to use filter and projection approach for post-processing. These approaches
proved to be effective to refine border prediction based on numerical tests.
In the future, it should be helpful to investigate the effect of factors on
approximation schemes. Several major factors have been identified from previous
study, such as number of snapshots, number of modes, design of experiments, number
of variables.
176
CHAPTER 9 CONCLUSION
For design optimization and uncertainty quantification, we need to examine large
number of candidate configurations. This is commonly done by doing simulations and/or
experiments at a number of design points and approximating at untested points. Usually
a single approach is used to combine all the information in order to make a prediction.
The objective of this work are to check on combining multiple ways to predict by
integrating various data sources and numerical models. A series of approaches have
been proposed and elaborated in this work.
Firstly, this work proposes to develop surrogates from different data sets. Multiple
predictions based on different data sets are assumed to be independent and combined
using Bayesian inference to increase reliability. We apply the idea of multiple data sets
to predictions of a single inaccessible point rather than estimating the whole variable
space. Data sets are sampled along different lines converging to the target point. One-
dimensional models are then performed individually and combined at target point.
Converging lines proved to be much more reliable and accurate than multi-dimensional
model based on the modeling of a physical function (drag coefficient function).
Secondly, the linear regression multi-fidelity surrogate (LR-MFS) was proposed
to combine dataset with different fidelity, especially for high-fidelity with noise. The linear
regression is commonly used, balanced between accuracy, cost and simplicity. The LR-
MFS is less likely to overfit noise by limiting the number of parameters. The LR-MFS is
derived from standard linear regression with analytical solutions to obtain parameters to
guarantee global optima. The treasure of linear regression techniques could be applied
directly to LR-MFS, such as prediction variance, optimal DoE. The LR-MFS was
177
demonstrated using polynomial response surface (PRS) in this work. The proposed LR-
MFS was first compared with three Bayesian frameworks using a benchmark dataset
from the simulations of a fluidized-bed process. The accuracy of Bayesian frameworks
varied significantly, and the LR-MFS was comparable to the best GP-based approach.
The effect of multiple low-fidelity datasets was also discussed. The three-level fidelity
MFS was inferior to the one-level MFS for limited number of high-fidelity dataset. Then
the LR-MFS was evaluated to approximate a non-linear numerical test function with
different sampling plans. The LR-MFS bested Co-Kriging for 55%~63% cases with
increasing number of high-fidelity(HF) samples. For both examples, the LR-MFS
proved to be better than fitting only HF samples, and robust for HF samples with
different level of noise. LR-MFS could avoid the error for fitting LF data as demonstrated
for the numerical example. The LR-MFS heavily relies on the model form of the
discrepancy function. In the future, the selection of basis terms will be examined using
existing schemes such as stepwise regression. After evaluation using the benchmark
datasets, the proposed scheme has been applied for two engineering applications: (1)
behavioral emulation of exascale computer architectures; (2) predicting strength of
composite laminates with a central hole.
Thirdly, this work studies the effect of varying sampling cost on the design of
experiments. A strategy for adaptive sampling with varying sampling cost is proposed
(AS-C). Two surrogates are developed to approximate the target function and cost
function, respectively. A value function is defined as the ratio between the standard
deviation of prediction and estimated cost. The value function is used to indicate the
uncertainty for reduction per cost. A new sample is then added at the point with
178
maximum value. The proposed AS-C method was evaluated using 1D and 2D algebraic
functions and algebraic cost functions. The adaptive sampling stops just before the total
sampling cost exceeding the prescribed budget. Four different cost functions and 100
sets of different initial samples were produced to evaluate the AS-C. AS-C was
compared with standard adaptive sampling (AS) regarding R-squared, maximum error,
and evolution with increasing total sampling budget. For global approximation of the test
function in the whole input variable space, AS-C led to more samples than AS, and AS-
C was more accurate than AS in the large majority of DOEs.
Fourthly, this work explores the trade-off between exploration and replication for
the design of experiments. Previous work has shown that exploration is more effective
than replication especially when the functional form of response is complicated, but it
was based on simulated test data assuming independent and identically distributed
normal noise. In this work, we first tested the conclusion favoring exploration using
actual test data of strength of open hole composite plates as function of the ratio of
plate width to hole diameter, w/D. We proposed a resampling procedure based on
subsets of experimental strength to compare exploration and replication. The results
showed that testing all four configurations with fewer replicates led to more accurate
predictions than testing only three configurations with more replicates, thus confirming
the previous simulated results. Prediction of polynomial response surface(PRS) with
fixed order/terms had similar accuracy while increasing replicates for different
distributions. Introducing more replicates did not benefit prediction accuracy noticeably.
An exploration strategy enables a higher order PRS to be used, which is key to increase
prediction accuracy for complex structural behavior.
179
At the end, we investigated approximation of flow field and approximation of
scalar quantities. Field data is commonly encountered in engineering analysis (e.g. flow
field) and usually contains rich information. Reconstructing field data at varying
conditions has been extensively studied for effective design optimization. The field data
is assumed to be a linear combination of major modes. Proper orthogonal
decomposition (POD) is adopted here to extract major modes of field data. The mode
coefficients are then approximated using several surrogate models. We found that field
prediction using POD with surrogates is less capable to preserve extreme value such as
maximum pressure from a CFD analysis. In contrast, a surrogate model could
approximate maximum pressure accurately in variable space. Therefore, approximation
schemes for field should depend on the quantity of interest.
180
APPENDIX A ANALYTICAL EXPRESSIONS FOR DRAG COEFFICIENT
The analytic expression for drag coefficient function was cited from [55]. The
range of applicability for cf is 5Re 2 10 , M 1.75 , and Kn 0.01 , where Kn denotes
Knudsen number. 5Re =2 10 is the subcritical Reynolds number above which the
attached boundary layer becomes turbulent. Attention is limited to continuum flows,
namely, Kn<0.01. DC is given in Eq. (A-1) for subcritical ( crM M 0.6 ), supersonic (
1<M 1.75 ) and intermediate ( crM <M 1 ) Mach number regimes.
, , ,
,
,sup
(Re,M)
M(Re) (Re) (Re) if M M
M
(Re,M) if M M 1.0
(Re,M) if 1.0 M 1.75
cr
D
D std D M D std cr
cr
D sub cr
D
C
C C C
C
C
(A-1)
where
1
0.687
, 1.16
24 42500(Re) (1 0.15Re ) 0.42 1
Re ReD stdC
(A-2)
1
0.684
, 0.669
24 483(Re) 1 0.15Re 0.513 1
Re RecrD MC
(A-3)
,sup ,M 1 ,M 1.75 ,M 1 sup(Re,M) (Re) (Re) (Re) (M,Re)D D D DC C C C
(A-4)
0.813 1
,M 1 0.793
24 3550(Re) 1 0.118Re 0.69 1
Re ReDC
(A-5)
0.867 1
,M 1.75 0.634
24 861(Re) 1 0.107Re 0.646 1
Re ReDC
(A-6)
33
,sup
sup ,sup
1 ,sup ,sup1
log Re(M,Re) (M)
j
i
i j i i jj
Cf
C C
(A-7)
2 3
1,sup (M) 2.963 4.392M 1.169M 0.027M 0.233exp (1 M) / 0.011f (A-8)
2 3
2,sup (M) 6.617 12.11M 6.501M 1.182M 0.174exp (1 M) / 0.01f (A-9)
2 3
3,sup (M) 5.866 11.57M 6.665M 1.312M 0.350exp (1 M) / 0.012f (A-10)
1,sup 2,sup 3,sup6.48; 8.93; 12.21C C C (A-11)
181
,sub ,M ,M 1 ,M sub(Re,M) (Re) (Re) (Re) (M,Re)
cr crD D D DC C C C (A-12)
33
,sub
sub ,sub
1 ,sub ,sub1
log Re(M,Re) (M)
j
i
i j i i jj
Cf
C C
(A-13)
2 3
1,sub (M) 1.884 8.422M 13.70M +8.162Mf (A-14)
2 3
2,sub (M) 2.228 10.35M 16.96M +9.840Mf (A-15)
2 3
3,sub (M) 4.362 16.91M 19.84M -6.296Mf (A-16)
1,sub 2,sub 3,sub6.48; 9.28; 12.21C C C (A-17)
182
APPENDIX B ONE-DIMENSIONAL FUNCTION EXTRAPOLATION USING SURROGATES
Surrogate modeling is commonly used to estimate function values efficiently and
accurately at unsampled points. The estimation procedure is called interpolation when
target points are inside the convex hull of sampled points while extrapolation otherwise.
This work explores one-dimensional deterministic function extrapolation using
surrogates. We first define a new error metric, relative average error, for quantifying
overall performance of extrapolation technique. Ordinary Kriging and Linear Sheppard
surrogates proved to be safer on several challenging functions than polynomial
response surfaces, support vector regression or radial basis neural functions. This
reflected that prediction of these surrogates converge to mean value of samples at
points far from samples.
It’s commonly recognized that long-range extrapolation is likely to be less
accurate than short-range extrapolation. Two kinds of effective extrapolation distance
are defined to indicate how far we can extrapolate test functions. We propose using the
correlation between the nearest sample and the prediction point given by Ordinary
Kriging as indicator of effective extrapolation distance. The relationship between
effective extrapolation distance and corresponding correlation over the distance is
examined by several test functions. A large value of correlation is associated with
effective extrapolation distance.
B.1 Background
In surrogate modeling, it is common to sample a function f at several points and
fit them with an explicit function in order to estimate the function at other points[150].
This is often required for optimization or reliability analysis in which thousands of
183
function evaluations are common, and each sample often means expensive simulation
or costly or time-consuming experiment.
Function estimation is defined as interpolation when target points are inside the
convex hull of sampled data points and extrapolation otherwise. For one-dimensional
samples, convex hull is the smallest interval containing the samples. Although many
research results have been reported on the accuracy of surrogate modeling, most
focused on the prediction accuracy in interpolation. Extrapolation is usually associated
with large estimation errors [46] and commonly encountered in three situations:
1. Sampling pattern such as Latin Hypercube sampling is adopted, which typically does not sample at or near the boundaries of sampling region.
2. For function estimation in high-dimensional space, we usually cannot afford enough points to avoid extrapolation. For example, in twenty-dimensional box, more than million points (220) are required.
3. Region of interest changes after samples are collected[36].
Besides the above conditions, extrapolation may be useful when the target points
cannot be sampled via simulation or experiment due to the need to know future events,
inadequacy of simulation software or high cost to perform experiments[151]. As a first
step to explore effective extrapolation scheme in engineering problems, attention is
limited here to one-dimensional (1D) function extrapolation.
This work investigates general issues on extrapolation using surrogates. Section
B.2 illustrates possible behavior of surrogates for extrapolation and potentials of
extrapolation using surrogates. Section B.3 proposes an error metric designed for
extrapolation. Extrapolation of a few examples using five surrogates are compared in
Section B.4. Section B.5 discusses the possibility of estimating extrapolation distance
using surrogates.
184
B.2 Possible Behaviors of Extrapolation using Surrogates
Estimation of several analytical functions using ordinary kriging is presented to
illustrate possible behavior in the extrapolation region, which is assumed here to be
inaccessible. Figure B-1 presents three 1D functions which have different function
behavior between the accessible and inaccessible domains. These three functions are
estimated using Ordinary Kriging in Figure B-2. It is seen extrapolation results
approximate the true function value surprisingly well. Since extrapolation in Kriging is
based on correlation between function values based on distance, we evaluate first the
correlation between inaccessible domain and accessible domain.
Figure B-1. Three one-dimensional test functions for extrapolation. A) Forrester
function, B) ( ) sin( )f x x and C) 2( ) ( 6) /13 2f x x . Vertical line denotes
border of inaccessible domain which is on the left and accessible domain which is on the right.
Figure B-2. Extrapolation of three one-dimensional test functions. A) Forrester function,
B) ( ) sin( )f x x and C) 2( ) ( 6) /13 2f x x . Vertical line denotes border of
C B A
0 0.1 0.2 0.3 0.4 0.5-1
0
1
2
3
41D Forrester function
x
y
-2 0 2-1
-0.5
0
0.5
1sin(x)
x
y
0 2 4 6 8 10 12-1
-0.5
0
0.5
1
1.5
2Extrapolation of -(x-6)2/13+2
x
y
A B C
185
inaccessible domain which is on the left and accessible domain which is on the right.
Denote by r the ratio of length of extrapolation distance to that of accessible
domain. As expected, extrapolation error increases with r as shown in Figure B-3 for the
log function. In this work, the length of inaccessible domain and accessible domain are
set to be equal, which would typically be considered as long-range extrapolation.
Figure B-3. Extrapolation of the logarithmic function ( ) log( )f x x . A) Extrapolation of the
response using Kriging and B) absolute error for extrapolation.
Another factor determining the extrapolation accuracy is how close the samples
to the boundary are. Figure B-4 illustrates that by using samples close to the boundary.
Extrapolation accuracy improves obviously after shifting samples close to inaccessible
domain.
The examples and discussion above identified three issues, which are correlation
between inaccessible domain and accessible domain, relative extrapolation distance
and absolute extrapolation distance, which are important for extrapolation research.
B
0 2 4 6 8 10 12
-4
-2
0
2
4
6
8
x
f(x)
Extrapolation using Kriging
Border
True function
Extrapolation results
Samples
A
0 2 4 60
1
2
3
4
5
x
Absolu
te e
rror
Absolute error for extrapolation
00.20.40.60.81r
186
Figure B-4. Extrapolation of logarithmic function from different sampling domain
( ) log(12.8 )f x x . A) Extrapolation from the full sampling domain, B)
absolute error for extrapolation from the full sampling domain, C) extrapolation from the reduced sampling domain and D) absolute error for extrapolation from the reduced sampling domain.
B.3 An Error Metric for 1D Extrapolation
We denote extrapolation result by ˆ( )f x . The performance of one-dimensional
extrapolation technique may be quantified by various error measures. Relative error
cr ( )e x is
cr
ˆ ( ) ( )( )=
( )
f x f xe x
f x
(B-1)
cr ( )e x may be misleading when the function changes sign. So one often uses the
range of the function instead of the function value for normalization. In addition, for
C
B A
0 2 4 6 8 10 12
0
1
2
3
4
x
f(x)
Extrapolation using Kriging
Border
True function
Extrapolation results
Samples
0 2 4 6 8 10 12
0
1
2
3
4
x
f(x)
Extrapolation using Kriging
Border
True function
Extrapolation results
Samples
0 2 4 60
0.02
0.04
0.06
0.08
0.1
0.12
0.14
x
Absolu
te e
rror
Absolute error for extrapolation
0 1 2 3 4 5 60
0.5
1
1.5
2
x
Absolu
te e
rror
Absolute error for extrapolation
D
187
extrapolation, we are often interested in the error in predicting change from a boundary
point bx . This error, ( )ece x is:
ˆ ( ) ( )
( )= 1( ) ( )
bec
b
f x f xe x
f x f x
(B-2)
For example, if based on this year’s record we predict that gas prices will rise
from $4/gallon today to $5/gallon a year from now, and they rise only to $4.50, we may
consider ( )=100%ece x this as rather than cr ( )=11%e x error. Of course, this error measure
will fail if the change in the function is near zero, and for this case an alternate relative
error is defined as ( )re x . Denoting ( )range f as the range of true function in the
extrapolation domain, ( )re x is:
ˆ ( ) ( )
( )=( )
r
f x f xe x
range f
(B-3)
( )re x is used in the following for extrapolation comparisons. We may also use
1 1( ( )) max( ) min( )range f x f f , where 1f denotes function value in the range between
extrapolation point x and the closest sample. In order to evaluate the overall
performance of extrapolation, we use the average error AE in extrapolation domain:
( )b
t
x
r
x
b t
e x dx
AEx x
(B-4)
B.4 Surrogate Comparison for Extrapolation
Surrogates have different performance for interpolation and extrapolation. We
test the performance of five popular surrogates: Ordinary Kriging, Polynomial response
surface (PRS), Radial basis neural network (RBNN), Linear Shepard (moving least
188
squares), and Support vector regression (SVR). Four test functions were extracted from
well-known multidimensional functions taken from [5] and shown in Figure B-5.
Figure B-5. Four test functions for surrogate selection. Domains close to origin are
inaccessible domain. A) Forrester function, B) 1D Braining-Hoo function, C) (c) 1D Ackley function and D) 1D Gramacy & Lee function.
The number of sampling points along each line is 6. Sampling points are
generated using Latin hypercube sampling with 5 iterations, which introduces
randomness in the position of the samples. To average out the effect of the positions of
the sampling points, 30 sets of samples are generated for each test function, and the
mean value of AE for all the sample sets are computed. The extrapolation results of
test functions are listed in Table B-1. All the surrogates except Ordinary Kriging and
Linear Shepard can generate huge errors. Kriging and Linear Shepard do not
extrapolate well, but do not incur huge errors. This is because function estimation using
A B
C D
189
Ordinary Kriging and Linear Shepard are weighted sum of samples and they eventually
revert to the mean of the samples. Ordinary Kriging was then selected for further testing
in the next section.
Table B-1. Average AE for extrapolation of test functions using 30 sets of samples
Surrogate models
Test functions Average AE of Forrester et al.(2008)
function
Average AE of one-dimensional
Branin-Hoo function
Average AE of one-dimensional Ackley function
Average AE of one-dimensional Gramacy & Lee (2009) function
Kriging 0.80 0.13 0.31 1.44 Quadratic PRS 13.36 0.97 2.01 6.13
Cubic PRS 7.54 0.53 4.02 60.22 Quartic PRS 93.62 3.18 8.67 442.14
RBNN 2322.5 0.2 0.54 25612 Linear Shepard 0.44 0.19 0.56 1.85
SVR 20.7 0.26 0.76 188.08
B.5 Estimating Extrapolation Distance
Kriging is based on a correlation structure between points based on their
distance. Large correlation between extrapolation points to closest sample may indicate
reliable extrapolation. We defined two types of effective extrapolation distance and tried
to find the relation between extrapolation distances of test functions and corresponding
correlation over that extrapolation distance.
B.5.1 Effective Extrapolation Distance
Ordinary Kriging assumes that the error at a point is normally distributed with a
mean of zero and a given standard deviation. Error bounds here are set be 95%
confidence interval of this normal distribution. The conservative extrapolation distance d
is defined for measuring how far the error bounds of the surrogate bound the true
function.
190
The second extrapolation distance is denoted as accuracy distance. Accurate
distance is inside conservative distance and in which estimated error bounds of the
points are less than 30% of ( )range f . We make an exception to the requirement of
being within the error bounds when they are very tight, allowing error bounds to be off
by 1% of ( )range f ). Two types of effective extrapolation distance are illustrated in
Figure B-6.
Figure B-6. Two types of effective extrapolation distance. A) Conservative extrapolation
distance and B) accurate extrapolation distance.
B.5.2 Identification of Effective Extrapolation Distance using Correlation
The error estimates get less dependable as we go deeper into the extrapolation
domain. We tried to find certain indicators of effective extrapolation distance based on
Kriging. Prediction of Ordinary Kriging is based on the assumption that correlation
between function values decay with distance at a rate controlled by , with the
correlation r between two points at a distance l being equal to
2( ) expr l l (B-5)
is usually found by maximizing likelihood of observing that the samples come
from Gaussian process. Large means short wavelength, large curvature, fast changing
function, and reverse for small . It is reasonable to expect that as the correlation
0 2 4 6 8-20
0
20
40
60
80
100
120
x
f(x)
Extrapolation using Kriging
Border of inaccessible domain
Conservative bound
Accurate bound
True function
Extrapolation
Sampling points
95% confidence range
0 5 10 15 20 25
0
100
200
300
400
500
600
x
f(x)
Extrapolation using Kriging
Border of inaccessible domain
Conservative bound
Accurate bound
True function
Extrapolation
Sampling points
95% confidence range
A B
191
between function values in the sampling domain and extrapolation domain diminishes,
the reliability of the error estimates deteriorates. Distance corresponds to given small
correlation as possible measure of how far we can go.
We have performed a test to figure out the relation between effective
extrapolation distance and corresponding correlation value. 10 multi-dimensional
functions: Branin-Hoo function, Ackley function, Gramacy & Lee (2009) function,
Hartmann 3-D Function, Hartmann 6-D Function, Sasena Function, Friedman Function,
Zhou (1998) Function, Franke's Function, Dette & Pepelyshev (2010) curved Function.
Figure B-7. Correlation and distance ratio over effective distance while extrapolating 30
1D functions. A) Correlation over accurate distance, B) distance ratio of accurate distance to accessible domain, C) correlation over conservative distance and D) distance ratio of conservative distance to accessible domain.
These functions are commonly used for testing algorithm performance and can
be found from [5]. 3 lines are extracted towards one random vertex from each function.
We use 6 uniformly spaced samples to train Kriging. In Figure B-7, we present
B
0
0.2
0.4
0.6
0.8
1
1
Box plot of correlation at accurate bound
0
0.2
0.4
0.6
0.8
1
1
Box plot of distance ratio at accurate bound
0
0.2
0.4
0.6
0.8
1
1
Box plot of correlation at conservative bound
0
0.2
0.4
0.6
0.8
1
1.2
1
Box plot of distance ratio at conservative boundA
C D
192
corresponding correlation value between effective extrapolation bounds and closest
sample. Accurate bounds are associated with large correlation value. The third quartiles
of correlation values corresponding to accurate bounds and conservative bounds are
both 0.99. The box plots of distance ratios are dispersed and imply extrapolation
accuracy vary with functions.
B.6 Concluding Remarks
This work first illustrated the possibility of extrapolating 1D function using
surrogates and proposed an average error metric designed for quantifying the
performance of extrapolation technique. Testing extrapolation on several challenging
functions indicated that Kriging and Linear Shepard were safer than other surrogates.
We defined two types of effective extrapolation distance and correlation of Ordinary
Kriging has been demonstrated as a possible indicator for effective extrapolation
distance based on the tests of 30 one-dimensional functions.
193
APPENDIX C WEIBULL DISTRIBUTION WITH A HEAVY TAIL
The Weibull distribution was first shaped to follow ( , )W a b with a heavy tail to
amplify the difference with the normal distribution, where the scale parameter a=2, the
shape parameter b=1.5 as seen in Eq. (C-1). Then ( , )W a b was scaled through a linear
transformation to imitate the experimental variability of the analytical test function
according to Eq.(C-2), Eq.(C-3) and Eq.(C-4), where is the strength of the
analytical test function, 1( , )WF p a b is the inverse cumulative distribution at percentile p,
,W a b is the mean of ( , )W a b . ,al from Eq. (C-2) was adopted to scale up the
variability of the synthetic strength and ,bl in Eq.(C-3) was used to guarantee the
mean value of the synthetic strength is ,f .
1
/~ ,
bb
x ab kk W a b e
a a
(C-1)
1 1
0.2 ,,
(97.5 , ) (2.5 , )W W
fal
F a b F a b
(C-2)
, , , ,Wbl f al a b (C-3)
, , + ,s k al bl (C-4)
,f
194
APPENDIX D SPECIMEN STRENGTH SUMMARY FOR OHT TESTING
Table D-1. Specimen strength at w/d=3. Nominal thickness for one ply is 0.0086 inch.
plynorm
norm
tStrength Strength
t
Pre-preg
batch
number
Strength
[ksi]
Ave.
specimen
thickness
[in]
Number of
plies in
laminate
Normalized
Strength
[ksi]
Failure
Mode
1
61.768 0.167 20 60.111 LGM
59.168 0.168 20 57.689 LGM
55.954 0.169 20 54.913 LGM
57.743 0.170 20 57.228 LGM
57.041 0.169 20 56.146 LGM
60.357 0.170 20 59.625 LGM
2
66.554 0.165 20 63.710 LGM
69.057 0.165 20 66.327 LGM
62.830 0.166 20 60.717 LGM
67.037 0.167 20 65.185 LGM
63.317 0.168 20 61.697 LGM
63.750 0.168 20 62.225 LGM
3
58.432 0.171 20 58.149 LGM
54.575 0.171 20 54.269 LGM
63.433 0.170 20 62.819 LGM
59.470 0.170 20 58.605 LGM
56.688 0.168 20 55.249 LGM
59.979 0.169 20 58.817 LGM
195
Table D-2. Specimen strength at w/d=4. Nominal thickness for one ply is 0.0086 inch.
plynorm
norm
tStrength Strength
t
Pre-
preg
batch
number
Strength [ksi] Ave.
specimen
thickness
[in]
Number of
plies in
laminate
Normalized
Strength
[ksi]
Failure
Mode
1
69.734 0.165 20 66.903 LGM
66.548 0.166 20 64.040 LGM
64.929 0.167 20 62.878 LGM
67.434 0.168 20 65.950 LGM
64.613 0.168 20 63.248 LGM
60.781 0.169 20 59.603 LGM
2
67.778 0.167 20 65.683 LGM
67.386 0.168 20 65.707 LGM
65.272 0.169 20 64.052 LGM
68.817 0.170 20 68.070 LGM
63.226 0.171 20 62.767 LGM
65.002 0.171 20 64.706 LGM
3
62.466 0.166 20 60.336 LGM
65.718 0.167 20 63.763 LGM
60.186 0.168 20 58.658 LGM
61.175 0.169 20 60.138 LGM
63.366 0.170 20 62.678 LGM
59.326 0.171 20 58.815 LGM
196
Table D-3. Specimen strength at w/d=6. Nominal thickness for one ply is 0.0086 inch.
plynorm
norm
tStrength Strength
t
Pre-preg
batch
number
Strength
[ksi]
Ave.
specimen
thickness
[in]
Number of
plies in
laminate
Normalized
Strength
[ksi]
Failure
Mode
1
65.081 0.171 20 70.113 LGM
70.862 0.170 20 64.881 LGM
66.084 0.169 20 66.797 LGM
68.646 0.167 20 64.996 LGM
66.182 0.169 20 64.522 LGM
65.173 0.170 20 68.600 LGM
2
63.417 0.168 20 68.077 LGM
69.299 0.169 20 71.889 LGM
72.877 0.170 20 69.745 LGM
70.050 0.171 20 69.875 LGM
73.024 0.165 20 68.162 LGM
70.541 0.166 20 71.016 LGM
3
65.243 0.164 20 62.379 LGM
68.869 0.165 20 66.039 LGM
67.953 0.166 20 65.570 LGM
67.890 0.165 20 65.225 LGM
64.027 0.167 20 62.321 LGM
64.980 0.169 20 63.986 LGM
197
Table D-4. Specimen strength at w/d=8. Nominal thickness for one ply is 0.0086 inch.
plynorm
norm
tStrength Strength
t
Pre-preg
batch
number
Strength
[ksi]
Ave.
specimen
thickness
[in]
Number of
plies in
laminate
Normalized
Strength
[ksi]
Failure
Mode
1
66.696 0.170 20 65.746 LGM
71.789 0.167 20 69.842 LGM
66.359 0.171 20 65.870 LGM
69.543 0.171 20 69.017 LGM
69.239 0.170 20 68.541 LGM
65.602 0.169 20 64.508 LGM
2
69.609 0.167 20 67.484 LGM
72.620 0.168 20 71.072 LGM
77.366 0.170 20 76.242 LGM
67.859 0.169 20 66.754 LGM
73.933 0.169 20 72.522 LGM
71.999 0.168 20 70.185 LGM
3
74.593 0.169 20 73.219 LGM
68.716 0.170 20 67.784 LGM
70.205 0.170 20 69.300 LGM
69.854 0.169 20 68.486 LGM
75.330 0.167 20 73.002 LGM
68.976 0.168 20 67.219 LGM
198
APPENDIX E VARIABILITY FOR PREDICTION ACCURACY WHEN COEFFICIENT OF VARIATION
IS 0.2
Figure E-1. Approximating the strength of composite laminate with the trade-off
between exploration and replication. Variability of mean RMSE with normal noise, coefficient of variation is 0.2. A) 4×4 test matrix, B) 5×5 test matrix, C) 6×6 test matrix and D) 7×7 test matrix.
A
0 50 100 1500.1
0.2
0.3
0.4
0.5
Composite:Variability of RMSE with normal noise
Number of samples
RM
SE
4*4 test matrix
95% Confi. Int.
0 50 100 150 2000.1
0.2
0.3
0.4
0.5
Composite:Variability of RMSE with normal noise
Number of samples
RM
SE
5*5 test matrix
95% Confi. Int.
0 100 200 3000.1
0.2
0.3
0.4
0.5
Composite:Variability of RMSE with normal noise
Number of samples
RM
SE
6*6 test matrix
95% Confi. Int.
0 100 200 300 400
0.1
0.2
0.3
0.4
0.5
Composite:Variability of mean RMSE with normal noise
Number of samples
RM
SE
7*7 test matrix
95% Confi. Int.
B
C D
199
LIST OF REFERENCES
[1] Mack, Y., Goel, T., Shyy, W., Haftka, R., and Queipo, N., "Multiple surrogates for the shape optimization of bluff body-facilitated mixing," Proc. Proceedings of the 43rd AIAA Aerospace Sciencis Meeting and Exhibit, pp. 10-13.
[2] Goel, T., Haftka, R. T., Shyy, W., and Queipo, N. V., 2007, "Ensemble of surrogates," Structural and Multidisciplinary Optimization, 33(3), pp. 199-216.
[3] Viana, F. A., Haftka, R. T., and Steffen Jr, V., 2009, "Multiple surrogates: how cross-validation errors can help us to obtain the best predictor," Structural and Multidisciplinary Optimization, 39(4), pp. 439-457.
[4] Hoeting, J. A., Madigan, D., Raftery, A. E., and Volinsky, C. T., 1999, "Bayesian model averaging: a tutorial," Statistical science, pp. 382-401.
[5] Zhang, Y., Kim, N. H., Park, C., and Haftka, R. T., "Function Extrapolation at One Inaccessible Point Using Converging Lines," Proc. ASME 2015 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers, pp. V02BT03A027-V002BT003A027.
[6] Zhang, Y., Park, C., Kim, N. H., and Haftka, R. T., 2017, "Function prediction at one inaccessible point using converging lines," Journal of Mechanical Design, 139(5), p. 051402.
[7] Y. Zhang, N. H. K., C. Park, R. T. Haftka, 2015, "One-dimensional Function Extrapolation Using Surrogates," 11th World Congress on Structural and Multidisciplinary OptimizationSydney, Australia.
[8] Zhang, Y., Kim, N.-H., Park, C., and Haftka, R. T., 2017, "Multi-Fidelity Surrogate Based on Single Linear Regression," ArXiv e-prints.
[9] Zhang, Y., Neelakantan, A., Kumar, N., Park, C., Haftka, R. T., Kim, N. H., and Lam, H., "Multi-fidelity Surrogate Modeling for Application/Architecture Co-design," Proc. International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems, Springer, pp. 179-196.
[10] Yiming Zhang, J. M., Jaco Schutte, Nam-ho Kim, Raphael T. Haftka, 2017, "Predicting B-Basis Allowable at Untested Points from Experiments and Simulations of Plates with Holes," 12th World Congress on Structural and Multidisciplinary OptimizationGermany.
[11] Zhang, Y., Meeker, J., Schutte, J., Kim, N., and Haftka, R., "On Approaches to Combine Experimental Strength and Simulation with Application to Open-Hole-Tension Configuration," Proc. Proceedings of the American Society for Composites: Thirty-First Technical Conference.
200
[12] Zhang, Y., Kim, N. H., Park, C., and Haftka, R. T., "Effect of Varying Test Cost on Design of Experiments," Proc. 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, p. 1163.
[13] Zhang, Y., Haftka, R. T., Kim, N.-H., Schutte, J. F., and Seneviratne, W. P., "Allocation of Samples Between Exploration and Replication for Open-Hole-Tension Test," Proc. American Society of Composites-30th Technical Conference.
[14] Zhang, Y., Schutte, J. F., Seneviratne, W. P., Kim, N. H., and Haftka, R. T., 2017, "Sampling by Exploration and Replication for Estimating Experimental Strength of Composite Structures," AIAA Journal.
[15] Kerschen, G., Golinval, J.-c., Vakakis, A. F., and Bergman, L. A., 2005, "The method of proper orthogonal decomposition for dynamical characterization and order reduction of mechanical systems: an overview," Nonlinear dynamics, 41(1-3), pp. 147-169.
[16] Bui-Thanh, T., Damodaran, M., and Willcox, K. E., 2004, "Aerodynamic data reconstruction and inverse design using proper orthogonal decomposition," AIAA journal, 42(8), pp. 1505-1516.
[17] Willcox, K., 2006, "Unsteady flow sensing and estimation via the gappy proper orthogonal decomposition," Computers & fluids, 35(2), pp. 208-226.
[18] Galbally, D., Fidkowski, K., Willcox, K., and Ghattas, O., 2010, "Non-linear model reduction for uncertainty quantification in large-scale inverse problems."
[19] Jeong, S., Solenthaler, B., Pollefeys, M., and Gross, M., 2015, "Data-driven fluid simulations using regression forests," ACM Transactions on Graphics (TOG), 34(6), p. 199.
[20] Gunes, H., Sirisup, S., and Karniadakis, G. E., 2006, "Gappy data: To Krig or not to Krig?," Journal of Computational Physics, 212(1), pp. 358-382.
[21] Qiu, Y., and Bai, J., 2015, "Stationary flow fields prediction of variable physical domain based on proper orthogonal decomposition and kriging surrogate model," Chinese Journal of Aeronautics, 28(1), pp. 44-56.
[22] Luo, J., Tang, X., Duan, Y., and Liu, F., "An Iterative Inverse Design Method of Turbomachinery Blades by Using Proper Orthogonal Decomposition," Proc. ASME Turbo Expo 2015: Turbine Technical Conference and Exposition, American Society of Mechanical Engineers, pp. V02BT39A026-V002BT039A026.
[23] Carlberg, K., and Farhat, C., 2009, "An adaptive POD-Krylov reduced-order modeling framework for repeated analyses problems."
201
[24] Yiming Zhang, N.-H. K., Chan-Young Park , Raphael T. Haftka, 2015, "One-dimensional Function Extrapolation Using Surrogates," 11th World Congress on Structural and Multidisciplinary OptimisationSydney Australia.
[25] Zhang, Y., Kim, N. H., Park, C., and Haftka, R. T., "Function Extrapolation of Noisy Data using Converging Lines," Proc. AIAA Modeling and Simulation Technologies Conference, p. 2144.
[26] Zhang, Y., Kim, N.-H., Park, C., and Haftka, R. T., 2017, "Multi-Fidelity Surrogate Based on Single Linear Regression," arXiv preprint arXiv:1705.02956.
[27] yiming Zhang, N.-H. K., Chan-Young Park , Raphael T. Haftka, 2018, "A Strategy for Adaptive Sampling When Sampling Cost Is Variable In Design Space," Asian Congress of Structural and Multidisciplinary optimization (abstract only)Dalian, China.
[28] yiming Zhang, N.-H. K., Chan-Young Park , Raphael T. Haftka, 2018 "A Strategy for Adaptive Sampling When Sampling Cost Is Variable In Design Space," AIAA Journal (submitted).
[29] Y. Zhang, R. T. H., N.H. Kim, W. Seneviartne and J. F. Schutte, 2015, "Allocation of Samples between Exploration and Replication for Open-Hole-Tension Test," Proceedings of the American Society of Composites- 30th Annual Technical ConferenceEast Lansing, Michigan.
[30] Queipo, N. V., Haftka, R. T., Shyy, W., Goel, T., Vaidyanathan, R., and Tucker, P. K., 2005, "Surrogate-based analysis and optimization," Progress in aerospace sciences, 41(1), pp. 1-28.
[31] Qian, Z., Seepersad, C. C., Joseph, V. R., Allen, J. K., and Wu, C. J., 2006, "Building surrogate models based on detailed and approximate simulations," Journal of Mechanical Design, 128(4), pp. 668-677.
[32] Brooks, D. G., Carroll, S. S., and Verdini, W. A., 1988, "Characterizing the domain of a regression model," The American Statistician, 42(3), pp. 187-190.
[33] Barnard, E., and Wessels, L., 1992, "Extrapolation and interpolation in neural network classifiers," Control Systems, IEEE, 12(5), pp. 50-53.
[34] Shan, S., and Wang, G. G., 2010, "Metamodeling for high dimensional simulation-based design problems," Journal of Mechanical Design, 132(5), p. 051009.
[35] Glaz, B., Goel, T., Liu, L., Friedmann, P. P., and Haftka, R. T., 2007, "Application of a weighted average surrogate approach to helicopter rotor blade vibration reduction," AIAA Paper, 1898.
202
[36] Balabanov, V., Weckner, O., and Wu, J., 2014, "Reducing Error of Polynomial Approximation Outside of Designated Design Space for Practical Problems."
[37] Hooker, G., and Rosset, S., "DARE: Data-Augmented Regression for Extrapolation."
[38] Wilson, A. G., and Adams, R. P., "Gaussian Process Kernels for Pattern Discovery and Extrapolation," Proc. ICML (3), pp. 1067-1075.
[39] Roache, P. J., 2009, Fundamentals of verification and validation, hermosa publ.
[40] Sidi, A., 2003, Practical extrapolation methods: Theory and applications, Cambridge University Press.
[41] Hooker, G., 2004, "Diagnostics and extrapolation in machine learning," stanford university.
[42] An, D., Kim, N. H., and Choi, J.-H., 2015, "Practical options for selecting data-driven or physics-based prognostics algorithms with reviews," Reliability Engineering & System Safety, 133, pp. 223-236.
[43] Loh, W.-Y., Chen, C.-W., and Zheng, W., 2007, "Extrapolation errors in linear model trees," ACM Transactions on Knowledge Discovery from Data (TKDD), 1(2), p. 6.
[44] George, J. J., 2014, Weather forecasting for aeronautics, Academic press.
[45] Rapach, D. E., and Zhou, G., 2013, "Forecasting stock returns," Handbook of Economic Forecasting, 2(Part A), pp. 328-383.
[46] Chaudhuri, A., and Haftka, R. T., "Optimistic Bias in Surrogate Prediction near Surrogate Optima."
[47] Durelli, A., and Rajaiah, K., 1981, "Quasi-square hole with optimum shape in an infinite plate subjected to in-plane loading," Journal of Mechanical Design, 103(4), pp. 866-870.
[48] Stein, M., 1987, "Large sample properties of simulations using Latin hypercube sampling," Technometrics, 29(2), pp. 143-151.
[49] Rasmussen, C. E., 2006, "Gaussian processes for machine learning."
[50] Viana, F., and Goel, T., 2010, "SURROGATES toolbox user’s guide," ht tp://fchegury. googlepages. com,[retrieved Dec. 2009].
[51] Bernardo, J., Berger, J., Dawid, A., and Smith, A., 1992, "Some Bayesian Numerical Analysis," Bayesian Statistics, 4.
203
[52] Murphy, K. P., 2007, "Conjugate Bayesian analysis of the Gaussian distribution," def, 1(2σ2), p. 16.
[53] Bingham, S. S. D., 2016, "virtual library of simulation experiments: test functions and datasets."
[54] Dette, H., and Pepelyshev, A., 2012, "Generalized Latin hypercube design for computer experiments," Technometrics.
[55] Parmar, M., Haselbacher, A., and Balachandar, S., 2010, "Improved drag correlation for spheres and application to shock-tube experiments," Aiaa Journal, 48(6), pp. 1273-1276.
[56] Forrester, A. I., and Keane, A. J., 2009, "Recent advances in surrogate-based optimization," Progress in Aerospace Sciences, 45(1), pp. 50-79.
[57] Frangos, M., Marzouk, Y., Willcox, K., and van Bloemen Waanders, B., 2010, "Surrogate and reduced-order modeling: a comparison of approaches for large-scale statistical inverse problems [Chapter 7]."
[58] Fernández-Godino, M. G., Park, C., Kim, N.-H., and Haftka, R. T., 2016, "Review of multi-fidelity models," arXiv preprint arXiv:1609.07196.
[59] Peherstorfer, B., Willcox, K., and Gunzburger, M., 2016, "Survey of multifidelity methods in uncertainty propagation, inference," and optimization. Technical Report TR-16-1. Aerospace Computational Design Laboratory, Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA, USA.
[60] Balabanov, V., Grossman, B., Watson, L., Mason, W., and Haftka, R., "Multifidelity response surface model for HSCT wing bending material weight," Proc. 7th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, p. 4804.
[61] Kennedy, M. C., and O'Hagan, A., 2000, "Predicting the output from a complex computer code when fast approximations are available," Biometrika, 87(1), pp. 1-13.
[62] Qian, P. Z., and Wu, C. J., 2008, "Bayesian hierarchical modeling for integrating low-accuracy and high-accuracy experiments," Technometrics, 50(2), pp. 192-204.
[63] Han, Z., Zimmerman, R., and Görtz, S., 2012, "Alternative cokriging method for variable-fidelity surrogate modeling," AIAA journal, 50(5), pp. 1205-1210.
[64] Kuya, Y., Takeda, K., Zhang, X., and Forrester, A. I., 2011, "Multifidelity surrogate modeling of experimental and computational aerodynamic data sets," AIAA journal, 49(2), p. 289.
204
[65] Park, C., Kim, N. H., and Haftka, R. T., "Including ρ in Multi-Fidelity Surrogate Prediction Can Make Discrepancy Extrapolation Accurate by Reducing Bumpiness," Proc. 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, p. 0915.
[66] Kennedy, M. C., and O’Hagan, A., 2001, "Supplementary details on Bayesian calibration of computer models," Internal Report. URL http://www. shef. ac. uk/~ st1ao/ps/calsup. ps.
[67] Higdon, D., Kennedy, M., Cavendish, J. C., Cafeo, J. A., and Ryne, R. D., 2004, "Combining field data and computer simulations for calibration and prediction," SIAM Journal on Scientific Computing, 26(2), pp. 448-466.
[68] Bayarri, M. J., Berger, J. O., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., Lin, C.-H., and Tu, J., 2007, "A framework for validation of computer models," Technometrics, 49(2), pp. 138-154.
[69] Wang, L., Fang, X., Subramaniyan, A., Jothiprasad, G., Gardner, M., Kale, A., Akkaram, S., Beeson, D., Wiggs, G., and Nelson, J., "Challenges in uncertainty, calibration, validation and predictability of engineering analysis models," Proc. ASME 2011 Turbo Expo: Turbine Technical Conference and Exposition, American Society of Mechanical Engineers, pp. 747-758.
[70] Sankararaman, S., Ling, Y., and Mahadevan, S., 2011, "Uncertainty quantification and model validation of fatigue crack growth prediction," Engineering Fracture Mechanics, 78(7), pp. 1487-1504.
[71] Jin, R., Du, X., and Chen, W., 2003, "The use of metamodeling techniques for optimization under uncertainty," Structural and Multidisciplinary Optimization, 25(2), pp. 99-116.
[72] Viana, F., 2010, "SURROGATES toolbox user’s guide," Gainesville, FL, USA. [73] Le Gratiet, L., 2013, "Multi-fidelity Gaussian process regression for computer
experiments," Université Paris-Diderot-Paris VII.
[74] Park, C., Haftka, R. T., and Kim, N. H., 2016, "Remarks on multi-fidelity surrogates," Structural and Multidisciplinary Optimization, pp. 1-22.
[75] Campbell, K., 2002, "Exploring Bayesian model calibration: a guide to intuition," Los Alamos Technical Report LA-UR-02-7175.
[76] Swiler, L. P., and Trucano, T. G., 2005, "Calibration Under Uncertainty," Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States).
[77] Matsumura, T., Haftka, R. T., and Kim, N. H., 2015, "Accurate predictions from noisy data: replication versus exploration with applications to structural failure," Structural and Multidisciplinary Optimization, pp. 1-18.
205
[78] Dewettinck, K., De Visscher, A., Deroo, L., and Huyghebaert, A., 1999, "Modeling the steady-state thermodynamic operation point of top-spray fluidized bed processing," Journal of Food Engineering, 39(2), pp. 131-143.
[79] Le Gratiet, L., 2013, "Bayesian analysis of hierarchical multifidelity codes," SIAM/ASA Journal on Uncertainty Quantification, 1(1), pp. 244-269.
[80] Currin, C., Mitchell, T., Morris, M., and Ylvisaker, D., 1991, "Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments," Journal of the American Statistical Association, 86(416), pp. 953-963.
[81] Xiong, S., Qian, P. Z., and Wu, C. J., 2013, "Sequential design and analysis of high-accuracy and low-accuracy computer codes," Technometrics, 55(1), pp. 37-46.
[82] Couckuyt, I., Dhaene, T., and Demeester, P., 2014, "ooDACE toolbox: a flexible object-oriented Kriging implementation," Journal of Machine Learning Research, 15, pp. 3183-3186.
[83] Barrett, R. F., Borkar, S., Dosanjh, S. S., Hammond, S. D., Heroux, M. A., Hu, X. S., Luitjens, J., Parker, S. G., Shalf, J., and Tang, L., 2013, "On the role of co-design in high performance computing," vol, 24, pp. 141-155.
[84] Heroux, M. A., Doerfler, D. W., Crozier, P. S., Willenbring, J. M., Edwards, H. C., Williams, A., Rajan, M., Keiter, E. R., Thornquist, H. K., and Numrich, R. W., 2009, "Improving performance via mini-applications," Sandia National Laboratories, Tech. Rep. SAND2009-5574, 3.
[85] Kumar, N., Pascoe, C., Hajas, C., Lam, H., Stitt, G., and George, A., "Behavioral emulation for scalable design-space exploration of algorithms and architectures," Proc. International Conference on High Performance Computing, Springer, pp. 5-17.
[86] Banerjee, T., Hackl, J., Shringarpure, M., Islam, T., Balachandar, S., Jackson, T., and Ranka, S., "CMT-Bone—A Proxy Application for Compressible Multiphase Turbulent Flows," Proc. High Performance Computing (HiPC), 2016 IEEE 23rd International Conference on, IEEE, pp. 173-182.
[87] Barrett, R. F., Vaughan, C. T., and Heroux, M. A., 2011, "MiniGhost: a miniapp for exploring boundary exchange strategies using stencil computations in scientific parallel computing," Sandia National Laboratories, Tech. Rep. SAND, 5294832.
[88] Karlin, I., Keasler, J., and Neely, J., 2013, "Lulesh 2.0 updates and changes," Lawrence Livermore National Laboratory (LLNL), Livermore, CA.
206
[89] Kumar, N., Sringarpure, M., Banerjee, T., Hackl, J., Balachandar, S., Lam, H., George, A., and Ranka, S., "CMT-bone: a mini-app for compressible multiphase turbulence simulation software," Proc. Cluster Computing (CLUSTER), 2015 IEEE International Conference on, IEEE, pp. 785-792.
[90] Kennedy, M. C., and O'Hagan, A., 2001, "Bayesian calibration of computer models," Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(3), pp. 425-464.
[91] Dosanjh, S. S., Barrett, R. F., Doerfler, D., Hammond, S. D., Hemmert, K. S., Heroux, M. A., Lin, P. T., Pedretti, K. T., Rodrigues, A. F., and Trucano, T., 2014, "Exascale design space exploration and co-design," Future Generation Computer Systems, 30, pp. 46-58.
[92] Barrett, R. F., Crozier, P. S., Doerfler, D., Heroux, M. A., Lin, P. T., Thornquist, H., Trucano, T., and Vaughan, C. T., 2015, "Assessing the role of mini-applications in predicting key performance characteristics of scientific and engineering applications," Journal of Parallel and Distributed Computing, 75, pp. 107-122.
[93] Xiong, Y., Chen, W., and Tsui, K.-L., 2008, "A new variable-fidelity optimization framework based on model fusion and objective-oriented sequential sampling," Journal of Mechanical Design, 130(11), p. 111401.
[94] Ellis, M., and Mathews, E., 2001, "A new simplified thermal design tool for architects," Building and environment, 36(9), pp. 1009-1021.
[95] Zheng, L., Hedrick, T. L., and Mittal, R., 2013, "A multi-fidelity modelling approach for evaluation and optimization of wing stroke aerodynamics in flapping flight," Journal of Fluid Mechanics, 721, pp. 118-154.
[96] Jin, R., Chen, W., and Sudjianto, A., 2005, "An efficient algorithm for constructing optimal design of computer experiments," Journal of Statistical Planning and Inference, 134(1), pp. 268-287.
[97] Huang, D., Allen, T., Notz, W., and Miller, R., 2006, "Sequential kriging optimization using multiple-fidelity evaluations," Structural and Multidisciplinary Optimization, 32(5), pp. 369-382.
[98] Fischer, P. F., Lottes, J. W., and Kerkemeier, S. G., 2008, "nek5000 Web page," Web page: http://nek5000. mcs. anl. gov.
[99] Supercomputer, V., "LLNL webpage."
[100] Knight Jr, N. F., and Reeder, J. R., 2006, "User-defined material model for progressive failure analysis."
207
[101] Park, C., Haftka, R. T., and Kim, N.-H., "Experience with Several Multi-fidelity Surrogate Frameworks."
[102] Vitali, R., Haftka, R. T., and Sankar, B. V., 2002, "Multi-fidelity design of stiffened composite panel with a crack," Structural and Multidisciplinary Optimization, 23(5), pp. 347-356.
[103] Whitney, J. M., and Nuismer, R., 1974, "Stress fracture criteria for laminated composites containing stress concentrations," Journal of composite materials, 8(3), pp. 253-265.
[104] Hallett, S. R., Green, B. G., Jiang, W.-G., Cheung, K. H., and Wisnom, M. R., 2009, "The open hole tensile test: a challenge for virtual testing of composites," International Journal of Fracture, 158(2), pp. 169-181.
[105] Nuismer, R., and Whitney, J. M., 1975, "Uniaxial failure of composite laminates containing stress concentrations," Fracture mechanics of composites, ASTM International.
[106] Bhachu, K. S., Haftka, R. T., and Kim, N. H., 2016, "Comparison of Methods for Calculating B-Basis Crack Growth Life Using Limited Tests," AIAA Journal, pp. 1-12.
[107] Romero, V., Swiler, L., and Urbina, A., "An initial comparison of methods for representing and aggregating experimental uncertainties involving sparse data," Proc. 13th AIAA Non-Deterministic Approaches Conference.
[108] Romero, V., Swiler, L., Urbina, A., and Mullins, J., 2013, "A Comparison of Methods for Representing Sparsely Sampled Random Quantities," Sandia National Laboratories report SAND2013-4561 printed September.
[109] Haftka, R. T., and Gürdal, Z., 2012, Elements of structural optimization, Springer Science & Business Media.
[110] Shan, S., and Wang, G. G., 2010, "Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions," Structural and Multidisciplinary Optimization, 41(2), pp. 219-241.
[111] Carlsson, L. A., Adams, D. F., and Pipes, R. B., 2014, Experimental characterization of advanced composite materials, CRC press.
[112] Chen, S., Jiang, Z., Yang, S., Apley, D. W., and Chen, W., 2015, "Nonhierarchical multi‐model fusion using spatial random processes," International Journal for Numerical Methods in Engineering.
[113] Young, D. S., 2010, "Tolerance: an R package for estimating tolerance intervals," Journal of Statistical Software, 36(5), pp. 1-39.
208
[114] Draper, N. R., and Smith, H., 2014, Applied regression analysis, John Wiley & Sons.
[115] Bhachu, K. S., 2015, "Effects of measured data uncertainty on damage tolerant design of aircraft structures," University of Florida.
[116] Wang, G. G., and Shan, S., 2007, "Review of metamodeling techniques in support of engineering design optimization," Journal of Mechanical design, 129(4), pp. 370-380.
[117] Viana, F. A., 2011, "Multiple surrogates for prediction and optimization," University of Florida.
[118] Jin, R., Chen, W., and Sudjianto, A., "On sequential sampling for global metamodeling in engineering design," Proc. ASME 2002 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers, pp. 539-548.
[119] Salem, M. B., Roustant, O., Gamboa, F., and Tomaso, L., 2015, "Universal Prediction Distribution for Surrogate Models," arXiv preprint arXiv:1512.07560.
[120] Xu, S., Liu, H., Wang, X., and Jiang, X., 2014, "A Robust Error-Pursuing Sequential Sampling Approach for Global Metamodeling Based on Voronoi Diagram and Cross Validation," Journal of Mechanical Design, 136(7), p. 071009.
[121] Rudolph, D., and Stitt, G., "An interpolation-based approach to multi-parameter performance modeling for heterogeneous systems," Proc. Application-specific Systems, Architectures and Processors (ASAP), 2015 IEEE 26th International Conference on, IEEE, pp. 174-180.
[122] Banerjee, T., Gadou, M., and Ranka, S., 2016, "A genetic algorithm based approach for multi-objective hardware/software co-optimization," Sustainable Computing: Informatics and Systems, 10, pp. 36-47.
[123] Jones, D. R., Schonlau, M., and Welch, W. J., 1998, "Efficient global optimization of expensive black-box functions," Journal of Global optimization, 13(4), pp. 455-492.
[124] Chaudhuri, A., 2014, "Implications of optimization cost for balancing exploration and exploitation in global search and for experimental optimization," University of Florida.
[125] J. Forrester, A. I., Keane, A. J., and Bressloff, N. W., 2006, "Design and analysis of" Noisy" computer experiments," AIAA journal, 44(10), pp. 2331-2339.
209
[126] Ramu, P., Kim, N. H., Haftka, R. T., and Queipo, N. V., "System reliability analysis using tail modeling," Proc. Proceedings of the 11th AIAA/ISSMO Multidisciplinary Analysis & Optimization Conference.
[127] Tsai, S. W., and Wu, E. M., 1971, "A general theory of strength for anisotropic materials," Journal of composite materials, 5(1), pp. 58-80.
[128] Hinton, M. J., Kaddour, A. S., and Soden, P. D., 2004, Failure criteria in fibre reinforced polymer composites: the world-wide failure exercise, Elsevier.
[129] Hinton, M., and Kaddour, A., 2013, "Triaxial test results for fibre-reinforced composites: The Second World-Wide Failure Exercise benchmark data," Journal of Composite Materials, 47(6-7), pp. 653-678.
[130] LLorca, J., González, C., Molina‐Aldareguía, J. M., Segurado, J., Seltzer, R., Sket, F., Rodríguez, M., Sádaba, S., Muñoz, R., and Canal, L. P., 2011, "Multiscale modeling of composite materials: a roadmap towards virtual testing," Advanced Materials, 23(44), pp. 5130-5147.
[131] Hallett, S. R., Jiang, W.-G., and Wisnom, M. R., 2009, "Effect of stacking sequence on open-hole tensile strength of composite laminates," AIAA journal, 47(7), pp. 1692-1699.
[132] Rice, R. C., et al. , 2003, "Development of MMPDS Handbook Aircraft Design Allowables," Prepared for the 7th Joint DOD/FAA/NASA Conference on Aging Aircraft.
[133] Tomblin, J., Sherraden, J., Seneviratne, W., and Raju, K., 2002, "A-basis and B-basis design allowables for epoxy-based prepreg Toray T700SC-12K-50C/# 2510 plain weave fabric," Advanced General Aviation Transport Experiments, AGATEWP3, pp. 3-033051.
[134] Mehrez, L., Moens, D., and Vandepitte, D., 2012, "Stochastic identification of composite material properties from limited experimental databases, part I: Experimental database construction," Mechanical systems and signal processing, 27, pp. 471-483.
[135] Jordan J. Handler, J. S. D., Salvatore Liguore, Andrew Parrish and Steven Wanthal, "Development of a High Fidelity Test Database for Failure Prediction Methods Validation," Proc. Proceedings of the American Society for Composites 2014-Twenty-ninth Technical Conference on Composite Materials.
[136] Glaz, B., Goel, T., Liu, L., Friedmann, P. P., and Haftka, R. T., 2009, "Multiple-surrogate approach to helicopter rotor blade vibration reduction," AIAA Journal, 47(1), pp. 271-282.
[137] Chaudhuri, A., and Haftka, R. T., 2014, "Efficient global optimization with adaptive target setting," AIAA Journal, 52(7), pp. 1573-1578.
210
[138] Mackman, T., Allen, C., Ghoreyshi, M., and Badcock, K., 2013, "Comparison of adaptive sampling methods for generation of surrogate aerodynamic models," AIAA journal, 51(4), pp. 797-808.
[139] Matsumura, T., 2013, Effective safety measures with tests followed by design correction for aerospace structures.
[140] Tomblin, J., and Seneviratne, W., 2009, Laminate Statistical Allowable Generation for Fiber-Reinforced Composite Materials: Lamina Variability Method, Office of Aviation Research and Development, Federal Aviation Administration.
[141] International, A., 2011, "ASTM D5766 / D5766M-11, . Standard Test Method for Open-Hole Tensile Strength of Polymer Matrix Composite Laminates,"West Conshohocken, PA.
[142] Dirikolu, M. H., Aktaş, A., and Birgören, B., 2001, "Statistical analysis of fracture strength of composite materials using Weibull distribution," Turkish Journal of Engineering and Environmental Sciences, 26(1), pp. 45-48.
[143] Deo, R. B., and Saff, C. R., 1996, Composite materials: Testing and design, ASTM International.
[144] Wisnom, M., Khan, B., and Hallett, S., 2008, "Size effects in unnotched tensile strength of unidirectional and quasi-isotropic carbon/epoxy composites," Composite Structures, 84(1), pp. 21-28.
[145] Mooney, C. Z., Duval, R. D., and Duval, R., 1993, Bootstrapping: A nonparametric approach to statistical inference, Sage.
[146] Chakravarti, I. M., and Laha, R. G., 1967, "Handbook of methods of applied statistics," Handbook of methods of applied statistics, John Wiley & Sons.
[147] Welch, B. L., 1947, "The generalization ofstudent's' problem when several different population variances are involved," Biometrika, 34(1/2), pp. 28-35.
[148] Tryggvason, G., 2012, "A Front-tracking/Finite-Volume Navier-Stokes Solver for Direct Numerical Simulations of Multiphase Flows," October, 19, pp. 11-30p.
[149] Bendsoe, M. P., and Sigmund, O., 2013, Topology optimization: theory, methods, and applications, Springer Science & Business Media.
[150] Queipo, N. V., Haftka, R. T., Shyy, W., Goel, T., Vaidyanathan, R., and Kevin Tucker, P., 2005, "Surrogate-based analysis and optimization," Progress in aerospace sciences, 41(1), pp. 1-28.
[151] An, D., Choi, J.-H., Kim, N. H., and Pattabhiraman, S., 2011, "Fatigue life prediction based on Bayesian approach to incorporate field data into probability model," Structural engineering & mechanics, 37(4), p. 427.
211
BIOGRAPHICAL SKETCH
Yiming Zhang received his Ph.D. in the Department of Mechanical and
Aerospace Engineering at University of Florida (UF) in the spring of 2018. His work
focused on innovative data analytics to characterize mechanical systems especially on
predictive modeling, uncertainty quantification and optimization. He worked under Dr.
Raphael T. Haftka and Dr. Nam-Ho Kim as a part of the Structural & Multidisciplinary
Optimization Group during the graduate study. Before the Ph.D. study, he received
Master of Science from UF with a focus on topology optimization and Bachelor of
Science from Shanghai Jiao Tong University with a focus on error compensation for
CNC machining tool. He has been a main participant for several multi-disciplinary
projects across university, industry and national labs. He has worked as research
assistant on data analytics for the design and validation of various mechanical systems
including composite laminate, computational fluid dynamics for turbulence flow and
Exascale computer architecture. He has been serving as reviewers for Structural and
Multidisciplinary Optimization Journal.