Generation of Pareto Optimal Ensembles of Calibrated

Parameter Sets for Climate Models

Keith Dalbey, Ph.D.Sandia National Labs, Dept 1441, Optimization and Uncertainty Quantification

Michael Levy, Ph.D.Sandia National Labs, Dept 1442, Numerical Analysis and Applications

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear

Security Administration under Contract DE-AC04-94AL85000.

December 12-17, 2010

Outline• Motivation• Approach: Pareto Ensemble• What Does “Pareto Optimal” Mean?• Finding a “Pareto Optimal” Ensemble• Results of Tuning Climate Model• Summary & Future Work

ReferencesJackson et al, “Error reduction and convergence in climate prediction,”

Journal of Climate, 2008.

Eddy & Lewis, “Effective generation of pareto sets using genetic programming,” Proc. of ASME Design Engineering Technical Conference, 2001.

Dalbey & Karystinos, “Fast generation of space-filling latin hypercube sample designs,” Proc. of 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, 2010.

Motivation Calibrating (tuning) climate models • choosing values of model parameters to predict well

Is difficult because• They have many inputs and outputs• Diverse parameters sets can match observations similarly well• Errors can compensate: “2 wrongs can make a right” under

historical conditions• Climate change (new conditions) might expose a previously

hidden mis-calibration, so…

History matching is necessary but not sufficient for good predictions.

• The future is uncertain, but we can quantify the uncertainty (estimate statistics) for possible future climates.

Approach: Pareto EnsembleHow can we make good statistical predictions?

Use a diverse ensemble of “good” parameter sets to determine the range/spread of possible future climates

QUESTION: What’s the definition of a “good” parameter set? There are multiple outputs and what’s good for one output can be bad for another.

(AN) ANSWER: It’s Pareto optimal. A point (parameter set) is Pareto optimal if there is no other point that is as good or better than it in ALL outputs.

What does the “Pareto” mean?It’s just the name of the person who discovered it…Vilfredo Federico Damaso Pareto was an Italian engineer, sociologist, economist, and philosopher.

What Does “Pareto Optimal” Mean?

2D Pareto front schematics

What Does “Pareto Optimal” Mean?•Usually, the current approx. of the true Pareto front.

•The Pareto front defines the “zero sum game” of all optimal compromises you could make.

•Unlike a weighted combination of objective functions, it lets you choose a specific compromise/ combination AFTER the optimization is complete.

• It does NOT say which compromise/combination is best, just what all the “optimal” choices are.

• It says “Don’t choose anything Pareto non-optimal because there’s something better in all criteria.”

Finding a “Pareto Optimal” Ensemble•Used the Multi Objective Genetic Algorithm (MOGA) in DAKOTA’s (Design Analysis Kit for Optimization and Terascale

Applications) JEGA (John Eddy’s Genetic Algorithm) sub-package

•GA’s typically need 1000’s of simulations, I could only afford 1000…

•Used test problem (find surface of radius=1 6D hyper-sphere in input space, 10 outputs) to tune MOGA settings and initial population (space-filling, specifically Binning

Optimal, Symmetric Latin Hypercube Sampling, or BOSLHS), for:• Large Pareto Ensemble

• Mean radius close to 1

• Uniform spread

• Small radius variance

Finding a “Pareto Optimal” Ensemble

Use DAKOTA’s MOGA on a test problem with 6 inputs and 10 outputs; true solution is a radius 1 hypersphere

Default Monte Carlo seed

PDF’s of the Pareto Ensemble’s

1. # of points

2. Point spread

3. Mean radius

4. Standard deviation of radius

1 2

3 4

Finding a “Pareto Optimal” Ensemble

Use DAKOTA’s Multi Objective Genetic Algorithm on a test problem with 6 inputs and 10 outputs

true solution is a radius 1 hypersphere

BOSLHS seedDefault Monte Carlo seed

Results of Tuning Climate Model

Summary & Future Work•Climate model parameters that match history well might not predict well (climate change might expose a previously hidden mis-calibration of parameters).

•Plan: Use a diverse ensemble of “good” (Pareto optimal) parameter sets to determine the range/spread of possible future climates.

•Used MOGA to find a (very large) Pareto optimal ensemble of calibrated parameter sets.

•Next steps: –down select Pareto optimal ensemble, and–simulate smaller ensemble out to 2100.

Some “Good” Parameter SetsInputs % change in output mismatch relative to CCSM4 defaultRHMINL RHMINH ALFA TAU [hrs]C0 [m -̂1] KE [(m^2s/kg) 0̂.5/s]TREFHT T U PS RELHUM LHFLX LWCF SWCF PRECT RADBAL

0.9348 0.7941 0.5527 4.426 3.445E-3 6.883E-6 10.5869 1.33 6.515 0.64 -9.13728 38.049 -33.32 -20.5 50.546 -45.9170.9348 0.8789 0.3379 3.020 4.711E-3 7.867E-6 4.73702 0.42 9.581 0.75 23.3127 31.136 -26.88 -31.94 49.582 -65.1220.9393 0.6727 0.2348 2.316 5.086E-3 4.148E-6 -0.309 3.88 11.42 1.44 -15.2271 6.6722 3.5036 14.204 13.99 -99.3760.9496 0.7055 0.2365 6.652 5.836E-3 7.211E-6 14.4249 3.39 8.135 0.99 -29.4557 45.088 -31.76 -29.09 49.752 -73.4570.9277 0.8039 0.3483 3.254 4.383E-3 3.383E-6 1.38947 0.47 1.366 1.22 -10.3006 30.064 -27.13 -16.6 46.626 -65.0480.9316 0.7992 0.0715 2.151 5.133E-3 8.195E-6 -4.1146 1.59 14.62 1.12 44.5008 -2.331 0.5983 24.616 4.9049 -99.9920.9348 0.6586 0.2090 2.316 5.273E-3 3.383E-6 -0.2539 3.38 7.114 0.62 -23.2472 5.3236 2.0664 20.214 11.527 -61.240.9348 0.8610 0.3379 3.020 4.711E-3 7.867E-6 1.34654 0.67 8.201 1.2 19.6143 24.863 -31.78 -26.22 40.619 -85.4330.9393 0.6982 0.2348 2.316 5.086E-3 4.148E-6 3.17856 3.13 15.23 1.61 -11.5716 5.8576 -1.338 6.613 14.649 -90.4690.9418 0.7371 0.2365 6.652 5.836E-3 7.211E-6 13.0888 2.74 8.034 0.39 -26.0691 51.814 -30.86 -25.73 49.191 -84.6720.9316 0.7795 0.0715 2.151 5.133E-3 8.195E-6 -5.7248 2.6 19.24 0.81 35.4784 -1.642 4.6604 31.289 1.5978 -78.5710.9324 0.7362 0.0973 0.910 4.570E-3 9.617E-6 -1.7177 2.77 27.21 0.37 27.8201 -15.75 30.85 45.384 -7.412 -99.8090.9324 0.8320 0.1171 1.848 4.570E-3 8.523E-6 -1.021 2.58 27.97 0.45 58.6333 0.156 -6.863 13.883 5.2161 -92.776

CCSM4 default 0.91 0.8 0.1 1 3.500E-3 1.000E-6 0 0 0 0 0 0 0 0 0 0

Range0.95 0.9 0.6 8 6.000E-3 1.000E-50.8 0.6 0.05 0.5 3.000E-3 3.000E-6