Applications of Bayesian sensitivity Applications of Bayesian sensitivity and uncertainty analysis to the and uncertainty analysis to the statistical analysis of computer statistical analysis of computer simulators for carbon dynamicssimulators for carbon dynamics

Marc KennedyMarc Kennedy

Clive Anderson, Stefano Conti, Tony O’HaganClive Anderson, Stefano Conti, Tony O’Hagan

Probability & Statistics, University of SheffieldProbability & Statistics, University of Sheffield

OutlineOutline

Uncertainties in computer simulatorsUncertainties in computer simulators Bayesian inference about simulator outputsBayesian inference about simulator outputs

– Creating an Creating an emulatoremulator for the simulator for the simulator– Deriving uncertainty and sensitivity measuresDeriving uncertainty and sensitivity measures

Example applicationExample application Some recent extensionsSome recent extensions

Uncertainties in computer Uncertainties in computer simulatorssimulators

Consider a complex deterministic code with Consider a complex deterministic code with a vector of inputs and single outputa vector of inputs and single output

Use of the code is subject to:Use of the code is subject to:– Input uncertaintyInput uncertainty– Code uncertaintyCode uncertainty

)(xfy

Input uncertaintyInput uncertainty

The inputs to the simulator are unknown for a The inputs to the simulator are unknown for a given real world scenariogiven real world scenario

Therefore the true value of the output is uncertainTherefore the true value of the output is uncertain A Monte Carlo approach is often used to take this A Monte Carlo approach is often used to take this

uncertainty into accountuncertainty into account– Sample from the probability distribution of XSample from the probability distribution of X

– Run the simulator for each point in the sample to give a Run the simulator for each point in the sample to give a sample from the distribution of Ysample from the distribution of Y

– Very inefficient…not practical for complex codesVery inefficient…not practical for complex codes

Code uncertaintyCode uncertainty

The code output at a given input point is The code output at a given input point is unknown until we run it at that pointunknown until we run it at that point– In practice codes can take hours or days to run, so In practice codes can take hours or days to run, so

we have a limited number of runswe have a limited number of runs

We have some prior beliefs about the outputWe have some prior beliefs about the output– Smooth function of the inputsSmooth function of the inputs

Bayesian inference about Bayesian inference about simulator outputssimulator outputs

Bayesian solution involves building an Bayesian solution involves building an emulatoremulator Highly efficientHighly efficient

– Makes maximum use of all available informationMakes maximum use of all available information– A single set of simulator runs is required to train the A single set of simulator runs is required to train the

emulator. All sensitivity and uncertainty information is emulator. All sensitivity and uncertainty information is derived directly from thisderived directly from this

– The inputs for these runs can be chosen to give good The inputs for these runs can be chosen to give good information about the simulator outputinformation about the simulator output

A natural way to treat the different uncertainties A natural way to treat the different uncertainties within a coherent frameworkwithin a coherent framework

Inference about functions using Inference about functions using Gaussian processesGaussian processes

We model as an unknown function We model as an unknown function having a Gaussian process prior distributionhaving a Gaussian process prior distribution

hh(.) is a vector of regression functions and (.) is a vector of regression functions and are unknown coefficientsare unknown coefficients

)),(,)((~],)([ 22 cNf T βhβ

Prior expectation of the model output as a function of the inputs

)(f

β

Inference about functions using Inference about functions using Gaussian processesGaussian processes

We model as an unknown function We model as an unknown function having a Gaussian process prior distributionhaving a Gaussian process prior distribution

c(.,.) is a correlation function, which defines c(.,.) is a correlation function, which defines our beliefs about smoothness of the output our beliefs about smoothness of the output and is the GP varianceand is the GP variance

)),(,)((~],)([ 22 cNf T βhβ

Prior beliefs about covariance between model outputs

)(f

2

Choice of correlation functionChoice of correlation function

We use the product of univariate Gaussian We use the product of univariate Gaussian functions:functions:

Where is a measure of the roughness of Where is a measure of the roughness of the function in the the function in the kkth inputth input

p

kkkk xxbc

1

2})'(exp{)'( xx,

kb

roughness = 0.5roughness = 0.5

roughness = 0.2roughness = 0.2

roughness = 0.1roughness = 0.1

roughness = 0.01roughness = 0.01

Conditioning on code runsConditioning on code runs

Conditional on the observed set of training Conditional on the observed set of training runs,runs,

is still a Gaussian process, with simple is still a Gaussian process, with simple analytical forms for the posterior mean and analytical forms for the posterior mean and covariance functionscovariance functions

),( ii fy x ni ,,2,1

)(f

2 code runs2 code runs

2 code runs2 code runs

2 code runs2 code runs

Large b

Small b

3 code runs3 code runs

5 code runs5 code runs

More about the emulatorMore about the emulator

The emulator The emulator meanmean is an estimate of the is an estimate of the model output and can be used as a surrogatemodel output and can be used as a surrogate

The emulator is much more…The emulator is much more…– It is a It is a probability distributionprobability distribution for the whole for the whole

functionfunction– This allows us to derive inferences for many This allows us to derive inferences for many

output related quantities, particularly integralsoutput related quantities, particularly integrals

Inference for integralsInference for integrals

For particular forms of input distribution For particular forms of input distribution (Gaussian or uniform), analytical forms (Gaussian or uniform), analytical forms have been derived for integration-based have been derived for integration-based sensitivity measuressensitivity measures

– Main effects of individual inputsMain effects of individual inputs

– Joint effects of pairs of inputsJoint effects of pairs of inputs

– Sensitivity indicesSensitivity indices

Example ApplicationExample Application

Sheffield Dynamic Global Sheffield Dynamic Global Vegetation ModelVegetation Model (SDGVM) (SDGVM)

Developed within the Centre for Terrestrial Developed within the Centre for Terrestrial Carbon DynamicsCarbon Dynamics

Our job with SDGVM is to: Our job with SDGVM is to: – Apply Apply sensitivity analysis sensitivity analysis for model testingfor model testing

– Identify the greatest sources of uncertaintyIdentify the greatest sources of uncertainty

– Correctly reflect the uncertainty in predictionsCorrectly reflect the uncertainty in predictions

Plant respiration

Photosynthesis

Loss

Soil respiration

Loss

– Terrestrial carbon source if NEP is negative

– Terrestrial carbon sink if NEP is positive

Net Ecosystem Production

(CARBON FLUX)

Some Inputs ParametersSome Inputs Parameters

Leaf life spanLeaf life span Leaf areaLeaf area Budburst temperature Budburst temperature Senescence temperatureSenescence temperature Wood densityWood density Maximum carbon storageMaximum carbon storage Xylem conductivityXylem conductivity

Soil clay %Soil clay % Soil sand %Soil sand % Soil depthSoil depth Soil bulk densitySoil bulk density

Main Effect: Leaf life spanMain Effect: Leaf life span

100 150 200 250 300 350

leaf life-span

01

02

03

0

me

an

NE

P

If leaves die young, NEP is predicted to be higher, on average. Why?

Main Effect: Leaf life span (updated)Main Effect: Leaf life span (updated)

100 150 200 250 300 350

leaf life-span

05

10

15

20

25

30

Me

an

NE

P

If leaves die young, SDGVM allowed a second growing season, resulting in increased carbon uptake. This problem was fixed by the modellers

Main Effect: Senescence TemperatureMain Effect: Senescence Temperature

4 5 6 7 8 9 10

senescence

01

02

03

0

me

an

NE

P

Small values mean the leaves stay until the temperature is very low

Large values mean the leaves drop earlier, so reduce the growing season

When soil bulk density was added to the active parameter set, the Gaussian Process model did not fit the training data properly

Error discovered in the soil moduleError discovered in the soil module

NEP

-20

0

20

40

60

80

0 500000 1000000 1500000

NEP

-20

0

20

40

60

80

0 500000 1000000 1500000

Before… After…

Bulk density Bulk density

Our GP model depends on the output being a smooth function of the inputs. The problem was again fixed by the modellers

SDGVM: new sensitivity SDGVM: new sensitivity analysisanalysis

Extended sensitivity analysis to 14 input Extended sensitivity analysis to 14 input parameters (using a more stable version)parameters (using a more stable version)

Assumed uniform probability distributions Assumed uniform probability distributions for each of the parametersfor each of the parameters

The aim here is to identify the greatest The aim here is to identify the greatest potential sources of uncertaintypotential sources of uncertainty

160 170 180 190 200

max. age (years)

150

160

170

180

190

1.8 2.0 2.2 2.4 2.6

water potential (M Pa)

150

160

170

180

190

160 180 200

leaf life span (days)

150

160

170

180

190

0.0035 0.0040 0.0045

minimum growth rate (m)

150

160

170

180

190

NE

P (

g/m

2 /y)

NE

P (

g/m

2 /y)

Leaf life span 69.1% by investing effort to learn by investing effort to learn more about this parameter, more about this parameter, output uncertainty could be output uncertainty could be

significantly reducedsignificantly reduced

Minimum growth rate 14.2%

Water potential 3.4%

Maximum age 1.0%

Percentage of total output variance

Extensions to the theoryExtensions to the theory

Multiple outputsMultiple outputs

So far we have created independent So far we have created independent emulators for each outputemulators for each output– Ignores information about the correlation Ignores information about the correlation

between outputsbetween outputs We are experimenting with simple models We are experimenting with simple models

linking the outputs togetherlinking the outputs together This is an important first step in treating This is an important first step in treating

dynamic emulatorsdynamic emulators and in and in aggregating code aggregating code outputsoutputs

Dynamic emulatorsDynamic emulators

Physical systems typically evolve over timePhysical systems typically evolve over time Their behaviour is modelled via dynamic Their behaviour is modelled via dynamic

codescodes

– wherewhere xx are tuning constants andare tuning constants and zztt are context-are context-specific driversspecific drivers

– Recursive emulation ofRecursive emulation of yytt over the appropriate over the appropriate time span shows promising resultstime span shows promising results

),,( 1 ttt zxyfy

CENTURY output ( ) and dynamic emulator ( )

Aggregating outputsAggregating outputs

Motivated by the UK carbon budget problemMotivated by the UK carbon budget problem– The total UK carbon absorbed by vegetation is a sum of The total UK carbon absorbed by vegetation is a sum of

individual pixels/sitesindividual pixels/sites

– Each site has a different set of input parameters (e.g. Each site has a different set of input parameters (e.g. vegetation/soil properties), but some of these are vegetation/soil properties), but some of these are correlatedcorrelated

This is a multiple output codeThis is a multiple output code– Each site represents a different outputEach site represents a different output

Bayesian uncertainty analysis is being extended, Bayesian uncertainty analysis is being extended, to make inference about the sumto make inference about the sum

ReferencesReferences

For Bayesian analysis of computer models:For Bayesian analysis of computer models:– Kennedy, M. C. and O’Hagan, A. (2001). Kennedy, M. C. and O’Hagan, A. (2001).

Bayesian calibration of computer models (with Bayesian calibration of computer models (with discussion) J. Roy. Statist. Soc. B, 63: 425-464discussion) J. Roy. Statist. Soc. B, 63: 425-464

For Bayesian Sensitivity analysis:For Bayesian Sensitivity analysis:– Oakley, J. E. and O’Hagan, A. (2004). Oakley, J. E. and O’Hagan, A. (2004).

Probabilistic sensitivity analysis of complex Probabilistic sensitivity analysis of complex models: A Bayesian approach. J. Roy. Statist. models: A Bayesian approach. J. Roy. Statist. Soc. B, 66: 751-769Soc. B, 66: 751-769