Sensitivity analysis

Marko TainioDecision analysis and Risk Management

course in Kuopio21.3.2011

Sensitivity analysis studies on what happens inside the Black Box

Black Box(the model)

DataResults

Outline of lecture

• What is sensitivity analysis?– Why to use sensitivity analysis– What options there are?

• Example of two sensitivity analysis methods– Nominal Range Sensitivity– Rank-order Correlation

• Other sensitivity analysis methods

What is sensitivity analysis

Definition of sensitivity analysis

• http://en.wikipedia.org/wiki/Sensitivity_analysis

• Sensitivity analysis (SA) is the study of how the variation (uncertainty) in the output of a mathematical model can be apportioned, qualitatively or quantitatively, to different sources of variation in the input of the model.

• Put another way, it is a technique for systematically changing parameters in a model to determine the effects of such changes.

Main idea of SA

• In sensitivity analysis you change the input parameters to see how the model results response to these changes

• Thus, sensitivity analysis resembles laboratory research where you control input and measure the outcome– Same statistical methods are applied in laboratory

studies and in sensitivity studies (correlation, regression analysis, ANOVA)!

Different sensitivity analysis methods

Frey and Patil, 2002 divides sensitivity analyses to three broad categories:1. Mathematical• Suitable for deterministic models

2. Statistical (or probabilistic)• Usually based on simulation and statistical

parameters.

3. Graphical• Presenting of sensitivity with graphs, charts etc.

When to use sensitivity analysis

• In simple, always when making risk or decision models!

• Two main advantages:– You can guide your own modeling work by testing

the sensitivity of the model while doing the assessment;

– You can also communicate to possible users the main uncertainties related to assessment

Calculation of sensitivity analysis:- Nominal Range Sensitivity

- Rank-order Correlation

Setting

• How many square meters of tables we have in this building?

• The model is simple:– (Number of tables) x (average width) x (average

height) = n x w x h

• Since we don’t know any of these parameters, we assume some distributions for themParameter Best guess Min MaxNumber of tables (#) 150 50 300Height (m) 1 0,5 1,2Width (m) 1,5 1 2

Nominal Range Sensitivity Analysis Method

• NRSA is used to evaluate the effect on model outputs of varying only one of the model inputs across its entire range of plausible values, while holding all other inputs at their nominal or base-case values

• Equation:

Page 14, Frey 2

NRSA sensitivity analysis

Parameter Nominal input Min input Max inputNumber of tables (#) 150 50 300Height (m) 1 0,5 1,2Width (m) 1,5 1 2

Nominal output Min output Max output NRSAResult (all) 225 25 720 -Results (number of tables) 225 75 450 1,7Results (height) 225 113 270 0,7Results (width) 225 150 300 0,7

Model is most sensitive to Number of tables parameter.

Qualities of NRSA analysis

• Advantages:– Works with deterministic models (no need for Monte

Carlo)– Easy to use and apply in number of models

• Disadvantages:– Works only with linear models– Doesn’t take into account interactions/correlations

between input parameters• NRSA is a good screening level sensitivity analysis

tool

Sample and Rank Correlation Coefficients

Model (aka. Black Box)

Number of tables

Height

Width

50 150 300

0.5 1.0 1.2

1 1.5 2.0

Result

104 214 384

Sample and Rank Correlation Coefficients

Two options for correlation analysis:1. Parametric or Pearson– For linear models

2. Non-parametric or Spearman or rank– Also for non-linear models– Importance analysis

• Correlation varies between -1 and 1– The value of -1 represents a perfect negative

correlation while a value of +1 represents a perfect positive correlation

In Monte Carlo, correlation is calculated between samples

Sample Number of tables Height Width1 101,732485 0,852764085 1,6274748332 232,0156268 0,78158924 1,6117668233 131,4324874 0,844481494 1,5267400294 209,13953 0,855728127 1,4942924245 275,3905506 0,994355641 1,509057036 209,9895839 1,052050684 1,3373796087 229,2318928 0,569119462 1,1548386268 229,3910416 1,112370096 1,3631459769 281,1751494 0,634582688 1,26433879810 240,1094749 0,814638682 1,68579465311 139,4776508 0,748580973 1,14413535312 165,4660823 0,85503873 1,57991072413 143,5213879 0,827288099 1,80334039614 115,9071316 1,017827005 1,74052938515 249,5953871 1,048682453 1,9042137816 155,7281212 0,907887852 1,5123782217 198,3111363 0,722974214 1,06103277818 74,6221445 0,996051913 1,62260763119 184,3685813 1,000092521 1,6807430520 98,38646505 0,712855585 1,459809743

Sample Results1 141,18962 292,27933 169,45644 267,42845 413,23446 295,45357 150,66068 347,83099 225,594610 329,745611 119,459512 223,525613 214,11714 205,336215 498,420916 213,825617 152,124318 120,604419 309,904920 102,3843

Correlation

Result of the rank-order correlation sensitivity analysis

The uncertainty in the results correlates 80% with the uncertainty of „Number of tables” parameter.

Qualities of correlation sensitivity analysis

• Advantage:– Easy to compute– Correlation available in most of the computer

modeling tools (including Excel)

• Disadvantage:– Correlation is not causation– Non-linear and non-monotonic models are

problematic

Other sensitivity analysis methods

Research on sensitivity analysis

• Frey et al. 2003 (Evaluation of Selected Sensitivity Analysis Methods Based Upon Applications to Two Food Safety Process Risk Models) lists 11 different sensitivity analysis methods

• They also made recommendations on which sensitivity analysis to use in which situation

• Report available: http://www.ce.ncsu.edu/risk/Phase2Final.pdf

Different sensitivity analysis methods

• Mathematical Methods for Sensitivity Analysis– Nominal Range Sensitivity Analysis Method– Differential Sensitivity Analysis (DSA)

• Statistical Methods for Sensitivity Analysis– Sample and Rank Correlation Coefficients– Regression Analysis– Rank Regression – Analysis of Variance– Classification and Regression Tree– Sobol’s Indices– Fourier Amplitude Sensitivity Test (FAST)

• Graphical Methods for Sensitivity Analysis– Scatter Plots – Conditional Sensitivity Analysis

Selection of the sensitivity analysis (Frey et al. 2004)

• Some selection criteria's:– What are the objectives of sensitivity analysis?– Based upon the objectives, what information is needed from

sensitivity analysis?– What are the characteristics of the model that constrain or

indicate preference regarding method selection?– How detailed is the analysis?– What are the characteristics of the software that may constrain

selection of methods?– What are the specifications of the computing resources?– Can “push-button” methods adequately address characteristics

of interest in the analysis?– Is the implementation of the selected sensitivity analysis

method post-hoc?

Frey et al. pages 46-47, http://www.ce.ncsu.edu/risk/Phase3Final.pdf

Some objectives of sensitivity analysis

• Rank ordering the importance of model inputs (e.g., critical control points);

• Identifying combination of input values that contribute to high exposure and/or risk scenarios;

• Identifying and prioritizing key sources of variability and uncertainty;

• Identifying critical limits;• Evaluating the validity of the model.

What Information is Needed from Sensitivity Analysis?

• Qualitative or quantitative ranking of inputs• Discrimination of the importance among

different inputs• Grouping of inputs that are of comparable

importance• Identification of inputs that are not important• Identification of critical limits• Identification of inputs and ranges that produce

high exposure or risk• Identification of trends in the model response

Frey et al. Pages 58, http://www.ce.ncsu.edu/risk/Phase3Final.pdf

Further reading

• Frey et al. 2004. Recommended Practice Regarding Selection, Application, and Interpretation of Sensitivity Analysis Methods Applied to Food Safety Process Risk Models: http://www.ce.ncsu.edu/risk/Phase3Final.pdf

• Frey et al. 2003. Evaluation of Selected Sensitivity Analysis Methods Based Upon Applications to Two Food Safety Process Risk Models: http://www.ce.ncsu.edu/risk/Phase2Final.pdf

• Patil and Frey 2004. Comparison of sensitivity analysis methods based on applications to a food safety risk assessment model. Risk Analysis 24 (3): 573-585