Exploitation of OA techniques to support IA & decision making
description
Transcript of Exploitation of OA techniques to support IA & decision making
Exploitation of OA techniques to support IA & decision making
9th Feb 2010Colin Drysdale
E-mail : [email protected]
2
Overview
• What is Operational Analysis (OA)?
• Difficulties facing a decision maker – how can the analytical community help?
• Ways to inform a decision maker
• Ways to ensure the decision maker understands the information
• An example of a technique that sets out to both inform & make the information understandable
3
What is Operational Analysis (OA)?
• OA is the name given by the military to Operational Research (OR)
• OR, also known as Operations Research or Management Science (OR/MS) is the discipline of applying advanced analytical methods to help make better decisions– Source: The OR society (http://www.orsoc.org.uk)
4
Typical outputs of OA
• The articulation of analytical problems to be solved
• Identification of the solution space
• Mathematical theories
• Suggested policies that could optimise systems or satisfy goals
• Logical structuring of issues– Leading to insights
• Measures of Effectiveness– Rather than Measures of Performance
5
Difficulties facing decision makers
6
Difficulties facing decision makers
• The influence of “providers” over “deciders”
• The maturity & availability of solutions
• Irreconcilable starting positions
• Multiple goals & constraints
• Uncertainty & risks
• Trade-offs
• Funds
(Not an exhaustive list)
7
What makes decisions succeed or fail?Informed
Investment Decisionmade
Decision makerinformed
Decision makerheads info'
Rigorous,uncertainty bounded
"true" cost info'for all options
Rigorous,uncertainty bounded
operationaleffectiveness info'
for all options
Assumptions arevalid (fit for
purpose)Method forcombining
performance data inscenarios is fit for
purpose
Scenarios are fitfor purpose Performance data
are fit for purpose
Cost estimatesare fit for purpose
Method forcombining cost
estimates inscenarios is fit for
purpose
&
&
& &
&&&
&&
&&
&
Decision makerunderstands info'
&
8
How can the analytical community help decision makers?
• We can calculate and collate information to inform decision makers
– Inform them of the cost implications of investment options
– Inform them of the operational effectiveness impact of the options
9
How can we help decision makers understand the gathered information?
• One way to inform decision makers that, additionally, seeks to help decision makers understand the gathered information is called Multi Criteria Decision Analysis– An example is coming up shortly
• Another way is to present information in more than one format – E.g. both tables & graphs
• Yet another way is to try to turn the implications of numbers back into words for the decision maker
10
Introducing MCDA as one way to inform decision makers
11
Background to MCDA
What MDCA is:• MCDA is the name given to a group of techniques by the OR Society
• A technique for supporting decision makers who are trying to simultaneously satisfy multiple, potentially conflicting goals
• MCDA techniques pop-up in all sorts of domains with different names & conducted in different ways to different standards of analytical rigour
• MCDA is a technique for helping decision makers make choices– Arguably it does not:
• do predictive modelling• produce Measures of Effectiveness
12
One way to inform decision makers
• It is difficult to view MCDA as an “advanced” analytical technique– It is classed as a form of Soft OA
• MCDA requires little or no mathematical knowledge to use– But beware of this
• Like any technique it has pit falls, these can be avoided by good practitioners, who understand the features of the decision to be supported
• Decision making Culture• Procedures• Standards of analytical rigour
13
What does MCDA do?
• Produces a Figure of Merit (FOM)
• Combines disparate criteria or factors
• Provides the relative ranking of the options
W2
W1
W3
Sa
FOMSb
Sc
FOM = Σ Si . Wi
14
MCDA sequence of events
1. Network of criteria / requirements
2. Criteria that can be measured, predicted, estimated, judged, or populated from available data
3. Define the relationship between performance and worth
4. Weight the links within the network
5. Evaluated criteria at nodes on the network (& then calculate the scores)
6. Conduct sensitivity analysis
7. Review findings with decision maker
15
MCDA Example
16
How MCDA works – simple example
• Equipment investment decision
• Equipment - Bicycle front light
• Scenario 1
– 10 mile daily commute after dark on a mixture of lit town roads and un-lit dual carriage way, single carriageway, & single track roads with passing places
• Requirements:
• Vision - to enable no reduction in speed from day light conditions
• Presence - no reduction in recognition to other road users compared to day light conditions
• All weather capable
• Light weight
• Reliable - very low probability of failure in use
• Reliable - graceful degradation or failure warning
• Available – for expected maximum mission duration
• User able to choose where to fix the light
Warning! – this simplified example contains some elements notappropriate to MOD Business Case support
17
14 Vision
15 Presence
17 Fixing options
18 Availability
21 FOM Scenario 1
22 Purchase Cost
23 Running Costs
36 weight
37 All Weather
1. MCDA network of criteria
18
14 Vision
15 Presence
17 Fixing options
18 Availability
21 FOM Scenario 1
22 Purchase Cost
23 Running Costs
24 Oversize
26 Helmet
27 Head
28 Luminous flux
31 Failure warning
33 Mission lengthsupported
35 Flashing mode
36 weight
37 All Weather
1. MCDA network of criteria
WARNING: This is not necessarily howyou should combine cost information
for Government Investment decisions consult the Treasury “Green Book”.
19
2. Criteria that can be “measured”
Lm
(£)
(£ per annum)
Y/N
Y/N
Y/N
Hr
Y/N
g
Y/N
14 Vision
15 Presence
17 Fixing options
18 Availability
21 FOM Scenario 1
22 Purchase Cost
23 Running Costs
24 Oversize
26 Helmet
27 Head
28 Luminous flux
31 Failure warning
33 Mission lengthsupported
35 Flashing mode
36 weight
37 All Weather
No reliable information
Measured Quantity(measurand i/p)
Yes/No Question(Boolean i/p)
Subjective SMEjudgement
20
3. Performance & worth relationships
Lumens
Burn Time
Weight
500 (g)
(hr)10 2 3
1,000100
Value
Value
Value
Measured Quantity(measurand i/p)
Score
(0≤ Score ≤1)
Score Performance
1
0
1
0
1
21
3. Performance & worth relationships
PurchaseCost
(£)
RunningCost
(£)
100 200 30075
Value
Value
10 15 30
Measured Quantity(measurand i/p)
Score
(0≤ Score ≤1)
Score Performance
0
1
0
1
22
4. Weight contributions to outcomes - weight network links
14 Vision
15 Presence
17 Fixing options
18 Availability
21 FOM Scenario 1
22 Purchase Cost
23 Running Costs
24 Oversize
26 Helmet
27 Head
28 Luminous flux
31 Failure warning
33 Mission lengthsupported
35 Flashing mode
36 weight
37 All Weather
0.20.10.2
0.2
0.1
0.10.1
1
0.5
0.50.7
0.30.33
0.33
0.33
Contribution Weight
(ΣWi = 1)
Weight Contributions
23
Aspects of good practice
24
Sensitivity Analysis
• Having got initial results out of the models may have felt like arriving at the destination …
• This is NOT correct
– the Analysts job is not done
– there is a need to conduct Sensitivity Analysis
– to understand how uncertainty affects the ranking of the options
– indeed once uncertainty is considered it may not be possible to
discriminate between the options
25
5. Evaluate criteria – for each Option - Top FOM results
Option N Make N ModelPurchace (£)
Running (£ pa.)
OP (Lumens)
Weight (g) FOM
1 Dog ear Hedo 31.49 3.78 80 118 0.6413 Faith Four 1 254.49 0.51 960 420 0.5914 Faith Four 1+1 309.49 0.51 960 420 0.59
8 Chilly Minn Mum 291.49 0.51 960 320 0.5616 Heavy emotion Susan 200 79.99 0.51 200 240 0.54
26
Flaws with these answers
• Option 1, The Dog ear, “Hedo” should not come out as the best option as it is not really bright enough to be fit for purpose in this Scenario– It is successful because of its light weight and low purchase
price
Lumens
1,000100
Value
0
1
Lumens
1,000100
Value
0
1
27
Flaws with these answers
• Options 13 & 14, The Faith, “Four 1” and The “Four 1+1” should not be equal in merit
100 200 30075
PurchaseCost
(£)
Value
0
1
100 200 30075
PurchaseCost
(£)
Value
0
1
PurchaseCost
(£)
Value
0
1
28
7.Review findings with decision maker
• Should not use this technique in the absence of decision makers from relevant parts of the process
• Review findings with decision maker
• Sometimes the analysis (or thought process) does not get as far as MCDA results– We gain some insight that reframes or satisfactorily solves the
exam question before the analysis is concluded
29
What MCDA can include
• Scenarios
• Linear or non-linear conversion of performance against each criterion into some from of worth to the decision maker (e.g. military worth)
• Sensitivity analysis (e.g. to address uncertainty & risk)
30
How I actually made the decision
31
Faith, Four 1
Chilly, Devil
Faith, Two 1
Dog ear, Hedo
How I actually made the decision
PurchaseCost (£)
Lumens
Faith, Sight
0
100
200
300
400
500
600
700
800
900
1000
0 50 100 150 200 250 300
32
Conclusion
Previously: – Discussed the role of causal mapping to make the development of
MCDA networks adequately rigorous to support Business Case decisions
Today:– Stated IA and OA studies can inform decision makers– Considered failure modes for decisions & how the analytical
community can help decision makers succeed– Explained what MCDA is and what it does– Identified that there are good and bad practices– Suggested some issues for discussion covering which practices
are acceptable for any given purpose
36
Options and data
37
Evaluated criteria – evaluate network “leaf” nodes
Option 1 2 3 4 5 6 7 8 9 10N Make Dog ear Dog ear Chilly Chilly Chilly Chilly Chilly Chilly Faith FaithN Model Hedo S4 Control Control Minn2Devil Mada Bull Minn Mum Sight Two 1Purchace (£) 31.49 269.99 148.49 162 201.49 219.49 247.49 291.49 72 164.49Running (£ pa.) 3.78 0.51 0.51 0.51 0.51 0.51 0.51 0.51 68.36 0.51Oversize 1 0 1 1 1Helment 1 1 1 1Head 1OP (Lumens) 80 700 240 240 700 480 700 960 240 480Fail warn 1 1 0 1 0 1 1 1Burn time (hr) 45 3 3 3 1 3 3 3 2.75 2Flash 1 0 1 1 1 1 1 1Weight (g) 118 497 98 98 102 228 276 320 220 275
Option 11 12 13 14 15 16 17 18 19 20N Make Faith Faith Faith Faith Heavy emotion Heavy emotion Hoff-mister Hoff-mister Hoff-mister Hoff-misterN Model Two 2 Two 1+1 Four 1 Four 1+1 Susan 120 Susan 200 Small Small 110 Small Small 110 +Small Small 100X2 Small Samll 400Purchace (£) 204.49 204.49 254.49 309.49 62.5 79.99 79.99 109.49 174.99 224.99Running (£ pa.) 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51Oversize 1 1 1 1 1 1 1 1 1 1Helment 1 1 1 1 1Head 1 1 1OP (Lumens) 480 480 960 960 120 200 110 110 200 400Fail warn 1 1 1Burn time (hr) 4 2 2 2 2 2 3 3 3 2Flash 1 1 1 1 1Weight (g) 405 275 420 420 280 240 175 175 230 317
38
And the answer was ….
Option 1 2 3 4 5 6 7 8 9 10N Make Dog ear Dog ear Chilly Chilly Chilly Chilly Chilly Chilly Faith FaithN Model Hedo S4 Control Control Minn2Devil Mada Bull Minn Mum Sight Two 1Purchace (£) 31.49 269.99 148.49 162 201.49 219.49 247.49 291.49 72 164.49Running (£ pa.) 3.78 0.51 0.51 0.51 0.51 0.51 0.51 0.51 68.36 0.51Oversize 1 0 1 1 1Helment 1 1 1 1Head 1OP (Lumens) 80 700 240 240 700 480 700 960 240 480Fail warn 1 1 0 1 0 1 1 1Burn time (hr) 45 3 3 3 1 3 3 3 2.75 2Flash 1 0 1 1 1 1 1 1Weight (g) 118 497 98 98 102 228 276 320 220 275
FOM 0.64 0.40 0.49 0.53 0.51 0.46 0.51 0.56 0.50 0.50
Option 11 12 13 14 15 16 17 18 19 20N Make Faith Faith Faith Faith Heavy emotion Heavy emotion Hoff-mister Hoff-mister Hoff-mister Hoff-misterN Model Two 2 Two 1+1 Four 1 Four 1+1 Susan 120 Susan 200 Small Small 110 Small Small 110 +Small Small 100X2 Small Samll 400Purchace (£) 204.49 204.49 254.49 309.49 62.5 79.99 79.99 109.49 174.99 224.99Running (£ pa.) 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51Oversize 1 1 1 1 1 1 1 1 1 1Helment 1 1 1 1 1Head 1 1 1OP (Lumens) 480 480 960 960 120 200 110 110 200 400Fail warn 1 1 1Burn time (hr) 4 2 2 2 2 2 3 3 3 2Flash 1 1 1 1 1Weight (g) 405 275 420 420 280 240 175 175 230 317
FOM 0.38 0.48 0.59 0.59 0.46 0.54 0.53 0.51 0.38 0.40
Because I was just generating an example I did not conduct any sensitivity analysis
39
4. Objective & Threshold definition
1.0
0
Value
Top Speed(miles per hour)
Top Speed
100 150 200
1.0
0
Value
Top Speed(miles per hour)
Top Speed
100 150 200
or “effectiveness envelope”