Expert-Opinion Elicitation
description
Transcript of Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
Expert-Opinion Elicitation
Robert C. PatevNorth Atlantic Division – Regional Technical Specialist
(978) 318-8394
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• Subjective Estimation• Elicitation Process
– Background– Expert-Opinion Elicitation (EOE) Process
• Probability– Axioms of Probability– Medians and Percentiles
• Training Example
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
Subjective Estimation
• Uses of one or more experts to estimate a probability (qualitative or quantitative) for use in engineering risk analysis– Good for first estimate of probabilities– Quick, cost effective and efficient method– Problems:
• Not a formal elicitation• Usually not well documented• Probabilities may not be repeatable or defendable• Probabilities may be highly subjective and biased• Probabilities have larger uncertainties compared to
structured elicitation values
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
Subjective Estimation
• How good are we at quantifying subjective estimates?
• Let us see…..
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
Subjective Estimation
• How good are we at quantifying subjective estimates?– Class Example:
• How may ships passed through the Panama Canal last year?
– Give best estimate
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• Background– Process developed by RAND Corporation in
late 1950’s - early 1960’s• Delphi Method• Scenario Analysis
– Effects of thermonuclear war– Civil Defense strategic planning
• Examine if U.S. population could survive a nuclear attack
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• Background– Definition– A formal (protocol), heuristic (through
discussion) process of obtaining information or answers to specific questions called issues
• e.g., failure rates or probabilities, and failure consequences
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• Background– EOE is used for preliminary risk evaluation
(screening) is not really intended to replace more complex reliability models
– EOE has been used by industry and government agencies to develop failure probabilities when there is a lack of failure information
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• Drawbacks– Subjective process
• Not consensus building• Inherently contains bias and dominance
– Difficult to process result to determine reliability or hazard rates
• Assumptions need to be made
• Current Usage in USACE– Supplement to other models
• Calculate reliability (not for critical components)• Event tree probabilities• Used in consultation with HQUSACE
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• EOE Process– Participants
• Experts• Observers• Listeners• Technical Integrator and Facilitator• Peer Reviewers
– ITR process and results
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• EOE Process– Identification and Selection of Experts
• Strong relevant expertise• Familiarity and knowledge with issues• Willingness to act as impartial evaluators• Willingness to participate, prepare, and
provide needed input• Strong communication skills, interpersonal
skills, and ability to generalize
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• EOE Process– Inform experts of issues
• “Read ahead” materials• Site visits
– Train experts– Elicitation
• First opinion• Discussion among experts• Second opinion
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• Probability– General expressions
• Percent (1% probability of failure)• Fraction (1/100)• Relative frequency (1 out of 1000)
– Axioms of Probability
• 0 < Pf < 1
• Sum of probabilities over all possible outcomes must equal 1.
– This assume events are independent.
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• Statistics– Median
• e.g., Median income, median age • Rank value• For odd n, value with rank of (n+1)/2• For even n, average of value with rank n/2 or
(n/2) + 1• Used to limit extreme values
– Average• Sum of Xi divided by sample size
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
Median Vs. Average
• Sample 1– 100– 100– 200– 300– 400
• Median = 200• Average = 220
• Sample 2– 100– 100– 200– 300– 2000
• Median = 200• Average = 540
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• Percentiles– A p-percentile value (Xp) based on a sample is
the value of the parameter such that p% of the data is less than or equal to Xp
• e.g., The median is the 50th percentile
Expert-Opinion Elicitation
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
Chickamauga Lock and DamChickamauga Lock and Dam Expert ElicitationIssue #4 - River Wall Blocks
Event Full Description Expert-opinion elicitation SummaryName of Issue Table
First SecondQuestion #1a: Response Response
Unconstrained flow of water through the river wall adversely effecting the ability to dewater the lock and/or filling/emptying operation
Given the failure mode identified for the river wall without any advanced maintenance (fix-as-fails scenario), what is the probability of that failure by the year 2005?
Median = 0.10% Median = 0.1%
Minimum = 0.01%
Expert #1 0.0% 0.0% 25 Percentile = 0.03%Expert #2 0.0% 0.0%Expert #3 0.1% 0.1% Median = 0.10%Expert #4 0.1% 0.1%Expert #5 0.1% 0.1% 75 Percentile = 0.10%Expert #6 1.0% 1.0%
90 Percentile = 0.55%
High = 1.00%
Minimum = 0.01% 0.01%Median = 0.10% 0.10%
Maximum = 1.00% 1.00%
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
Chickamauga Lock and Dam Expert ElicitationIssue #4 - River Wall Filling/Emptying Cracking
Event Full Description Expert-opinion elicitation SummaryName of Issue Table
First SecondQuestion #1d: Response Response
Unconstrained flow of water through the river wall adversely effecting the ability to dewater the lock and/or filling/emptying operation
Given the failure mode identified for the River Wall Blocks without any advanced maintenance (fix-as-fails scenario), what is the probability of that failure by the year 2050?
Median = 30.00% Median = 30.0%
Minimum = 25.00%
Expert #1 30.0% 30.0% 25 Percentile = 30.00%Expert #2 20.0% 25.0%Expert #3 40.0% 40.0% Median = 30.00%Expert #4 30.0% 30.0%Expert #5 50.0% 40.0% 75 Percentile = 37.50%Expert #6 30.0% 30.0%
90 Percentile = 40.00%
High = 40.00%
Minimum = 20.00% 25.00%Median = 30.00% 30.00%
Maximum = 50.00% 40.00%
Chickamauga Lock and Dam
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
Chickamauga Lock and Dam Expert ElicitationIssue #4 - River Wall Blocks
Event Full Description Expert-opinion elicitationName of Issue
First SecondQuestion #3a: Response Response
Unconstrained flow of water through the river wall adversely effecting the ability to dewater the lock and/or filling/emptying operation
Given the fix-as-fails probability of failure for the river wall filling/emptying system, what is the probability that the chamber closure time would be 30 days for less? Between 31 and 89 days? 90 days or greater? NOTE: All three branches must add up to a total of 1.
< 30 days 31-89 days 90 day + < 30 days 31-89 days 90 day +
Expert #1 99.0% 1.0% 0.0% 99.0% 1.0% 0.0%Expert #2 80.0% 15.0% 5.0% 80.0% 15.0% 5.0%Expert #3 95.0% 5.0% 0.0% 95.0% 5.0% 0.0%Expert #4 95.0% 5.0% 0.0% 95.0% 5.0% 0.0%Expert #5 90.0% 8.0% 2.0% 90.0% 8.0% 2.0%Expert #6 85.0% 10.0% 5.0% 85.0% 10.0% 5.0%
Summary Minimum = 80.00% 1.00% 0.00% 80.00% 1.00% 0.00%Table 25 Percentile = 86.25% 5.00% 0.00% 86.25% 5.00% 0.00%
Median = 92.50% 6.50% 1.00% 92.50% 6.50% 1.00%75 Percentile = 95.00% 9.50% 4.25% 95.00% 9.50% 4.25%
Maximum = 99.00% 15.00% 5.00% 99.00% 15.00% 5.00%
Chickamauga Lock and Dam
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
Cumulative Probabality of Failure (curve-fit)
0.000
0.200
0.400
0.600
0.800
1.000
2000 2010 2020 2030 2040 2050 2060 2070
Year
Pro
bab
alit
y
Cumulative
Chickamauga Lock and Dam
“ Building Strong “
Delivering Integrated, Sustainable, Water Resources Solutions
• Class Example– Six experts required– Unknown issue given to experts
• Define assumptions of issue• Elicit first values• First results• Expert Discussion• Elicit second values• Show final elicitation results
Expert-Opinion Elicitation