The uptake of value of information methods Solutions found and challenges to come Alan Brennan...
-
Upload
shannon-griffin -
Category
Documents
-
view
217 -
download
1
Transcript of The uptake of value of information methods Solutions found and challenges to come Alan Brennan...
The uptake of value of information methods
Solutions found and challenges to come
Alan BrennanDirector of Operational ResearchScHARR
Aim of this session
• Discuss the roles of VoI analysis • Demonstrate recent technical progress • Growth in interest in and use of VoI
methods• Discuss challenges for further uptake• Some recommendations
• Personal Views !
What is Value of Information?
• Given …. a choice between strategies, a decision rule for adoption of strategies,some uncertainty VoI analysis tells us…› How valuable more information would be
to reduce uncertainty, to help us choose › Which uncertain parameters are most crucial› How valuable a sample of size n=10 would be
compared to n=100, or n=1000 ….
Probabilistic Sensitivity Analysis Without VoI
• C-E plane and CEAC• i.e. describes how uncertain we are
-£2,000
-£1,500
-£1,000
-£500
£0
£500
£1,000
£1,500
£2,000
-1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6
Inc QALY
Inc
Co
st
Cost Effectiveness Acceptability of T1 versus T0
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
£- £20,000 £40,000 £60,000 £80,000 £100,000 £120,000 £140,000
Threshold (MAICER)
Pro
ba
bil
ity
Co
st
Eff
ec
tiv
e
T1
Probabilistic Sensitivity Analysis With VoI adds (1)
All
(tr
ial,
util
+du
rati
ons)
Tri
al +
Uti
lity
Tri
al
EVSI n=50£-£200£400£600£800
£1,000£1,200£1,400
EVSI n=50 2 level EVPI
EVI for Parameters: research options
• Which parameters or groups of parameters are important
• i.e. the causes - why we are uncertain
Probabilistic Sensitivity Analysis With VoI adds (2)
EVSI :- 2 Level Algorithm Results(1,000 outer x 1,000 inner simulations)
£0
£200
£400
£600
£800
£1,000
£1,200
£1,400
0 50 100 150 200 250
Sample Size (n)
EV
SI (
£)
% response/ utility trial +duration obsn'l study
% response and utility trial
duration of responseobservational study
utility observational study
% response trial
• Value of different samples and research design combinations
• i.e. how best to resolve the uncertainty
Probabilistic Sensitivity Analysis - VoI difficulty myth
• “It is complex”
• “It takes a long time”
• “Methods are not developed”
• “People can’t understand it”
Probabilistic Sensitivity Analysis - VoI truth
• If you have done a probabilistic sensitivity analysis i.e.
•Then you are 90% to 95% there
-£2,000
-£1,500
-£1,000
-£500
£0
£500
£1,000
£1,500
£2,000
-1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6
Inc QALY
Inc
Co
st
Cost Effectiveness Acceptability of T1 versus T0
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
£- £20,000 £40,000 £60,000 £80,000 £100,000 £120,000 £140,000
Threshold (MAICER)
Pro
bab
ility
Co
st E
ffec
tive
T1
Prioritising and Planning HTA
• Recommendations from systematic review of the use of modelling in planning and prioritising trials (HTA 2003; Vol 7: number 23)
• Results of Pilot Study show ..• Beneficial in refining research design and
quantifying uncertainty• choice of timing and topic are vital • who should do the analysis is important
Industry Roles – Societal Perspective
• Undertaking probabilistic sensitivity analysis and VoI to be ahead of re-imbursement authorities
• VoI to identify likelihood of cost-effectiveness and priorities for further data collection› Very early Discovery phase work› Phase II results available – now design phase III› Phase III results available – what else do we
need to make the economic case?
Industry Roles – Commercial Perspective
• Approaches still developing (and commercial in confidence)
• Define Net benefit not as λ * QALY – cost but rather as commercial
sales / profit Or • model linking sales to likelihood of re-
imbursement or extent of cost-effectiveness
Growth in UptakeA personal view
• 1996 – Claxton and Posnett – “what are all these squiggles, it will never catch on”
• 1999 – doing our review, realising the conceptual validity and practicability of VoI
• 2000 – some applications but people are unsure on methods
• 2001 – IHEA York / MDM San Diego 2 or 3 speeches (all UK)
Growth in UptakeA personal view
• 2002 – CHEBS Focus fortnight with ourselves Karl Claxton and Tony Ades – methods sorted
• Nice Appraisals beginning to use VoI• 2002 – MDM Baltimore
-7 people UK and Canada – lots of interest • 2003 – MDM Chicago
-15 to 20 people (US, Canada, Netherlands, UK) even more interest
Technical Problems Recently Solved:EVPI
• Correct method for EVPI = 2 level simulation
EVPI =
• 1 level simulation EVPI works if ….(a) the net benefit functions are linear
functions of the -i for all of the decisions d and all of the possible values of the parameter set of interest i, and (b) if i and -i are independent. “
)NB(d,max|)NB(d,max EEEd
id ii
Technical Problems Recently Solved:EVSI
• Correct Method for EVSI = 2 level simulation (Bayesian update given simulated collected data)
• EVSI easy with conjugate distributions (i.e. retain same functional form when additional data is collected and synthesised)› Normal, Beta, Gamma, Lognormal
• EVSI more complex without conjugacy
)NB(d,max|,max EXdNBEEEVSI
di
dX i
Technical Problems Recently Solved: Shortcuts
• Complex models can be ‘emulated’ e.g. Gaussian Processes, which also offer quicker EVPI and EVSI for 1 variable calculation functionality
• Laplace approximation can help shortcut to a 1 level simulation for EVSI calculations (poster)
• The EVSI curve has a common but not universal shape – exponential of square root of n
• Valuing additonal data in survival trials (poster)
)NB(d,max|,max EXdNBEEEVSI
di
dX i
Technical Problems Recently Solved: Software
• EXCEL for models• R / Splus advantages
› More sampling functionality› More optimisation functionality› Faster running times› Easier code
• WinBUGS for complex posterior distributions – interlinkage with R and Splus
Key Challenges: Uptake for Sensitivity Analysis - AIDA
• Awareness Interest Desire Action
• Publications / Conferences / Seminars• Training
York / Oxford Advanced Modelling CourseCHEBS characterising uncertainty and analysing outputs courses (2004)Bristol WinBUGS course (2004)
• Network of experts• Web resources
Key Challenges: Uptake in trial design
• Engaging with traditional trial designers• Clinically significant difference is a
proxy for the decision rule• Need to work together to compare
results and approaches on pilot trials • particularly ones with Economic and
Quality of life data to be collected
Key Challenges: Meeting Criticisms
1. How do you know uncertainty is properly characterised i.e. you might be uncertain about the uncertainty
2. You need to multiply by the number of people affected by the decision - How long does the technology / decision last – 2 years, 5 years, 10 years? An issue even without doing EVI
3. Information has value beyond the jurisdiction
Key Challenges: Technical
• Correlation, Correlation, Correlation
• Integrated VoI and evidence synthesis quickly
• Reversible decisions
• Deciding to wait – options pricing
Conclusions
• Significant recent progress, interest and growth
• Key challenges› Speeding up and teaching standard routines to
do the calculations› Working through when exactly VoI is likely to
be most valuable› Engaging sceptics to collaborate
• Now is the time ….