When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and...
-
Upload
alice-nevills -
Category
Documents
-
view
213 -
download
0
Transcript of When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and...
![Page 1: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/1.jpg)
When is there Sufficient Evidence?
Karl Claxton,
Department of Economics and Related Studies and
Centre for Health Economics,
University of York.
![Page 2: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/2.jpg)
Sufficient evidence for decision making
• Informing decisions• Prioritising research
– Examples from the NHS HTA Pilot
• Designing research• Evaluating regulatory policies • Implementation• UK policy impact• Implications and challenges
![Page 3: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/3.jpg)
The decisions
• Should a technology be adopted given existing information?
– Which clinical strategies are cost-effective?
– For which patient groups?
• Is additional evidence required?
– What type of evidence
– What type of studies
– For which patient groups
– How much evidence
![Page 4: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/4.jpg)
Informing decisions
• Requirements– Structuring clinical decision problems
– Characterisation of decision uncertainty
– Value of additional research
– Decisions consistent with objectives and constraints
![Page 5: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/5.jpg)
Structuring clinical decision problems
• Requirements– Compare all alternative interventions/strategies
– Explore the full range of clinical policies
– For range of patient groups
– Over an appropriate time horizon
– Combine evidence from variety sources
![Page 6: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/6.jpg)
Characterising uncertainty
• Evidence for all model parameters– Systematic searching for all parameters
– Methods of synthesis
• Direct evidence– Different types of study (potential bias/exchangeability)
• Networks of evidence– Indirect and mixed comparisons
– Evidence on functions of parameters
• Probabilistic analysis of the decision model
![Page 7: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/7.jpg)
Mixed and indirect comparisons
A B C D
1 x x
2 x x
3 x x
4 x x
5 x x
6 x x
Alternative interventions
RCTs• Compare all interventions
• No RCT of all• Pair wise comparisons?• Use all the information
– Mixed comparisons– Same assumptions as
random effect meta analysis
– Estimate posterior LOR with correlations
![Page 8: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/8.jpg)
Networks of evidence
A
B+C
B C
A+B+C
A+B A+C
![Page 9: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/9.jpg)
Is further evidence required?• Expected value of perfect information (EVPI)
– Maximum return to research (decision problem)
– Necessary condition (EVPI>costs)
What type of evidence?• EVPI for parameters
– Maximum return to different types of research
– Focusing research design
Value of additional research
![Page 10: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/10.jpg)
An example
• A pilot study of value of information analysis to inform the NHS health technology assessment programme
– Screening for age-related macular degeneration
– Manual chest physiotherapy techniques for asthma and chronic obstructive pulmonary disease
– *long-term antibiotic treatment for preventing recurrent urinary tract infections (UTI) in children*
![Page 11: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/11.jpg)
Structuring the decision problem
No UTI
1 UTI
2 UTIs
3 UTIs
4 UTIs
Age atESRD onset
Frequency of recurrent UTIs
Number of pyelonephritic attacks
Progressive renal scaring
End-stage renal disease
TransplantPyelonephritic
attack
Pyelonephritic attack
Pyelonephritic attack
Pyelonephritic attack
Number of attacks
Progressive renal
scaringDevelopment
of ESRD
Dialysis
No UTI
1 UTI
2 UTIs
3 UTIs
4 UTIs
Age atESRD onset
Frequency of recurrent UTIs
Number of pyelonephritic attacks
Progressive renal scaring
End-stage renal disease
TransplantPyelonephritic
attack
Pyelonephritic attack
Pyelonephritic attack
Pyelonephritic attack
Number of attacks
Progressive renal
scaringDevelopment
of ESRD
Dialysis
![Page 12: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/12.jpg)
The evidence
• Effectiveness– Existing reviews (variable quality)
– Meta analysis, Multiple parameter synthesis
• Natural history– Epidemiological studies
– Pooled trial baselines
– Registry studies
• Quality of life – Published studies
– Survey
• Costs – Published studies
– Published unit costs and dosage (BNF, PSSRU, CIPFA)
![Page 13: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/13.jpg)
Characterising decision uncertainty
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
£0 £10,000 £20,000 £30,000 £40,000 £50,000 £60,000
Threshold for cost-effectiveness
Probability cost-effective
Intermittent
Cotrimoxazole
Nitrofurantoin
Trimethoprim
Frontier
![Page 14: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/14.jpg)
The irrelevance of inference?
• The choice between alternative technologies should be based on expectation. Inference is irrelevant to treatment choice
• The only valid reason to characterise the uncertainty surrounding outcomes of interest is to establish the value of additional information
• Distinguish the separate steps of deciding which technology should be chosen, given existing information, from the question of whether more evidence is required to support this decision
![Page 15: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/15.jpg)
• Expected cost of uncertainty– Adoption based on existing information is uncertain
– There is a probability of making the wrong decision
– There are costs of making the wrong decision
• Expected value of perfect information (EVPI) – Maximum return to research (decision problem)
– Necessary condition (EVPI>costs)
Is further evidence required?
![Page 16: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/16.jpg)
EVPI for the decision problem
Treatment A Treatment B Treatment C Optimal treatment Max NB
Iteration 1 11 12 13 C 13
Iteration 2 12 10 9 A 12
Iteration 3 13 18 15 B 18
Iteration 4 14 16 17 C 17
Iteration 5 15 14 11 A 15
Expected NB 13 14 13
Current information = 14Perfect information = 15EVPI = 15- 14 = 1
15
EVPI = Eθ maxj NB(j, θ) - maxj Eθ NB(j, θ)
![Page 17: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/17.jpg)
Population EVPI
£0
£500,000
£1,000,000
£1,500,000
£2,000,000
£2,500,000
£3,000,000
£3,500,000
£4,000,000
£0 £10,000 £20,000 £30,000 £40,000 £50,000 £60,000
Cost-effectiveness threshold
Populaion EVPI
Is further evidence required?
![Page 18: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/18.jpg)
EVPI for parameters
Some implications:• Information about an input is only valuable if it
changes our decision • Information is only valuable if the parameter does
not resolve at its expected value
General solution (linear and non linear models)
EVPPIφ = Eφ maxj Eψ |φ NB(j, φ,ψ) – maxj Eθ NB(j,θ)
Where: φ = parameter of interest
ψ = other uncertainties
![Page 19: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/19.jpg)
What type of evidence?
£0
£500,000
£1,000,000
£1,500,000
£2,000,000
£2,500,000
Full model
Frequency of UTI
Frequency of pyelonephritis
Risk of PRS
Utility of UTI/pyelonephritis
ESRD
Development of ESRDConsequences of ESRD
Effect of Long-term antibiotics
Effect of cotrimoxazoleEffect of nitrofuranatoinEffect of trimethoprimEffect < 6 monthsEffect > 6 months
Expected Value of Perfect Information
![Page 20: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/20.jpg)
Prioritising research
• Other methods– Unrelated to the returns from research– Research as a means changing clinical practice
• Statistical decision theory– Reduction in the costs of decision uncertainty– Value consistent with objective and constraints of
service provision– Allocation within and between clinical areas– Allocation between research and service provision
![Page 21: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/21.jpg)
Design of future research
• EVPI exceeds cost of research?– Necessary but not sufficient
• Benefit of sample information – Predict possible future samples and posteriors
– Expected net benefits with and with out sample information
– Expected value of sample information (EVSI)
• Societal payoff from proposed research– EVSI - cost of sampling
– Expected net benefit of sampling (ENBS)
![Page 22: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/22.jpg)
EVSI: predicting the results of proposed research
0.0 0.2 0.4 0.6 0.8 1.0
Probabi lity
Prior DistributionPredicted Posterior Distributions
N=1
0.0 0.2 0.4 0.6 0.8 1.0
Probabi lity
Prior DistributionPredicted Posterior Distributions
N=20
NBj
NBj
NBj
NBjNBj NBk
![Page 23: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/23.jpg)
£0
£500,000
£1,000,000
£1,500,000
£2,000,000
£2,500,000
£3,000,000
£3,500,000
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500
Sample size (n)
EVSI, cost(n), ENBS
EVSI
Cost(n)
ENBS
Expected net benefit of sample information (all parameters)
![Page 24: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/24.jpg)
£0
£200,000
£400,000
£600,000
£800,000
£1,000,000
£1,200,000
£1,400,000
£1,600,000
£1,800,000
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500
Sample size (n)
Expected net benefit of sample information
PIP
RSD,PHZ
UPA
ENBS and n* for a portfolio of research designs
![Page 25: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/25.jpg)
Efficient research designs
• Sufficient condition (ENBS>0)
• Optimal sample size
• Allocation of patients to the arms of a trial
• Optimal stopping of sequential trial
• Which endpoints should be included?
• Optimal follow-up
• What are relevant alternatives
• Optimal portfolios of research
• Optimal development programmes
![Page 26: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/26.jpg)
Regulation of health technologies
“...A reasonable basis for a claim [of cost-effectiveness] depends on a number of factors relevant to the benefits and costs of substantiating a particular claim. These factors include: the type of product, the consequences of a false claim, the benefits of a truthful claim, the costs of developing substantiation for the claim ...”
(FDA, section 114, modernization act of 1997)
![Page 27: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/27.jpg)
Regulation of health technologies
• Is it efficient to conduct further research?– Define a claim/submission as “substantiated” and evidence
as “competent and reliable” when it is not efficient to conduct further research
• Should demand:– More evidence for some technologies as compared to others– Different types of evidence for different technologies – Different amounts of evidence for the same technology in
different circumstances
• Role of the regulator?– Policing the priors
![Page 28: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/28.jpg)
Evaluating regulatory policies
• Societal value of information– Socially optimal development decisions
– Socially optimal amount/type of evidence
• Commercial value of information– Commercial payoff function
– Payoff conditional on licensing and reimbursement
– Commercially optimal development decisions
– Commercially optimal amount/type of evidence
• Optimal policies– Were commercial development and information decisions
match societal needs
![Page 29: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/29.jpg)
Implementation
• Realising the value of information
- EVPI = D-C
- “Realisable” EVPI = B-A
• Value of changing clinical practice
- Current information = C-A
- With additional information = D-B
• Evaluate policies to change clinical practice
- Can invest in information, implementation or both = D-A
Information
Implementation Current “Perfect”
Current A B
“Perfect” C D
• Value of information and the value of implementation
![Page 30: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/30.jpg)
UK Policy impact
• National Institute for Clinical Excellence– Pilot study to inform research recommendations
– Guidance on methods of technology appraisal– Decision analytic modelling required
– Evidence synthesis and probabilistic analysis required
– Value of information analysis recommended
• NHS Health Technology Assessment Programme– Pilot study of setting research priorities
• Medical Research Council– Programme on value of information
– Programme on evidence synthesis for decision making
– Cancer Trials Unit Programme
![Page 31: When is there Sufficient Evidence? Karl Claxton, Department of Economics and Related Studies and Centre for Health Economics, University of York.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c955503460f94952885/html5/thumbnails/31.jpg)
Implications and challenges
• Rational decision making- Adoption decisions
- Research decisions
- Additional objectives, constraints and irreversibility
• Characterising uncertainty- Indirect comparisons and networks of evidence
- Bias and exchangeability
- Model structure
• Computation- Correlation and complex data structures
- Non multi-linear and complex models
- Dimensions of design space