MBA 518 Operations Management Supply Chain Management Greg Magnan, PhD Fall, 2004.
New Research in Economic Modeling and Simulation Greg Samsa PhD.
-
Upload
cassandra-marshall -
Category
Documents
-
view
217 -
download
2
Transcript of New Research in Economic Modeling and Simulation Greg Samsa PhD.
New Research in Economic Modeling
and Simulation
New Research in Economic Modeling
and Simulation
Greg Samsa PhDGreg Samsa PhD
Organization of the talkOrganization of the talk
3 questionsHow does your work contribute to
economic and comparative effectiveness modeling?
What’s new in economic modeling and simulation?
What’s changing in how these models are applied?
3 questionsHow does your work contribute to
economic and comparative effectiveness modeling?
What’s new in economic modeling and simulation?
What’s changing in how these models are applied?
Question 1Question 1
How does your work contribute to economic and comparative effectiveness modeling?
Short answer: You provide the inputs
How does your work contribute to economic and comparative effectiveness modeling?
Short answer: You provide the inputs
Question 2Question 2
What’s new in economic modeling and simulation?
Short answer: Very complex models are now computationally feasible – the debate is shifting from what is possible to what is desired
What’s new in economic modeling and simulation?
Short answer: Very complex models are now computationally feasible – the debate is shifting from what is possible to what is desired
Question 3Question 3
What’s changing in how these models are applied?
Short answer: Decision makers are starting to take these models more seriously, and to embed them within more general strategies for learning
What’s changing in how these models are applied?
Short answer: Decision makers are starting to take these models more seriously, and to embed them within more general strategies for learning
Background Background
The previous speakers have provided definitions and examples of economic models
I’ll discuss “complex decision and cost-effectiveness models” – sufficiently complex to require simulation to implement
The previous speakers have provided definitions and examples of economic models
I’ll discuss “complex decision and cost-effectiveness models” – sufficiently complex to require simulation to implement
Observation 1Observation 1
A CEA model is a peculiar thing It is a counting machine intended to clarify
trade-offs among things that users value (e.g., survival, quality of life, costs)
Inputs are obtained from different sources – “the model” isn’t something that can be directly observed
The usual principles of validation don’t apply– I’ll discuss what makes a good model later
A CEA model is a peculiar thing It is a counting machine intended to clarify
trade-offs among things that users value (e.g., survival, quality of life, costs)
Inputs are obtained from different sources – “the model” isn’t something that can be directly observed
The usual principles of validation don’t apply– I’ll discuss what makes a good model later
Observation 2Observation 2
A dirty little secret:A CEA model is no stronger than its
weakest linkAdvocates focus on a model’s strengths,
but the weaknesses are also of importance
A dirty little secret:A CEA model is no stronger than its
weakest linkAdvocates focus on a model’s strengths,
but the weaknesses are also of importance
A personal confessionA personal confession
“The Duke Stroke Policy Model combines data from the best sources – natural history from Framingham, costs from national claims, utilities from a large survey developed specifically for this purpose…”
“The Duke Stroke Policy Model combines data from the best sources – natural history from Framingham, costs from national claims, utilities from a large survey developed specifically for this purpose…”
All true, but…All true, but…
Its conclusions depend on how survival, quality of life and costs vary by disability levelThese parameters were derived by
extrapolating from small studies of inconsistent quality
The user is essentially relying on (a) the face validity of the parameter estimates; and (b) sensitivity analyses
Its conclusions depend on how survival, quality of life and costs vary by disability levelThese parameters were derived by
extrapolating from small studies of inconsistent quality
The user is essentially relying on (a) the face validity of the parameter estimates; and (b) sensitivity analyses
N=?N=?
Natural history n=5,000Costs n=500,000Utilities n=1,500
Natural history, costs, utilities by disease state – n is variable (e.g., only 100 hemorrhagic strokes)
Impact of disability level n=20, mostly interview rather than observation
Natural history n=5,000Costs n=500,000Utilities n=1,500
Natural history, costs, utilities by disease state – n is variable (e.g., only 100 hemorrhagic strokes)
Impact of disability level n=20, mostly interview rather than observation
Question 1Question 1
How does your work contribute to economic and cost-effectiveness models?
How does your work contribute to economic and cost-effectiveness models?
Data sources for model inputsData sources for model inputs
Traditional efficacy trialsEffectiveness trialsRegistriesAdministrative dataSurveysObservational studiesLiterature reviews
Traditional efficacy trialsEffectiveness trialsRegistriesAdministrative dataSurveysObservational studiesLiterature reviews
Study planningStudy planning
Design studies to improve estimates of parameters that are: Important (e.g., using sensitivity analysis)Currently estimated with bias or
imprecision
Design studies to improve estimates of parameters that are: Important (e.g., using sensitivity analysis)Currently estimated with bias or
imprecision
ImplicationImplication
CEA models can not only organize thinking, decision making and communication about a topic, but can also be used to help set an agenda for research
CEA models can not only organize thinking, decision making and communication about a topic, but can also be used to help set an agenda for research
Question 2Question 2
What’s new in economic modeling and simulation?
What’s new in economic modeling and simulation?
Bayesian approachBayesian approach
A general approach to CEA modeling All parameter estimates are based on prior
distributions Ideally, correlations among parameters are
considered Ideally, these distributions reflect the impact
of covariatesThe output – posterior distributions – reflects
the impact of uncertainty
A general approach to CEA modeling All parameter estimates are based on prior
distributions Ideally, correlations among parameters are
considered Ideally, these distributions reflect the impact
of covariatesThe output – posterior distributions – reflects
the impact of uncertainty
Example outputExample output
“Uncertainty in all the model parameters was addressed using (a) prior distributions; and (b) resampling – in >95% of replications of the simulation the ICER was <$20,000/QALY…”
“Uncertainty in all the model parameters was addressed using (a) prior distributions; and (b) resampling – in >95% of replications of the simulation the ICER was <$20,000/QALY…”
AdvantagesAdvantages
This is a general, intellectually coherent way of modeling
Now computationally feasibleEuropeans and analysts like itA more sophisticated treatment of
uncertainty than 1- and multi-way sensitivity analysis
This is a general, intellectually coherent way of modeling
Now computationally feasibleEuropeans and analysts like itA more sophisticated treatment of
uncertainty than 1- and multi-way sensitivity analysis
DisadvantagesDisadvantages
Possible loss of transparencyParameter estimates might not be
possible / practical to obtainEasy for the model to take on a life
of its own
Possible loss of transparencyParameter estimates might not be
possible / practical to obtainEasy for the model to take on a life
of its own
What makes a good model?What makes a good model?
Model structure focuses on core of the issue
As simple as possible, but not too simple
Model is transparent Model inputs can be collected at the
required level of precision / quality
Model structure focuses on core of the issue
As simple as possible, but not too simple
Model is transparent Model inputs can be collected at the
required level of precision / quality
OpinionOpinion
Are more structurally, technically and computationally complex models such as Bayesian CEA models “good”?My opinion: sometimes
Are more structurally, technically and computationally complex models such as Bayesian CEA models “good”?My opinion: sometimes
Question 3Question 3
What’s changing in how CEA models are applied?
What’s changing in how CEA models are applied?
Back in the dayBack in the day
“Your health care organization should place our acute stroke drug on the formulary because its ICER indicates that it is good value for the money…”
“Your health care organization should place our acute stroke drug on the formulary because its ICER indicates that it is good value for the money…”
ProblemsProblems
The decision maker doesn’t have the same societal perspective as the analyst
The analysis ignores silos Even with discounting, lifetime impact is less
important to the decision maker than short term impacts
Unless accompanied by a back-of-the envelope calculation, the result isn’t transparent
The decision maker doesn’t have the same societal perspective as the analyst
The analysis ignores silos Even with discounting, lifetime impact is less
important to the decision maker than short term impacts
Unless accompanied by a back-of-the envelope calculation, the result isn’t transparent
ExampleExample
A back of the envelope model Suppose that an acute stroke drug keeps 2
people per 100 out of nursing homes. If they survive 3 years at $50,000 per year, the excess cost is $300,000 per 100 patients, or $3,000 per patient. So long as it costs less than $3,000, an acute stroke treatment that is even marginally effective is likely to be cost-effective as well.
A back of the envelope model Suppose that an acute stroke drug keeps 2
people per 100 out of nursing homes. If they survive 3 years at $50,000 per year, the excess cost is $300,000 per 100 patients, or $3,000 per patient. So long as it costs less than $3,000, an acute stroke treatment that is even marginally effective is likely to be cost-effective as well.
Current trendsCurrent trends
With calculations becoming less burdensome, it is easier to produced customized models • (e.g., including only costs of interest to the
decision maker)
Model results are embedded within more realistic frameworks such as comparative effectiveness
With calculations becoming less burdensome, it is easier to produced customized models • (e.g., including only costs of interest to the
decision maker)
Model results are embedded within more realistic frameworks such as comparative effectiveness
Ideal frameworkIdeal framework
Transparent Includes as many of the elements
of interest to the decision maker as possible
CEA model is descriptive, not prescriptive
Transparent Includes as many of the elements
of interest to the decision maker as possible
CEA model is descriptive, not prescriptive
ExampleExample
As an example of a formal decision making process intended to satisfy these criteria, I’ll describe how the oncology clinics at Duke systematically learnCEA models are one (albeit not the only)
tool that we use
As an example of a formal decision making process intended to satisfy these criteria, I’ll describe how the oncology clinics at Duke systematically learnCEA models are one (albeit not the only)
tool that we use
Oncology modeling at DukeOncology modeling at Duke
Rapid learning cancer clinicsCombine sound data collection
with an explicit mechanism for learning
Rapid learning cancer clinicsCombine sound data collection
with an explicit mechanism for learning
DataData
A data warehouse is used to generate multiple views, typically derived from linked files (e.g., cancer type, treatments, outcomes)
The lynchpin is a data set of patient-reported outcomes (derived from the PCM)
A data warehouse is used to generate multiple views, typically derived from linked files (e.g., cancer type, treatments, outcomes)
The lynchpin is a data set of patient-reported outcomes (derived from the PCM)
Data qualityData quality
The PCM contains 70+ items on a 0-10 scale (e.g., level of nausea during last 7 days)
Filled out in waiting room using e-tablets – migrating to web
Results are reported to clinicians in real time – for example, highlighting issues to discuss during the visit
The PCM contains 70+ items on a 0-10 scale (e.g., level of nausea during last 7 days)
Filled out in waiting room using e-tablets – migrating to web
Results are reported to clinicians in real time – for example, highlighting issues to discuss during the visit
IncentivesIncentives
Patients: confident that their concerns won’t be overlooked
Physicians: saves time in performing a review of symptomsPrinciple: (Sufficiently) valid data are
produced by design, not by accident
Patients: confident that their concerns won’t be overlooked
Physicians: saves time in performing a review of symptomsPrinciple: (Sufficiently) valid data are
produced by design, not by accident
Formal learning structureFormal learning structure
Learning from the databases occurs within a formal PDCA cycleRelevant stakeholders are representedThe stakeholders determine the level of
accuracy / precision required to make decisions
The stakeholders determine study design (e.g., interventional, observational)
Learning from the databases occurs within a formal PDCA cycleRelevant stakeholders are representedThe stakeholders determine the level of
accuracy / precision required to make decisions
The stakeholders determine study design (e.g., interventional, observational)
Types of designsTypes of designs
Observational designs with undirected machine learning
Observational designs with pre-specified hypotheses
Pre-post designs with interventions
Randomized trials
Observational designs with undirected machine learning
Observational designs with pre-specified hypotheses
Pre-post designs with interventions
Randomized trials
ExampleExample
Is it “worth it” to refer patients with high levels of psychological distress to specialized counseling?
Is it “worth it” to refer patients with high levels of psychological distress to specialized counseling?
InputsInputs
Observational data – natural history of outcomes by level of distress
CEA model to estimate what level of improvement would justify use of specialized counseling resources
Literature review on expected impact of counseling
Pre-post design to assess impact of counseling in our setting
Observational data – natural history of outcomes by level of distress
CEA model to estimate what level of improvement would justify use of specialized counseling resources
Literature review on expected impact of counseling
Pre-post design to assess impact of counseling in our setting
Criteria for learningCriteria for learning
A practice is worth changing if the alternative is cost-effective
We use CEA models that are simple to moderately complex
A practice is worth changing if the alternative is cost-effective
We use CEA models that are simple to moderately complex
CommentComment
Our goal is to systematically and explicitly embed learning into our usual procedures
Our goal is to systematically and explicitly embed learning into our usual procedures
Final thoughtsFinal thoughts
Health economics has always been quantitative – now, it is becoming more explicitly “statistical” as well
A statistically-inspired literature on CEA is rapidly developing – a distinguishing characteristic is the ability to accommodate increasingly complex models through advances in computation
Health economics has always been quantitative – now, it is becoming more explicitly “statistical” as well
A statistically-inspired literature on CEA is rapidly developing – a distinguishing characteristic is the ability to accommodate increasingly complex models through advances in computation
Final thoughtsFinal thoughts
The danger in this literature is that, if its perspective is entirely statistical, it can become divorced from reality
A particular area of promise lies in integrating CEA modeling with modern systematic approaches to learning
The danger in this literature is that, if its perspective is entirely statistical, it can become divorced from reality
A particular area of promise lies in integrating CEA modeling with modern systematic approaches to learning