Sequential Experimentation: A Different Approach to Prevention Research
-
Upload
cairo-walker -
Category
Documents
-
view
25 -
download
1
description
Transcript of Sequential Experimentation: A Different Approach to Prevention Research
![Page 1: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/1.jpg)
Sequential Sequential Experimentation: Experimentation: A A
Different Approach to Different Approach to
Prevention ResearchPrevention Research Annual Meeting of the Society for Prevention Research
Washington, DC
June, 2003
![Page 2: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/2.jpg)
Linda M. CollinsThe Methodology Center and
Department of Human Development & Family Studies
Penn State
Susan A. MurphyDepartment of Statistics and Institute for Social Research
University of Michigan
Vijay NairDepartments of Statistics and Industrial & Operations Engineering
University of Michigan
![Page 3: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/3.jpg)
OverviewOverview
• Why we are proposing a change in how we conduct prevention research
• Introduction to sequential experimentation
• Advantages we see to sequential experimentation
• Considerations and open questions
![Page 4: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/4.jpg)
Two broad scientific issues facing Two broad scientific issues facing prevention scienceprevention science
![Page 5: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/5.jpg)
How can we improve our understanding How can we improve our understanding of the mechanisms underlying of the mechanisms underlying
prevention?prevention?
• In early days of prevention, the field struggled to establish THAT prevention worked
• Now, the time is right to begin understanding HOW, WHY and FOR WHOM prevention works
• This will allow us to fine-tune, even optimize, interventions
![Page 6: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/6.jpg)
How can we hasten the building of a How can we hasten the building of a coherent body of knowledge about coherent body of knowledge about
prevention?prevention?
• The field has moved forward in important ways – but slowly
• Solid, replicable results are mixed with unreplicable and contradictory findings
• We need to speed up the accumulation of consistent, replicable findings
![Page 7: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/7.jpg)
We argue that a new approach is needed in order to address these issues.
But what is the difficulty with the current approach?
![Page 8: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/8.jpg)
Two prevention program Two prevention program evaluation research agendasevaluation research agendas
• Answering the bottom-line question of whether a program is effective
• Informing theory and design of prevention programs
![Page 9: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/9.jpg)
Two prevention program Two prevention program evaluation research agendas, evaluation research agendas,
one approachone approach
• We try to address both agendas with large, expensive prevention trials involving– lots of subjects
– power for detecting treatment effects
– lots of variables– post hoc analyses with the hope of informing theory and
figuring out how the program worked (or why it didn’t work)
![Page 10: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/10.jpg)
Two prevention research Two prevention research program evaluation research program evaluation research
agendas, one approachagendas, one approach
• This approach tries to serve both agendas simultaneously, but is it the BEST way to serve either one?
![Page 11: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/11.jpg)
How can we best inform How can we best inform theory?theory?
• Post-hoc or secondary analyses can be very misleading – e.g. post-hoc “dosage received” analyses
• Best way to inform theory: series of randomized experiments
![Page 12: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/12.jpg)
How can we best answer How can we best answer questions about program questions about program
effectiveness?effectiveness?
• Even sound post-hoc analyses cannot help improve the prevention trial that generated the data
• Best way to maximize program effectiveness: obtain data to inform the prevention trial BEFORE it is conducted
![Page 13: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/13.jpg)
We propose a different approach: sequential experimentation
![Page 14: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/14.jpg)
OverviewOverview
• Why we are proposing a change in how we conduct prevention research
• Introduction to sequential experimentation
• Advantages we see to sequential experimentation
• Considerations and open questions
![Page 15: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/15.jpg)
A different approach: A different approach: Sequential experimentationSequential experimentation
• Emerged from agriculture and engineering
• Philosophy: – scientific research is an iterative process of
deduction and induction– constant interplay among theory, experimentation,
data analysis
![Page 16: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/16.jpg)
A different approach: Sequential A different approach: Sequential experimentationexperimentation
• In engineering
– theory suggests many possible treatment combinations that would require a prototype
– not practical to build ALL prototypes – each one is resource-intensive
![Page 17: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/17.jpg)
A different approach: Sequential A different approach: Sequential experimentationexperimentation
• In prevention
– Theory suggests many candidate components and combinations of components that could make up a preventive intervention
– Not practical to bring EVERY possible combination of components to a prevention trial – each trial is resource-intensive
![Page 18: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/18.jpg)
The objective of sequential The objective of sequential experimentationexperimentation
• Sort through the space of all possible treatment combinations and dosages in an efficient way, to select programs and dosages most likely to be successful
![Page 19: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/19.jpg)
Sequential experimentation: A Sequential experimentation: A sequence of three phasessequence of three phases
• Screening of candidate program components
• Refining of program components
• Confirming the effectiveness of a refined program
![Page 20: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/20.jpg)
Screening phaseScreening phase
• Purpose: to sort through candidate components to decide which are effective and should be investigated further
• Starting point: candidate intervention components suggested by theory– e.g. normative education, resistance training, parental component
• Examine each via a series of randomized controlled experiments
![Page 21: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/21.jpg)
Screening Screening phasephase
• Components identified as effective in the screening phase are tentatively selected for inclusion in the intervention.
• Next, go to refining phase.
![Page 22: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/22.jpg)
Refining phaseRefining phase
• Purpose: identify interactions, investigate blocking variables and putative moderators, refine dosage– e.g.:
examine whether normative education and parental component interact
examine effects of gender determine optimal number of resistance training lessons
• Starting point: components identified as effective in the screening phase
![Page 23: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/23.jpg)
Refining phaseRefining phase
• Series of randomized experiments on components that passed the screening phase
• Depending on results, may go back to screening, or on to confirming phase
![Page 24: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/24.jpg)
Confirming phaseConfirming phase
• Purpose: construct and test preventive intervention
• Starting point: active program components, appropriate dosages, and important moderators identified during screening and refining phases
![Page 25: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/25.jpg)
Confirming phaseConfirming phase
• Develop a refined intervention, consisting of only active, effective components, delivered in appropriate amounts
• A full randomized prevention trial testing the refined intervention
![Page 26: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/26.jpg)
A few key points about sequential A few key points about sequential experimentationexperimentation
• Theory plays a critically important role, particularly in the screening and refining phases– Selection of candidate components– Informing the designs used
![Page 27: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/27.jpg)
A few key points about sequential A few key points about sequential experimentationexperimentation
• Extensive use made of designs such as balanced fractional factorials and response surfaces
• These designs use theory to inform choices about where to concentrate resources
• Much more efficient than full factorial designs– Number of cells and sample size requirements can be greatly
reduced, WHILE MAINTAINING STATISTICAL POWER
![Page 28: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/28.jpg)
A few key points about sequential A few key points about sequential experimentationexperimentation
• The screening and refining phases are NOT pilot studies
• They are randomized, controlled studies
![Page 29: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/29.jpg)
A few key points about sequential A few key points about sequential experimentationexperimentation
• Type II errors (overlooking active prevention component) may be worse here than Type I errors (mistakenly selecting inactive component)
• Type I error rate greater than usual p<.05 may be used
![Page 30: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/30.jpg)
OverviewOverview
• Why we are proposing a change in how we conduct prevention research
• Introduction to sequential experimentation
• Advantages we see to sequential experimentation
• Considerations and open questions
![Page 31: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/31.jpg)
Advantages we see to the Advantages we see to the sequential experimentation sequential experimentation
approachapproach
•Screening, refining phases address “how, why, and for whom” questions
![Page 32: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/32.jpg)
Advantages we see to the Advantages we see to the sequential experimentation sequential experimentation
approachapproach
•Results of screening and refining phases will be replicable and will lead to a coherent body of knowledge
![Page 33: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/33.jpg)
Advantages we see to the Advantages we see to the sequential experimentation sequential experimentation
approachapproach
• Programs that go to prevention trials have been refined; the idea is to produce larger effect sizes
increased statistical powerincreased public health benefits
![Page 34: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/34.jpg)
Advantages we see to the Advantages we see to the sequential experimentation sequential experimentation
approachapproach
• It may allow for more creative risk-taking– it is possible to try a brand-new approach
in the screening phase without the investment of an entire prevention trial
![Page 35: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/35.jpg)
Advantages we see to the Advantages we see to the sequential experimentation sequential experimentation
approachapproach
• Less likely to run into unanticipated problems
• Thus less likely to be presented with the familiar dilemma:– Midway through the prevention trial we realize
the program needs revision, but we don’t want to change the program and thereby make the evaluation difficult
![Page 36: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/36.jpg)
OverviewOverview
• Why we are proposing a change in how we conduct prevention research
• Introduction to sequential experimentation
• Advantages we see to sequential experimentation
• Considerations and open questions
![Page 37: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/37.jpg)
Considerations and open Considerations and open questionsquestions
• In most cases, the screening and refining phases can occur in a five-year funding cycle
• The confirming phase can occur in a second five-year funding cycle
• The support of NIH program staff will be needed
![Page 38: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/38.jpg)
Considerations and open Considerations and open questionsquestions
• Hierarchical data– we are working on this
• Steady stream of research subjects needed– series of experiments may take place over
several years– possibility of cohort effects
![Page 39: Sequential Experimentation: A Different Approach to Prevention Research](https://reader035.fdocuments.us/reader035/viewer/2022062517/56813174550346895d97ecfe/html5/thumbnails/39.jpg)
Considerations and open Considerations and open questionsquestions
•You can think of many others!