Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial...
Transcript of Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial...
![Page 1: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/1.jpg)
Musttheevaluationofcomplexinterventionsbe
complex?
IsoldeBirdthistleandSianFloydWithDanielKwaro,MaryamShahmanesh,DanielCarter,Abdhalah Ziraba,
FrancesCowan,KathyBaisley,JudithGlynnandcolleagues
ISSTDR2019
LearningfromtheimpactevaluationofDREAMS
![Page 2: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/2.jpg)
The fascination of what’s difficultHas dried the sap out of my veins, and rent
Spontaneous joy and natural content Out of my heart.
WBYeats,1916
![Page 3: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/3.jpg)
1. Whatisacomplexintervention?WhatmakesDREAMSone?
2. Howdoyouevaluateacomplexintervention?Withoutrandomisation,canwebenefitfromemulatingatrial?
3. WhatarethelessonslearnedfromevaluationofDREAMSsofar?
Areflectionin3parts
![Page 4: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/4.jpg)
TodrivedownHIVincidence,increasingconsensusthat:oMultiplestrategiesandmultiplesectorsareneeded(nosingleinterventionislikelytoworkalone)
o Thestrategiesmaydifferbycontext&ageo Similartrendinadolescenthealth&development…
Increasingadvocacyfor:o Combinationpackageso Coordinatedresponses
Complexinterventions
Thechangingprogrammelandscape
What makesaninterventioncomplex?
![Page 5: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/5.jpg)
“Complexityresidesin…
…thenumberofinteractingcomponents,thenumberanddifficultyofbehavioursrequiredbythosedeliveringorreceivingtheintervention,thenumberofgroupsororganisational levelstargetedbytheinterventions,thenumberandvariabilityofoutcomes,andthedegreeofflexibilityortailoringoftheinterventionpermitted.”
MarkPetticrew,“Whenarecomplexinterventions‘complex’?Whenaresimple interventions ‘simple’?”EuropeanJofPublicHealth,2011
![Page 6: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/6.jpg)
“Today,weareannouncingthatPEPFARisnowinvestingnearlyhalfabilliondollarstosupportanAIDS-freefutureforadolescentgirlsandyoungwomen.”
- USNationalSecurityAdvisorSusanERice,26Sept2015
![Page 7: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/7.jpg)
“DREAMSisaboutmultiple
solutionssurroundingoneproblem:
newHIVinfectionsamong
adolescentgirlsandyoung
women.”
PEPFAR.GOV
![Page 8: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/8.jpg)
DREAMStargets4relatedgroups…
![Page 9: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/9.jpg)
ThroughaCorePackagewithmanyinteractingcomponents…withmanyinteractingcomponents
![Page 10: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/10.jpg)
Heterogeneityinreal-worldimplementation- across65+districtsin15countries
15 DREAMS
![Page 11: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/11.jpg)
Heterogeneityinreal-worldimplementation- across65+districtsin15countries
DifferentinterventionsindifferentcontextsDifferentmodelsofimplementation
Differentstrategiesfor”layering”(mentors,safespaces,‘passports’,uniqueIDs…)
Chimbindi &Birdthistle:‘TranslatingDREAMSintopractice:Earlylessonsfromimplementationinsixsettings’
PLOSOne2018
15 DREAMS
![Page 12: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/12.jpg)
“Complexityresidesin……thenumberofinteractingcomponents,thenumberanddifficultyofbehavioursrequiredbythosedeliveringorreceivingtheintervention,thenumberofgroupsororganisationallevelstargetedbytheinterventions,thenumberandvariabilityofoutcomes,andthedegreeofflexibilityortailoringoftheintervention permitted.”
MarkPetticrew,“Whenarecomplexinterventions‘complex’?Whenaresimple interventions ‘simple’?”EuropeanJofPublicHealth,2011
Inalltheseways,andmore,DREAMSisacomplex
intervention
![Page 13: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/13.jpg)
Howdoweevaluatesuchacomplexintervention?
(whilepreserving clarityandutilityandspontaneous joy…?)
![Page 14: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/14.jpg)
Itdependsonourperspective&question
AcomplexquestionHowandwhetherthecomponentsworkindividually andtogether?Consideringtheirsynergies,phasechanges,feedbackloops,interactionsbetweenoutcomes,andtheprocessbywhichthecomponentsbringaboutchange.
AsimplequestionIsthewholepackageassociatedwithimprovedhealth?
MarkPetticrew,EuropeanJofPublicHealth,2011
![Page 15: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/15.jpg)
Itdependsonourperspective&question
AcomplexquestionHowandwhetherthecomponentsworkindividually andtogether?Consideringtheirsynergies,phasechanges,feedbackloops,interactionsbetweenoutcomes,andtheprocessbywhichthecomponentsbringaboutchange.
AsimplequestionIsthewholepackageassociatedwithimprovedhealth?
MarkPetticrew,EuropeanJofPublicHealth,2011
Westartedhere,because…EvidenceoftheindividualcomponentsisalreadyknownTheknowledgegapiswhethertheyhaveanimpactwhendeliveredtogetherasapackage.
![Page 16: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/16.jpg)
Howtoanswera‘simplequestion’aboutimpact,given…
Norandomisation
Whynot?Timing,feasibility,ethics
• Anurgencytobeginroll-outofDREAMS• DREAMSwouldtargetthemostvulnerableadolescent
girlsandyoungwomeninprioritydistricts• Equipoise:theinterventionwasexpectedtobebeneficial
(hardtojustifycontrolsorlargeexpenseofatrial)
![Page 17: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/17.jpg)
Populationlevelchange
Individuallevelchange
Processevaluation
• Largecommunity-wide opencohorts
• Analysed asC/Sbefore,during,after
• Nested,closedcohortsofAGYW
• Analysedlongitudinally
• In-depthqualitativeresearch
Withoutrandomisation(balance),canweestimatecausaleffects?
Anobservationaldesign– usingpopulation-basedlongitudinaldatawithindemographicsurveillancesites
![Page 18: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/18.jpg)
“TheC-Word…”
“…‘causal’[must]stopbeingconsideredtheC-wordthatinvestigatorsandeditorsavoid.Onlybypreciselydefiningthecausaleffectofinterestwillwehaveachanceofestimating itaccurately.”
MiguelHernan,“TheC-Word:Themorewediscussit,thelessdirtyitsounds”AJPH2018
![Page 19: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/19.jpg)
Answeringcausalquestionsusingobservationaldatabyemulatingatarget(hypothetical)trial
1. Specifyingthetargettrial2. Emulatingthetargettrial3. Triangulating
MiguelHernan, AJPH2018
Applyingcausalinferenceapproaches
Thoughnotrandomised,theprinciples&techniquesofarandomisedtrialcanhelp
![Page 20: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/20.jpg)
Populationlevelchange
Individuallevelchange
Processevaluation
• Largecommunity-wide opencohorts
• Analysed asC/Sbefore,during,after
• Nested,closedcohortsofAGYW
• Analysedlongitudinally
• In-depthqualitativeresearch
Applyingcausalinferenceapproaches
![Page 21: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/21.jpg)
1.Specifyingthetargettrial(thehypotheticalexperiment)
Classificationof‘treatment’groups
WhoisaDREAMSbeneficiary?apriorimeasures
• InvitedtoparticipateinDREAMSversusnot[analogoustoITT]
• Invitedandreceivedmin3corepackagecategoriesversus 0-2
Mustbeupdatedovertime(withnewbeneficiaries)…
Categoricalmeasure1. Never/None2. 2017only3. 2018only4. 2017&2018
Binarymeasureo AnyDREAMSby2018:
Yes/No
![Page 22: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/22.jpg)
62
29
2
914
5
22
48
1
91
38
1014
24
10
48
1014
0
86
48
2428
48
22
53
6
1713
83
46 45 4449
38
64
9
2723
0
10
20
30
40
50
60
70
80
90
100
HIVTesting&Counselling Socialassetbuilding Condom promotion andprovision Socialprotection SchoolbasedHIVprevention
CorePackage Categories
Never invited (N=166)
Invited in 2017 only (N=21)
Invited in 2018 only (N=109)
Invited both in 2017 & 2018 (N=281)
Expand contraceptive mix
Post-violence care
PrEP Parenting/ Caregiver programs
Community mobilisationand norms changes
ParticipationinDREAMScorepackageinterventionsin2018:18-22/24AGYWinNairobi
![Page 23: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/23.jpg)
2.Emulatingthetargettrial
‘Randomassignment’oftreatmentgroupsAimtoachieve(near)balanceonbaselinecovariatesà Adjustforall(measured)confoundingfactors,e.g.,
throughpropensityscore*adjustment
Generateoverallcausaleffectsà Predictoutcomeforfullsampleifallversus ifnone
gotDREAMS*Propensity score=probability ofreceivingtheintervention basedonconfounder values(useful ifmanyco-variates,esp/forrareoutcomes;comparedwithadjustmentforeachindividual confounding variable)
Applyingcounterfactualreasoningforcausalinference
![Page 24: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/24.jpg)
EarlyimpactsexpectedonKnowledgeofHIVStatus
Whatwouldbethedifference intheproportionofAGYWwhoknowtheirstatusifeverybodygotDREAMScomparedtoifnobodygotDREAMS?
Framingincausallanguagehelpsclearlyarticulatethequestionandidentifyasuitableestimand
ExamplesfromDREAMSevaluation(beforeendline dataareavailable)
![Page 25: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/25.jpg)
DAGforNairobi
![Page 26: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/26.jpg)
65.161.1
70.968.1
63.5
75.5
42.3
26.5
61.1
92.9 92.8 93.5
80.276.8
85.1
45.9
35.4
58.3
0
10
20
30
40
50
60
70
80
90
100
All:15-22 15-17 18-22 All:13-22 13-17 18-22 All:13-22 13-17 18-22
Nairobi[by2017] Siaya[by2018] uMkhanyakude [by2017]
NoAGYWareaDREAMSbeneficiaryAllAGYWareaDREAMSbeneficiary
Predictedproportions whoKnowtheirHIVStatus ifnoneversusallbenefitedfromDREAMS(in3DREAMSsettings)
![Page 27: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/27.jpg)
Theabsolutedifference• Nairobi:27.7%increase[95%CI:22.8%,32.6%]• Gem:12.1%increase[95%CI7.7-19.6]• KwaZulu Natal:effectmodificationbyage
– 13-17Yearolds:8.95%[95%CI4.8%,14.4%]– 18-22Yearolds:-2.8%[95%CI-11.1%,5.7%]
• ImportanceofCONTEXT– Verydifferenteffectsacrosssite;agegroup
• ImportanceofMECHANISM(how?)– differenteffectsdependingontargetingand
implementation – needprocesseval data!)
Backtothecausalquestion…WhatwouldbethedifferenceintheproportionofAGYW
whoknowtheirstatusifeverybodygotDREAMScomparedtoifnobodygotDREAMS?
![Page 28: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/28.jpg)
Processevaluationofcomplexinterventions:MedicalResearchCouncilguidanceBMJ 2015;350doi:https://doi.org/10.1136/bmj.h1258
Notenoughtoknowwhetheraninterventioniseffective,orevenbyhowmuch.
Importanttounderstandhowandwhy,andforwhom,especiallyinthe‘real-world’,undernon-trialconditions,ifwewantlessonsforreplication.
Forcomplexinterventions,‘outcomeevaluation’maynotbeenough
“Effectsizesdonotprovidepolicymakerswithinformationonhowaninterventionmightbereplicatedintheirspecificcontext,orwhethertrialoutcomeswillbereproduced.”
![Page 29: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/29.jpg)
Processevaluation- Anessentialpartofdesigningandtestingcomplexinterventions- Guidedby3keythemes…
![Page 30: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/30.jpg)
HighKnowledgeofHIVStatusinNairobiAreflectionofhowtheinterventionwasdeliveredandreceivedinthiscontext?
![Page 31: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/31.jpg)
Insightsfromprocessevaluation
DeliveryofHIVtestingthrough DREAMSinKenya…• HIVtestingwasofferedattimeof
enrolment intoDREAMS,toallAGYW,regardlessofage,circumstance,orperceivedrisk
• AllDREAMSinterventions arecoordinatedbyoneIP,sothisapproachisconsistentandsystematic
• IPswereexperienced inHIVtestingprior toDREAMS
• Testingmadeavailableincommunity-basedsettings,including DREAMSsafespaces,home-based testing,andreferralstofacilities
![Page 32: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/32.jpg)
Insightsfromprocessevaluation
HIVtestingthrough DREAMSwaspositivelyreceivedbyAGYWinKenya,e.g.,fortheconfidentiality…
“Initially there were these people who were afraid of going to the hospital, but right now you find that the HTS person comes to the safe space at least you can have the courage. Because for them they will come and test you and leave, they won’t talk about your results to other people. They will just tell you personally.”
- FGDwithoutofschoolAGYW
![Page 33: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/33.jpg)
Musttheevaluationofcomplexinterventionsbe
complex?
Ananswer:“Whenitishelpfultoseeandanalysethemassuch”
MarkPetticrew,EuropeanJofPublicHealth,2011
(And itusuallyis)
![Page 34: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/34.jpg)
Someclarityinthefaceofcomplexity…?
Ifthecomplexinterventionisnotrandomised,don’tnecessarilyshyawayfromcausality
Aimforcausalinference,whilewaryoftheassumptionsthatcaneasilybeviolatedunlesswe:
• Knowtheintervention,howitisimplemented(by/withwhom)andhowitchangesindifferentcontextsandovertime
• Consider,measureandaccountforconfounders• Triangulate:Usemultiple,complementaryapproachestoanswer
thequestion(withdifferentadvantagesanddisadvantages)
”Gesamtkunstwerk”(German: [gəˈzamtˌkʊnstvɛʁk],translatedas"synthesisofthe
arts",whendifferentformsarecombinedintoasingleunifiedwhole)
Anelegantaspirationforevaluationsofcomplexinterventions
![Page 35: Must the evaluation of complex interventions be complex?...the ‘real-world’, under non-trial conditions, if we want lessons for replication. For complex interventions, ‘outcome](https://reader034.fdocuments.us/reader034/viewer/2022052012/6029738018dc697373139f63/html5/thumbnails/35.jpg)
With thanks to colleagues & partners
IsoldeBirdthistlehasnoconflict ofinterestandnothingtodisclose